2.8 Machine learning lecture 8 course notes

The em algorithm

In the previous set of notes, we talked about the EM algorithm as applied to fitting a mixture of Gaussians. In this set of notes,we give a broader view of the EM algorithm, and show how it can be applied to a large family of estimation problemswith latent variables. We begin our discussion with a very useful result called Jensen's inequality

Jensen's inequality

Let $f$ be a function whose domain is the set of real numbers. Recall that $f$ is a convex function if $f^{\text{'}\text{'}}\left(x\right)\ge 0$ (for all $x\in \mathbb{R}$ ). In the case of $f$ taking vector-valued inputs, this is generalized to the condition that its hessian $H$ is positive semi-definite ( $H\ge 0$ ). If $f^{\text{'}\text{'}}\left(x\right)>0$ for all $x$ , then we say $f$ is strictly convex (in the vector-valued case, the corresponding statement is that $H$ must be positive definite, written $H>0$ ). Jensen's inequality can then be stated as follows:

Theorem. Let $f$ be a convex function, and let $X$ be a random variable. Then:

Moreover, if $f$ is strictly convex, then $\mathrm{E}\left[f\right(X\left)\right]=f\left(\mathrm{E}X\right)$ holds true if and only if $X=\mathrm{E}\left[X\right]$ with probability 1 (i.e., if $X$ is a constant).

Recall our convention of occasionally dropping the parentheses when writing expectations, so in the theorem above, $f\left(\mathrm{E}X\right)=f\left(\mathrm{E}\right[X\left]\right)$ .

For an interpretation of the theorem, consider the figure below.

Here, $f$ is a convex function shown by the solid line. Also, $X$ is a random variable that has a 0.5 chance of taking the value $a$ , and a 0.5 chance of taking the value $b$ (indicated on the $x$ -axis). Thus, the expected value of $X$ is given by the midpoint between $a$ and $b$ .

We also see the values $f\left(a\right)$ , $f\left(b\right)$ and $f\left(\mathrm{E}\right[X\left]\right)$ indicated on the $y$ -axis. Moreover, the value $\mathrm{E}\left[f\right(X\left)\right]$ is now the midpoint on the $y$ -axis between $f\left(a\right)$ and $f\left(b\right)$ . From our example, we see that because $f$ is convex, it must be the case that $\mathrm{E}\left[f\right(X\left)\right]\ge f\left(\mathrm{E}X\right)$ .

Incidentally, quite a lot of people have trouble remembering which way the inequality goes, and remembering a picture like this isa good way to quickly figure out the answer.

Remark. Recall that $f$ is [strictly] concave if and only if $-f$ is [strictly]convex (i.e., $f^{\text{'}\text{'}}\left(x\right)\le 0$ or $H\le 0$ ). Jensen's inequality also holds for concave functions $f$ , but with the direction of all the inequalities reversed ( $\mathrm{E}\left[f\right(X\left)\right]\le f\left(\mathrm{E}X\right)$ , etc.).

The em algorithm

Suppose we have an estimation problem in which we have a training set $\{x^{\left(1\right)},...,x^{\left(m\right)}\}$ consisting of $m$ independent examples. We wish to fit the parameters of a model $p(x,z)$ to the data, where the likelihood is given by

But, explicitly finding the maximum likelihood estimates of the parameters $\theta$ may be hard. Here, the $z^{\left(i\right)}$ 's are the latent random variables; and it is often the case that if the $z^{\left(i\right)}$ 's were observed, then maximum likelihood estimation would be easy.

In such a setting, the EM algorithm gives an efficient method for maximum likelihood estimation. Maximizing $\ell \left(\theta \right)$ explicitly might be difficult, and our strategy will be to instead repeatedlyconstruct a lower-bound on $\ell$ (E-step), and then optimize that lower-bound (M-step).

For each $i$ , let $Q_i$ be some distribution over the $z$ 's ( ${\sum }_z{Q}_i\left(z\right)=1$ , $Q_i\left(z\right)\ge 0$ ). Consider the following: If $z$ were continuous, then $Q_i$ would be a density, and the summations over $z$ in our discussion are replaced with integrals over $z$ .