2.8 Machine learning lecture 8 course notes  (Page 3/3)

This first inequality comes from the fact that

holds for any values of $Q_i$ and $\theta$ , and in particular holds for $Q_i=Q_i^{\left(t\right)}$ , $\theta ={\theta }^{(t+1)}$ . To get Equation  [link] , we used the fact that ${\theta }^{(t+1)}$ is chosen explicitly to be

and thus this formula evaluated at ${\theta }^{(t+1)}$ must be equal to or larger than the same formula evaluated at ${\theta }^{\left(t\right)}$ . Finally, the step used to get  [link] was shown earlier, and follows from $Q_i^{\left(t\right)}$ having been chosen to make Jensen's inequality hold with equality at ${\theta }^{\left(t\right)}$ .

Hence, EM causes the likelihood to converge monotonically. In our description of the EM algorithm, we said we'd run it until convergence. Given the resultthat we just showed, one reasonable convergence test would be to check if the increase in $\ell \left(\theta \right)$ between successive iterations is smaller than some tolerance parameter, and to declare convergence if EM is improving $\ell \left(\theta \right)$ too slowly.

Remark. If we define

then we know $\ell \left(\theta \right)\ge J(Q,\theta )$ from our previous derivation. The EM can also be viewed a coordinate ascent on $J$ , in which the E-step maximizes it with respect to $Q$ (check this yourself), and the M-step maximizes it with respect to $\theta$ .

Mixture of gaussians revisited

Armed with our general definition of the EM algorithm, let's go back to our oldexample of fitting the parameters $\Phi$ , $\mu$ and $\Sigma$ in a mixture of Gaussians. For the sake of brevity, we carry outthe derivations for the M-step updates only for $\Phi$ and ${\mu }_j$ , and leave the updates for ${\Sigma }_j$ as an exercise for the reader.

The E-step is easy. Following our algorithm derivation above, we simply calculate

Here, “ $Q_i(z^{\left(i\right)}=j)$ ” denotes the probability of $z^{\left(i\right)}$ taking the value $j$ under the distribution $Q_i$ .

Next, in the M-step, we need to maximize, with respect to our parameters $\Phi ,\mu ,\Sigma$ , the quantity

Let's maximize this with respect to ${\mu }_l$ . If we take the derivative with respect to ${\mu }_l$ , we find

Setting this to zero and solving for ${\mu }_l$ therefore yields the update rule

which was what we had in the previous set of notes.

Let's do one more example, and derive the M-step update for the parameters ${\Phi }_j$ . Grouping together only the terms that depend on ${\Phi }_j$ , we find that we need to maximize

However, there is an additional constraint that the ${\Phi }_j$ 's sum to 1, since they represent the probabilities ${\Phi }_j=p(z^{\left(i\right)}=j;\Phi )$ . To deal with the constraint that ${\sum }_{j=1}^k{\Phi }_j=1$ , we construct the Lagrangian

where $\beta$ is the Lagrange multiplier. We don't need to worry about the constraint that ${\Phi }_j\ge 0$ , because as we'll shortly see, the solution we'll find from this derivation will automatically satisfythat anyway. Taking derivatives, we find

Setting this to zero and solving, we get

i.e., ${\Phi }_j\propto {\sum }_{i=1}^m{w}_j^{\left(i\right)}$ . Using the constraint that ${\sum }_j{\Phi }_j=1$ , we easily find that $-\beta ={\sum }_{i=1}^m{\sum }_{j=1}^k{w}_j^{\left(i\right)}={\sum }_{i=1}^{m}1=m$ . (This used the fact that $w_j^{\left(i\right)}=Q_i(z^{\left(i\right)}=j)$ , and since probabilities sum to 1, ${\sum }_j{w}_j^{\left(i\right)}=1$ .) We therefore have our M-step updates for the parameters ${\Phi }_j$ :

The derivation for the M-step updates to ${\Sigma }_j$ are also entirely straightforward.