0.14 Maximum likelihood and complexity regularization

Review : maximum likelihood estimation

In the last lecture , we have $n$ i.i.d observations drawn from an unknown distribution

With loss function defined as $l(\theta ,Y_i)=-log{p}_{\theta }\left(Y_i\right)$ , the empirical risk is

Essentially, we want to choose a distribution from the collection of distributions within the parameter space that minimizes the empirical risk, i.e., we would like to select

where

The risk is defined as

Note that ${\theta }^{*}$ minimizes $R\left(\theta \right)$ over $\Theta$ .

Finally, the excess risk of $\theta$ is defined as

We recognized that the excess risk corresponding to this loss function is simply the Kullback-Leibler (KL) Divergence or Relative Entropy , denoted by $K(p_{{\theta }_1},p_{{\theta }_2})$ . It is easy to see that $K(p_{{\theta }_1},p_{{\theta }_2})$ is always non-negative and is zero if and only if $p_{{\theta }_1}=p_{{\theta }_2}$ . KL divergence measures how different two probability distributions are and therefore is natural to measure convergence of the maximum likelihood procedures. However, $K(p_{{\theta }_1},p_{{\theta }_2})$ is not a distance metric because it is not symmetric and does not satisfy the triangle inequality. For this reason, two other quantities play a key role in maximum likelihood estimation, namely Hellinger Distance and Affinity .

The Hellinger distance is defined as

We proved that the squared Hellinger distance lower bounds the KL divergence:

The affinity is defined as

we also proved that

Maximum likelihood estimation and complexity regularization

Suppose that we have n i.i.d training samples, ${\{X_i,Y_i\}}_{i=1}^n\stackrel{i.i.d.}{\sim }{p}_{XY}$  .

Using conditional probability, $p_{XY}$ can be written as

Let's assume for the moment that $p_X$ is completely unknown, but $p_{Y|X=x}\left(y\right)$ has a special form:

where $p_{Y|X=x}\left(y\right)$ is a known parametric density function with parameter $f^{*}\left(x\right)$ .