3.4 Maximum likelihood estimators of parameters

Let $r(l)$ be a sequence of independent, identically distributed Gaussian random variables having an unknown mean $$ but a known variance ${}_n^2$ . Often, we cannot assign a probability density to a parameter of a random variable's density; we simply do not knowwhat the parameter's value is. Maximum likelihood estimates are often used in such problems. In the specific case here, thederivative of the logarithm of the likelihood function equals $$\frac{\partial^1\ln p(, r)}{\partial }=\frac{1}{{}_n^2}\sum_{l=0}^{L-1} r(l)-$$ The solution of this equation is the maximum likelihood estimate, which equals the sample average. $$({}_{\mathrm{ML}})=\frac{1}{L}\sum_{l=0}^{L-1} r(l)$$ The expected value of this estimate $(, ({}_{\mathrm{ML}}))$ equals the actual value $$ , showing that the maximum likelihood estimate is unbiased. Themean-squared error equals $\frac{{}_n^2}{L}$ and we infer that this estimate is consistent.

Let's extend the previous example to the situation where neither the mean nor the variance of a sequence of independentGaussian random variables is known. The likelihood function is, in this case, $$p(, r)=\prod_{l=0}^{L-1} \frac{1}{\sqrt{2\pi {}_2}}e^{-(\frac{1}{2{}_2}(r(l)-{}_1)^2)}$$ Evaluating the partial derivatives of the logarithm of this quantity, we find the following set of two equations to solvefor ${}_1$ , representing the mean, and ${}_2$ , representing the variance.

The expected value of $({}_1^{\mathrm{ML}})$ equals the actual value of ${}_1$ ; thus, this estimate is unbiased. However, the expected value of the estimate of the variance equals ${}_2\frac{L-1}{L}$ . The estimate of the variance is biased, but asymptotically unbiased. This bias can be removed by replacingthe normalization of $L$ in the averaging computation for $({}_2^{\mathrm{ML}})$ by $L-1$ .