3.5 Maximum likelihood estimation - examples

Maximum likelihood estimation - examples

Exponential distribution

Let $X_1,X_2,\mathrm{...},X_n$ be a random sample from the exponential distribution with p.d.f.

$$f\left(x;\theta \right)=\frac{1}{\theta }{e}^{-x/\theta },0<x<\infty ,\theta \in \Omega =\left\{\theta ;0<\theta <\infty \right\}.$$

The likelihood function is given by

$$L\left(\theta \right)=L\left(\theta ;x_1,x_2,\mathrm{...},x_n\right)=\left(\frac{1}{\theta }{e}^{-x_1/\theta }\right)\left(\frac{1}{\theta }{e}^{-x_2/\theta }\right)···\left(\frac{1}{\theta }{e}^{-x_n/\theta }\right)=\frac{1}{{\theta }^n}\exp \left(\frac{-{\displaystyle \sum _{i=1}^n{x}_i}}{\theta }\right),0<\theta <\infty .$$

The natural logarithm of $L\left(\theta \right)$ is $$\ln L\left(\theta \right)=-\left(n\right)\ln \left(\theta \right)-\frac{1}{\theta }{\displaystyle \sum _{i=1}^n{x}_i},0<\theta <\infty .$$

Thus, $$\frac{d\left[\ln L\left(\theta \right)\right]}{d\theta }=\frac{-n}{\theta }+\frac{{\displaystyle \sum _{i=1}^n{x}_i}}{{\theta }^2}=0.$$ The solution of this equation for $\theta$ is $$\theta =\frac{1}{n}{\displaystyle \sum _{i=1}^n{x}_i}=\overline{x}.$$

Note that, $$\frac{d\left[\ln L\left(\theta \right)\right]}{d\theta }=\frac{1}{\theta }\left(-n+\frac{n\overline{x}}{\theta }\right)>0,\theta <\overline{x},$$

$$\frac{d\left[\ln L\left(\theta \right)\right]}{d\theta }=\frac{1}{\theta }\left(-n+\frac{n\overline{x}}{\theta }\right)=0,\theta =\overline{x},$$

$$\frac{d\left[\ln L\left(\theta \right)\right]}{d\theta }=\frac{1}{\theta }\left(-n+\frac{n\overline{x}}{\theta }\right)<0,\theta >\overline{x},$$

Hence, $\ln L\left(\theta \right)$ does have a maximum at $\overline{x}$ , and thus the maximum likelihood estimator for $\theta$ is $$\widehat{\theta }=\overline{X}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{X}_i.}$$ This is both an unbiased estimator and the method of moments estimator for $\theta$ .

Geometric distribution

Let $X_1,X_2,\mathrm{...},X_n$ be a random sample from the geometric distribution with p.d.f. $$f\left(x;p\right)={\left(1-p\right)}^{x-1}p,x=1,2,3,\mathrm{....}$$

The likelihood function is given by $$L\left(p\right)={\left(1-p\right)}^{x_1-1}p{\left(1-p\right)}^{x_2-1}p···{\left(1-p\right)}^{x_n-1}p=p^n{\left(1-p\right)}^{{\displaystyle \sum x_i-n}},0\le p\le 1.$$

The natural logarithm of $L\left(\theta \right)$ is $$\ln L\left(p\right)=n\ln p+\left({\displaystyle \sum _{i=1}^n{x}_i-n}\right)\ln \left(1-p\right),0<p<1.$$

Thus restricting p to $0<p<1$ so as to be able to take the derivative, we have $$\frac{d\ln L\left(p\right)}{dp}=\frac{n}{p}-\frac{{\displaystyle \sum _{i=1}^n{x}_i-n}}{1-p}=0.$$

Solving for p , we obtain $$p=\frac{n}{{\displaystyle \sum _{i=1}^n{x}_i}}=\frac{1}{\overline{x}}.$$ So the maximum likelihood estimator of p is $$\widehat{p}=\frac{n}{{\displaystyle \sum _{i=1}^n{X}_i}}=\frac{1}{X}$$

Again this estimator is the method of moments estimator, and it agrees with the intuition because, in n observations of a geometric random variable, there are n successes in the ${\displaystyle \sum _{i=1}^n{x}_i}$ trials. Thus the estimate of p is the number of successes divided by the total number of trials.

Normal distribution

Let $X_1,X_2,\mathrm{...},X_n$ be a random sample from $N\left({\theta }_1,{\theta }_2\right)$ , where $$\Omega =\left(\left({\theta }_1,{\theta }_2\right):-\infty <{\theta }_1<\infty ,0<{\theta }_2<\infty \right).$$ That is, here let ${\theta }_1=\mu$ and ${\theta }_2={\sigma }^2$ . Then $$L\left({\theta }_1,{\theta }_2\right)={\displaystyle \prod _{i-1}^n\left(\frac{1}{\sqrt{2\pi {\theta }_2}}\exp \left[-\frac{{\left(x_i-{\theta }_1\right)}^2}{2{\theta }_2}\right]\right),}$$ or equivalently, $$L\left({\theta }_1,{\theta }_2\right)={\left(\frac{1}{\sqrt{2\pi {\theta }_2}}\right)}^n\exp \left[-\frac{-{\displaystyle \sum _{i=1}^n{\left(x_i-{\theta }_1\right)}^2}}{2{\theta }_2}\right],\left({\theta }_1,{\theta }_2\right)\in \Omega .$$ The natural logarithm of the likelihood function is $$\ln L\left({\theta }_1,{\theta }_2\right)=-\frac{n}{2}\ln \left(2\pi {\theta }_2\right)-\frac{-{\displaystyle \sum _{i=1}^n{\left(x_i-{\theta }_1\right)}^2}}{2{\theta }_2}.$$

The partial derivatives with respect to ${\theta }_1$ and ${\theta }_2$ are $$\frac{\partial \left(\ln L\right)}{\partial {\theta }_1}=\frac{1}{{\theta }_2}{\displaystyle \sum _{i=1}^n\left(x_i-{\theta }_1\right)}$$ and $$\frac{\partial \left(\ln L\right)}{\partial {\theta }_2}=\frac{-n}{2{\theta }_2}+\frac{1}{2{\theta }_2^2}{\displaystyle \sum _{i=1}^n{\left(x_i-{\theta }_1\right)}^2}.$$

The equation $\frac{\partial \left(\ln L\right)}{\partial {\theta }_1}=0$ has the solution ${\theta }_1=\overline{x}$ . Setting $\frac{\partial \left(\ln L\right)}{\partial {\theta }_2}=0$ and replacing ${\theta }_1$ by $\overline{x}$ yields $${\theta }_2=\frac{1}{n}{{\displaystyle \sum _{i=1}^n\left(x_i-\overline{x}\right)}}^2.$$

By considering the usual condition on the second partial derivatives, these solutions do provide a maximum. Thus the maximum likelihood estimators $$\mu ={\theta }_1$$ and $${\sigma }^2={\theta }_2$$ are $${\widehat{\theta }}_1=\overline{X}$$ and $${\widehat{\theta }}_2=\frac{1}{n}{{\displaystyle \sum _{i=1}^n\left(X_i-\overline{X}\right)}}^2.$$

Where we compare the above example with the introductory one, we see that the method of moments estimators and the maximum likelihood estimators for $\mu$ and ${\sigma }^2$ are the same. But this is not always the case. If they are not the same, which is better? Due to the fact that the maximum likelihood estimator of $\theta$ has an approximate normal distribution with mean $\theta$ and a variance that is equal to a certain lower bound, thus at least approximately, it is unbiased minimum variance estimator. Accordingly, most statisticians prefer the maximum likelihood estimators than estimators found using the method of moments.

Binomial distribution

Observations: k successes in n Bernoulli trials.

$$f\left(x\right)=\frac{n!}{x!\left(n-x\right)!}{p}^x{\left(1-p\right)}^{n-x}$$

$$L\left(p\right)={\displaystyle \prod _{i=1}^{n}f\left(x_i\right)=}{\displaystyle \prod _{i=1}^n\left(\frac{n!}{x_i!\left(n-x_i\right)!}{p}^{x_i}{\left(1-p\right)}^{n-x_i}\right)}=\left({\displaystyle \prod _{i=1}^n\frac{n!}{x_i!\left(n-x_i\right)!}}\right)p^{x_i}{\left(1-p\right)}^{n-{\displaystyle \sum _{i=1}^n{x}_i}}$$

$$\ln L\left(p\right)={\displaystyle \sum _{i=1}^n{x}_i}\ln p+\left(n-{\displaystyle \sum _{i=1}^n{x}_i}\right)\ln \left(1-p\right)$$

$$\frac{d\ln L\left(p\right)}{dp}=\frac{1}{p}{\displaystyle \sum _{i=1}^n{x}_i}-\left(n-{\displaystyle \sum _{i=1}^n{x}_i}\right)\frac{1}{1-p}=0$$

$$\frac{\left(1-\widehat{p}\right){\displaystyle \sum _{i=1}^n{x}_i}-\left(n-{\displaystyle \sum _{i=1}^n{x}_i}\right)\widehat{p}}{\widehat{p}\left(1-\widehat{p}\right)}=0$$

$${\displaystyle \sum _{i=1}^n{x}_i}-\widehat{p}{\displaystyle \sum _{i=1}^n{x}_i}-n\widehat{p}+{\displaystyle \sum _{i=1}^n{x}_i}\widehat{p}=0$$

$$\widehat{p}=\frac{{\displaystyle \sum _{i=1}^n{x}_i}}{n}=\frac{k}{n}$$

Poisson distribution

Observations: $x_1,x_2,\mathrm{...},x_n$ , $$f\left(x\right)=\frac{{\lambda }^x{e}^{-\lambda }}{x!},x=0,1,2,\mathrm{...}$$

$$L\left(\lambda \right)={\displaystyle \prod _{i=1}^n\left(\frac{{\lambda }^{x_i}{e}^{-\lambda }}{x_i!}\right)}=e^{-\lambda n}\frac{\lambda {\displaystyle \sum _{i=1}^n{x}_i}}{{\displaystyle \prod _{i=1}^n{x}_i}}$$

$$\ln L\left(\lambda \right)=-\lambda n+{\displaystyle \sum _{i=1}^n{x}_i}\ln \lambda -\ln \left({\displaystyle \prod _{i=1}^n{x}_i}\right)$$

$$\frac{dl}{d\lambda }=-n+{\displaystyle \sum _{i=1}^n{x}_i}\frac{1}{\lambda }$$

$$-n+{\displaystyle \sum _{i=1}^n{x}_i}\frac{1}{\lambda }=0$$

$$\widehat{\lambda }=\frac{{\displaystyle \sum _{i=1}^n{x}_i}}{n}$$