3.4 Maximum likelihood estimation (mle)

Maximum likelihood estimation (mle)

Likelihood function

From a statistical standpoint, the data vector $x=\left(x_1,x_2,\mathrm{...},x_n\right)$ as the outcome of an experiment is a random sample from an unknown population. The goal of data analysis is to identify the population that is most likely to have generated the sample. In statistics, each population is identified by a corresponding probability distribution. Associated with each probability distribution is a unique value of the model’s parameter. As the parameter changes in value, different probability distributions are generated. Formally, a model is defined as the family of probability distributions indexed by the model’s parameters.

Let denote the probability distribution function (PDF) by $f\left(x\text{|}\theta \right)$ that specifies the probability of observing data y given the parameter w . The parameter vector $\theta =\left({\theta }_1,{\theta }_2,\mathrm{...},{\theta }_k\right)$ is a vector defined on a multi-dimensional parameter space. If individual observations, $x_i\text{'}s$ are statistically independent of one another, then according to the theory of probability, the PDF for the data $x=\left(x_1,x_2,\mathrm{...},x_n\right)$ can be expressed as a multiplication of PDFs for individual observations,

$$f\left(x,\theta \right)=f\left(x_1,\theta \right)f\left(x_2,\theta \right)···f\left(x_n,\theta \right),$$ $$L\left(\theta \right)={\displaystyle \prod _{i=1}^{n}f\left(x_i|\theta \right)}.$$

To illustrate the idea of a PDF , consider the simplest case with one observation and one parameter, that is, $n=k=1$ . Suppose that the data x represents the number of successes in a sequence of 10 independent binary trials (e.g., coin tossing experiment) and that the probability of a success on any one trial, represented by the parameter, $\theta$ is 0.2. The PDF in this case is then given by

$$f\left(x|\theta =0.2\right)=\frac{10!}{x!\left(10-x\right)!}{\left(0.2\right)}^x{\left(0.8\right)}^{10-x},\left(x=0.1,\mathrm{...},10\right),$$

which is known as the binomial probability distribution. The shape of this PDF is shown in the top panel of Figure 1 . If the parameter value is changed to say w = 0.7, a new PDF is obtained as $$f\left(x|\theta =0.7\right)=\frac{10!}{x!\left(10-x\right)!}{\left(0.7\right)}^x{\left(0.3\right)}^{10-x},\left(x=0.1,\mathrm{...},10\right);$$ whose shape is shown in the bottom panel of Figure 1 . The following is the general expression of the binomial PDF for arbitrary values of $\theta$ and n :

$$f\left(x|\theta \right)=\frac{n!}{\theta !\left(n-x\right)!}{\theta }^x{\left(1-\theta \right)}^{n-x},0\le \theta \le 1,x=0.1,\mathrm{...},n;$$

which as a function of y specifies the probability of data y for a given value of the parameter $\theta$ . The collection of all such PDFs generated by varying parameter across its range (0 - 1 in this case) defines a model.

Maximum likelihood estimation

Once data have been collected and the likelihood function of a model given the data is determined, one is in a position to make statistical inferences about the population, that is, the probability distribution that underlies the data. Given that different parameter values index different probability distributions ( Figure 1 ), we are interested in finding the parameter value that corresponds to the desired PDF.

The principle of maximum likelihood estimation (MLE) , originally developed by R. A. Fisher in the 1920s, states that the desired probability distribution be the one that makes the observed data most likely, which is obtained by seeking the value of the parameter vector that maximizes the likelihood function $L\left(\theta \right)$ . The resulting parameter, which is sought by searching the multidimensional parameter space, is called the MLE estimate , denoted by

$$\theta MLE=\left({\theta }_{1}MLE,\mathrm{...},{\theta }_{k}MLE\right).$$

Let p equal the probability of success in a sequence of Bernoulli trials or the proportion of the large population with a certain characteristic. The method of moments estimate for p is relative frequency of success (having that characteristic). It will be shown below that the maximum likelihood estimate for p is also the relative frequency of success.

Suppose that X is $b\left(1,p\right)$ so that the p.d.f. of X is $$f\left(x;p\right)=p^x{\left(1-p\right)}^{1-x},x=0,1,0\le p\le 1.$$ Sometimes is written $$p\in \Omega =\left[p:0\le p\le 1\right],$$ where $\Omega$ is used to represent parameter space, that is, the space of all possible values of the parameter. A random sample $X_1,X_2,\mathrm{...},X_n$ is taken, and the problem is to find an estimator $\text{u}\left({\text{X}}_{\text{1}},{\text{X}}_{\text{2}},\mathrm{...},{\text{X}}_{\text{n}}\right)$ such that $\text{u}\left(x_{\text{1}},x_{\text{2}},\mathrm{...},x_{\text{n}}\right)$ is a good point estimate of p , where $x_{\text{1}},x_{\text{2}},\mathrm{...},x_{\text{n}}$ are the observed values of the random sample. Now the probability that $X_1,X_2,\mathrm{...},X_n$ takes the particular values is

$$P\left(X_1=x_1,\mathrm{...},X_n=x_n\right)={\displaystyle \prod _{i=1}^n{p}^{x_i}{\left(1-p\right)}^{1-x_i}}=p{\displaystyle \sum x_i}{\left(1-p\right)}^{n-{\displaystyle \sum x_i}},$$

which is the joint p.d.f. of $X_1,X_2,\mathrm{...},X_n$ evaluated at the observed values. One reasonable way to proceed towards finding a good estimate of p is to regard this probability (or joint p.d.f.) as a function of p and find the value of p that maximizes it. That is, find the p value most likely to have produced these sample values. The joint p.d.f., when regarded as a function of p , is frequently called the likelihood function . Thus here the likelihood function is:

$$L\left(p\right)=L\left(p;x_1,x_2,\mathrm{...},x_n\right)=f\left(x_1;p\right)f\left(x_2;p\right)···f\left(x_n;p\right)=p^{{\displaystyle \sum x_i}}{\left(1-p\right)}^{n-{\displaystyle \sum x_i}},0\le p\le 1.$$

To find the value of p that maximizes $L\left(p\right)$ first take its derivative for $0<p<1:$

$$\frac{dL\left(p\right)}{dp}=\left({\displaystyle \sum x_i}\right)p^{n-{\displaystyle \sum x_i}}{\left(1-p\right)}^{n-{\displaystyle \sum x_i}}-\left(n-{\displaystyle \sum x_i}\right)p^{{\displaystyle \sum x_i}}{\left(1-p\right)}^{n-{\displaystyle \sum x_i}-1}.$$

Setting this first derivative equal to zero gives $$p^{{\displaystyle \sum x_i}}{\left(1-p\right)}^{n-{\displaystyle \sum x_i}}\left[\frac{{\displaystyle \sum x_i}}{p}-\frac{n-{\displaystyle \sum x_i}}{1-p}\right]=0.$$

Since $0<p<1$ , this equals zero when $$\frac{{\displaystyle \sum x_i}}{p}-\frac{n-{\displaystyle \sum x_i}}{1-p}=0.$$ Or, equivalently, $$p=\frac{{\displaystyle \sum x_i}}{n}=\overline{x}.$$

The corresponding statistics, namely ${\displaystyle \sum X_i/n=\overline{X}}$ , is called the maximum likelihood estimator and is denoted by $\widehat{p}$ ,that is, $$\widehat{p}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{X}_i=\overline{X}}.$$

When finding a maximum likelihood estimator, it is often easier to find the value of parameter that minimizes the natural logarithm of the likelihood function rather than the value of the parameter that minimizes the likelihood function itself. Because the natural logarithm function is an increasing function, the solution will be the same. To see this, the example which was considered above gives for $0<p<1$ ,

$$\ln L\left(p\right)=\left({\displaystyle \sum _{i=1}^n{x}_i}\right)\ln p+\left(n-{\displaystyle \sum _{i=1}^n{x}_i}\right)\ln \left(1-p\right).$$

To find the maximum, set the first derivative equal to zero to obtain

$$\frac{d\left[\ln L\left(p\right)\right]}{dp}=\left({\displaystyle \sum _{i=1}^n{x}_i}\right)\left(\frac{1}{p}\right)+\left(n-{\displaystyle \sum _{i=1}^n{x}_i}\right)\left(\frac{-1}{1-p}\right)=0,$$

which is the same as previous equation. Thus the solution is $p=\overline{x}$ and the maximum likelihood estimator for p is $\widehat{p}=\overline{X}.$

Motivated by the preceding illustration, the formal definition of maximum likelihood estimators is presented. This definition is used in both the discrete and continuous cases. In many practical cases, these estimators (and estimates) are unique. For many applications there is just one unknown parameter. In this case the likelihood function is given by $$L\left(\theta \right)={\displaystyle \prod _{i=1}^{n}f\left(x_i,\theta \right)}.$$