3.5 Cramer-rao bound (Page 2/2)

Statistical signal processing Page 2 / 2

Let's evaluate the Cramr-Rao bound for the example we have been discussing: the estimation of the mean and varianceof a length $L$ sequence of statistically independent Gaussian random variables. Let theestimate of the mean $_{1}$ be the sample average $_{1} r l L$ ; as shown in the last example, this estimate is unbiased. Let the estimate of the variance $_{2}$ be the unbiased estimate $_{2} r l_{1} 2 L 1$ . Each term in the Fisher information matrix $F$ is given by the expected value of the paired products of derivatives of thelogarithm of the likelihood function. $F i j_{i} p r r_{j} p r r$ The logarithm of the likelihood function is $p r r L 2 2_{2} 1 2_{2} l 0 L 1 r l_{1} 2$ its partial derivatives are

_{1} p r r 1_{2} l 0 L 1 r l_{1}

_{2} p r r L 2_{2} 1 2_{2} 2 l 0 L 1 r l_{1} 2

and its second partials are

_{1} 2 2 p r r L_{2}

_{1}_{2} 2 p r r 1_{2} 2 l 0 L 1 r l_{1}

_{2}_{1} 2 p r r 1_{2} 2 l 0 L 1 r l_{1}

_{2} 2 2 p r r L 2_{2} 2 1_{2} 3 l 0 L 1 r l_{1} 2

The Fisher information matrix has the surprisingly simple form

F L_{2} 0 0 L 2_{2} 2

its inverse is also a diagonal matrix with the elements on the main diagonal equalling the reciprocal of those in theoriginal matrix. Because of the zero-values off-diagonal entries in the Fisher information matrix, the errors betweenthe corresponding estimates are not inter-dependent. In this problem, the mean-square estimation error can be no smallerthan

_{1}_{1} 2_{2} L

_{2}_{2} 2 2_{2} 2 L

Note that nowhere in the preceding example did the form of the estimator enter into the computation of thebound. The only quantity used in the computation of the Cramr-Rao bound is the logarithm of the likelihood function, which is a consequence of the problem statement, nothow it is solved. Only in the case of unbiased estimators is the bound independent of the estimatorsused.

That's why we assumed in the example that we used an unbiased estimator for thevariance.

Because of this property, the Cramr-Rao bound is frequently used to assess the performance limits thatcan be obtained with an unbiased estimator in a particular problem. When bias is present, the exact form of the estimator'sbias explicitly enters the computation of the bound. All too frequently, the unbiased form is used in situations where the existence of an unbiased estimator can be questioned. As we shall see, one such problem is time delayestimation, presumably of some importance to the reader. This misapplication of the unbiased Cramr-Rao arises from desperation: the estimator is so complicated and nonlinear thatcomputing the bias is nearly impossible. As shown in this problem , biased estimators can yield mean-squared error smaller as well aslarger than the unbiased version of the Cramr-Rao bound. Consequently, desperation can yield misinterpretationwhen a general result is misapplied.

In the single-parameter estimation problem, the Cramr-Rao bound incorporating bias has the well-known form

Note that this bound differs somewhat from that originally given by Cramr (1946) p.480 ; his derivation ignores the additive bias term

b b

2 b 2 1 b 2 p r r 2

Note that the sign of the bias's derivative determines whether this bound is larger or potentially smaller than the unbiasedversion, which is obtained by setting the bias term to zero.

Efficiency

An interesting question arises: when, if ever, is the bound satisfied with equality? Recalling the details of thederivation of the bound, equality results when the quantity $x x$ equals zero. As this quantity is the expected value of the square of $x$ , it can only equal zero if $x 0$ . Substituting in the form of the column matrices and $x$ , equality in the Cramr-Rao bound results whenever

p r r I b F r b

This complicated expression means that only if estimationproblems (as expressed by the a priori density have the form of the right side of this equation can the mean-squared error equal the Cramr-Rao bound. In particular, the gradient of the log likelihood function can only depend on the observations through the estimator. In all other problems, the Cramr-Rao bound is a lower bound but not a tight one no estimator can have error characteristics that equal it. In such cases, we have limitedinsight into ultimate limitations on estimation error size with the Cramr-Rao bound. However, consider the case where the estimator is unbiased (

b 0

). In addition, note the maximum likelihood estimate occurs when the gradient of the logarithm of the likelihoodfunction equals zero:

p r r 0

when

_{ML}

. In this case, the condition for equality in the Cramr-Rao bound becomes

F_{ML} 0

As the Fisher information matrix is positive-definite, we conclude that if the estimator equals the maximum likelihoodestimator, equality in the Cramr-Rao bound can be satisfied with equality, only the maximum likelihood estimate will achieve it. To use estimation theoretic terminology, if an efficient estimate exists, it is the maximum likelihood estimate. This result stresses the importance of maximum likelihood estimates, despite theseemingly ad hoc manner by which they are defined.

Consider the Gaussian example being examined so frequently in this section. The components of the gradient of thelogarithm of the likelihood function were given earlier by and . These expressions can be rearranged to reveal

_{1} p r r_{2} p r r L_{2} 1 L l l r l_{1} L 2_{2} 1 2_{2} 2 l l r l_{1} 2

The first component, which corresponds to the estimate of the mean, is expressed in the form required for the existence of an efficient estimate. Thesecond component--the partial with respect to the variance

_{2}

-- cannot be rewritten in a similar fashion. No unbiased, efficient estimate of thevariance exits in this problem. The mean-squared error of the variance's unbiased estimate, but not the maximumlikelihood estimate, is lower-bounded by

2_{2} 2 L 1 2

. This error is strictly greater than the Cramr-Rao bound of

2_{2} 2 L 2

. As no unbiased estimate of the variance can have a mean-squared error equal to the Cramr-Rao bound (no efficient estimate exists for the variance in theGaussian problem), one presumes that the closeness of the error of our unbiased estimator to the bound implies that itpossesses the smallest squared-error of any estimate. This presumption may, of course, be incorrect.

Properties of the maximum likelihood estimator

The maximum likelihood estimate is the most used estimation technique for nonrandom parameters. Not only because of itsclose linkage to the Cramr-Rao bound, but also because it has desirable asymptotic properties in the context of any problem (Cramr (1946) pp. 500-506) .

The maximum likelihood estimate is at least asymptotically unbiased. It may be unbiased for any number of observations (as in the estimation of the mean of asequence of independent random variable) for some problems.
The maximum likelihood estimate is consistent.
The maximum likelihood estimates is asymptotically efficient. As more and more data are incorporated into an estimate, theCramr-Rao bound accurately projects the best attainable error and the maximum likelihood estimate hasthose optimal characteristics.
Asymptotically, the maximum likelihood estimate is distributed as a Gaussian random variable. Because of the previous properties, the mean asymptotically equals the parameter and the covariancematrix is $L F$ .

Most would agree that a "good" estimator should have these properties. What these results do not provide is assessment ofhow many observations are needed for the asymptotic results to apply to some specified degree of precision. Consequently,they should be used with caution; for instance, some other estimator may have a smaller mean-square error than themaximum likelihood for a modest number of observations.

<< Chapter < Page Page > Chapter >>

Read also:

Get Jobilize Job Search Mobile App in your pocket Now!

100% Free Mobile Applications
Receive real-time job alerts and never miss the right job again

Source: OpenStax, Statistical signal processing. OpenStax CNX. Dec 05, 2011 Download for free at http://cnx.org/content/col11382/1.1

Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Statistical signal processing' conversation and receive update notifications?

Ask

	Anthropology Biology Culture By Richley Crapo Start Assignment
	17 AP Key Terms 17 Endocrine System By OpenStax Start Key Terms
©flickr: U.S.	Biology Chapter 9 By Michael Sag Start Exam
	Pre-class Online Quiz - Case Control Studies By Mucho Mizinduko Start Quiz
	2 Psychology MCQ 2009 1 Exam By John Gabrieli Start Exam
	11 Sociology 11 Race and Ethnicity MCQ By OpenStax Start Quiz
	Clinical Issues in TB Management By Cath Yu Start Quiz
	5 Microeconomics 05 Elasticity By OpenStax Start Flashcards
	10 AP Key Terms 10 Muscle Tissue Key Terms By OpenStax Start Key Terms
	2011 Dynamics CRM By Danielrosenberger Start Quiz