Introduction to estimation theory

In searching for methods of extracting information from noisy observations, this chapter describes estimation theory , which has the goal of extracting from noise-corrupted observations the values of disturbanceparameters (noise variance, for example), signal parameters (amplitude or propagation direction), or signalwaveforms . Estimation theory assumes that the observations contain an information-bearing quantity, therebytacitly assuming that detection-based preprocessing has been performed (in other words, do I have something in theobservations worth estimating?). Conversely, detection theory often requires estimation of unknown parameters: Signal presenceis assumed, parameter estimates are incorporated into the detection statistic, and consistency of observations andassumptions tested. Consequently, detection and estimation theory form a symbiotic relationship, each requiring the otherto yield high-quality signal processing algorithms.

Despite a wide variety of error criteria and problem frameworks, the optimal detector is characterized by a single result: thelikelihood ratio test. Surprisingly, optimal detectors thus derived are usually easy to implement, not often requiringsimplification to obtain a feasible realization in hardware or software. In contrast to detection theory, no fundamentalresult in estimation theory exists to be summoned to attack the problem at hand. The choice of error criterion and itsoptimization heavily influences the form of the estimation procedure. Because of the variety of criterion-dependentestimators, arguments frequently rage about which of several optimal estimators is "better." Each procedure is optimum forits assumed error criterion; thus, the argument becomes which error criterion best describes some intuitive notion of quality.When more ad hoc, noncriterion-based procedures

Terminology in estimation theory

More so than detection theory, estimation theory relies on jargon to characterize the properties of estimators. Withoutknowing any estimation technique, let's use parameter estimation as our discussion prototype. The parameterestimation problem is to determine from a set of $L$ observations, represented by the $L$ -dimensional vector $r$ , the values of parameters denoted by the vector $$ . We write the estimate of this parameter vector as $()(r)$ , where the "hat" denotes the estimate, and the functional dependence on $r$ explicitly denotes the dependence of the estimate on the observations. Thisdependence is always present

Bias

An estimate is said to be unbiased if the expected value of the estimate equals the true value of theparameter: $(, ())=$ . Otherwise, the estimate is said to be biased : $(, ())\neq$ . The bias $b()$ is usually considered to be additive, so that $b()=(, ())-$ . When we have a biased estimate, the bias usually depends on the number of observations $L$ . An estimate is said to be asymptotically unbiased if the bias tends to zero for large $L$ : $\lim_{L\to }L\to$∞ b 0 . An estimate's variance equals the mean-squared estimation error only if the estimate is unbiased.

An unbiased estimate has a probability distribution where the mean equals the actual value of the parameter. Shouldthe lack of bias be considered a desirable property? If many unbiased estimates are computed from statisticallyindependent sets of observations having the same parameter value, the average of these estimates will be close to thisvalue. This property does not mean that the estimate has less error than a biased one; thereexist biased estimates whose mean-squared errors are smaller than unbiased ones. In such cases, the biased estimate isusually asymptotically unbiased. Lack of bias is good, but that is just one aspect of how we evaluate estimators.

Consistency

We term an estimate consistent if the mean-squared estimation error tends to zero as the number ofobservations becomes large: $\lim_{L\to }L\to$∞ 0 . Thus, a consistent estimate must be at least asymptotically unbiased. Unbiased estimates do exist whoseerrors never diminish as more data are collected: Their variances remain nonzero no matter how much data areavailable. Inconsistent estimates may provide reasonable estimates when the amount of data is limited, but have thecounterintuitive property that the quality of the estimate does not improve as the number of observations increases.Although appropriate in the proper circumstances (smaller mean-squared error than a consistent estimate over apertinent range of values of $L$ , consistent estimates are usually favored in practice.

Efficiency

As estimators can be derived in a variety of ways, their error characteristics must always be analyzed and compared.In practice, many problems and the estimators derived for them are sufficiently complicated to render analytic studiesof the errors difficult, if not impossible. Instead, numerical simulation and comparison with lower bounds on theestimation error are frequently used instead to assess the estimator performance. An efficient estimate has a mean-squared error that equals a particular lower bound: the Cramr-Rao bound . If an efficient estimate exists (the Cramr-Rao bound is the greatest lower bound), it is optimum in the mean-squaredsense: No other estimate has a smaller mean-squared error (see Maximum Likelihood Estimators for details).

For many problems no efficient estimate exists. In such cases, the Cramr-Rao bound remains a lower bound, but its value is smaller than that achievable by anyestimator. How much smaller is usually not known. However, practitioners frequently use the Cramr-Rao bound in comparisons with numerical error calculations. Anotherissue is the choice of mean-squared error as the estimation criterion; it may not suffice to pointedly assess estimatorperformance in a particular problem. Nevertheless, every problem is usually subjected to a Cramr-Rao bound computation and the existence of an efficient estimate considered.