0.3 Linear estimators

We derived the minimum mean-squared error estimator in the previous section with no constraint on the form of the estimator. Depending on the problem, thecomputations could be a linear function of the observations (which is always the case in Gaussian problems) ornonlinear. Deriving this estimator is often difficult, which limits its application. We consider here a variation of MMSEestimation by constraining the estimator to be linear while minimizing the mean-squared estimation error. Such linear estimators may not be optimum; the conditional expected value may be nonlinear and it always has the smallest mean-squared error. Despite this occasionalperformance deficit, linear estimators have well-understood properties, they interact will with other signal processingalgorithms because of linearity, and they can always be derived, no matter what the problem.

Let the parameter estimate $()(r)$ be expressed as $(r)$ where $()$ is a linear operator: $(a_1{r}_1+a_2{r}_2)=a_1(r_1)+a_2(r_2)$ where $a_1$ , $a_2$ are scalars. Although all estimators of this form are obviously linear, the term linear estimator denotes that member of this family that minimizes the mean-squarederror.

Because of the transformation's linearity, the theory of linear vector spaces can be fruitfully used to derive the estimator andto specify its properties. One result of that theoretical framework is the well-known Orthogonality Principle (Papoulis, pp. 407-414) The linear estimator is that particular linear transformation that yieldsan estimation error orthogonal to all linear transformations of the data. The orthogonality of the error to all linear transformations is termed the universality constraint. This principle provides us not only with a formal definition of the linear estimator butalso with the mechanism to derive it. To demonstrate this intriguing result, let $\cdot$ denote the absract inner product between two vectors and $()$ the associated norm.

The estimation error for the minimum mean-squared linear estimator can be calculated to some degree without knowledge ofthe form of the estimator. The mean-squared estimation error is given by

Note that the definition of the minimum mean-squared error linear estimator makes no explicit assumptions about the parameter estimation problem beingsolved. This property makes this kind of estimator attractive in many applications where neither the a priori density of the parameter vector nor the density of the observations is known precisely. Linear transformations,however, are homogeneous: A zero-values input yields a zero output. Thus, the linear estimator is especially pertinent tothose problems where the expected value of the parameter is zero. If the expected value is nonzero, the linear estimatorwould not necessarily yield the best result (See this problem )

Express the first example in vector notation so that the observation vector is written as $$r=A+n$$ where the matrix $A$ has the form $A=\left(\begin{array}{c}1\\ \\ 1\end{array}\right)$ . The expected value of the parameter is zero. The linear estimator has the form $({}_{\mathrm{LIN}})=Lr$ , where $L$ is a $1L$ matrix. The orthogonality Principle states that the linear estimator satisfies $$\forall , \text{for all}1L\text{matricies}M\colon ((Lr-)^TMr)=0$$ To use the Orthogonality Principle to derive an equation implicitly specifying the linear estimator, the "for alllinear transformations" phrase must be interpreted. Usually the quantity specifying the linear transformation must beremoved from the constraining inner product by imposing a very stringent but equivalent condition. In this example, thisphrase becomes one about matrices. The elements of the matrix $M$ can be such that each element of the observation vector multiplies each elementof the estimation error. Thus, in this problem the Othogonality Principle means that the expected value of thematrix consisting of all pairwise priducts of these elements must be zero. $$((Lr-)r^T)=0$$ Thus, two terms must equal each other: $(Lrr^T)=(r^T)$ . The second term equals $(^2)A^T$ as the additive noise and the parameter are assumed to be statistically independent quantities. The quantity $(rr^T)$ in the first term is the correlation matrix of the observations, which is given by $AA^T(^2)+K_n$ . Here, $K_n$ is the noise covariance matrix, and $(^2)$ is the parameter's variance. The quantity $AA^T$ is a $LL$ matrix with each element equaling 1. The noise vector has independent components; the covariance matrix thusequals ${}_n^{2}I$ . The equation that $L$ must satisfy is therefore given by $$\begin{pmatrix}{L}_1 & & L_L\\ \end{pmatrix}\begin{pmatrix}{}_n^2+{}_{}^2 & {}_{}^2 & & {}_{}^2\\ {}_{}^2 & {}_n^2+{}_{}^2 & & \\ & & & {}_{}^2\\ {}_{}^2 & & {}_{}^2 & {}_n^2+{}_{}^2\\ \end{pmatrix}=\begin{pmatrix}{}_{}^2 & & {}_{}^2\\ \end{pmatrix}$$ The components of $L$ are equal and are given by $L_i=\frac{{}_{}^2}{{}_n^2+L{}_{}^2}$ . Thus, the minimum mean-squared error linear estimator has the form $$({}_{\mathrm{LIN}})(r)=\frac{{}_{}^2}{{}_{}^2+\frac{{}_n^2}{L}}\frac{1}{L}\sum r(l)$$

Note that this result equals the minimum mean-squared error estimate derived earlier under the condition that $()=0$ . Mean-squared error, linear estimators, and Gaussian problems are intimately related to each other. The linearminimum mean-squared error solution to a problem is optimal if the underlying distributions are Gaussian.