3.5 Cramer-rao bound  (Page 21/2)

The mean-squared error for any estimate of a nonrandom parameter has a lower bound, the Cramr-Rao Bound (Cramr (1946) pp. 474-477) , which defines the ultimate accuracy of any estimation procedure. This lower bound, as shown later, is intimately related to the maximum likelihood estimator.

We seek a "bound" on the mean-squared error matrix $M$ defined to be $$M=((()-)(()-)^T)=(^T)$$ A matrix is "lower bounded" by a second matrix if the difference between the two is a non-negative definite matrix. Define thecolumn matrix $x$ to be $$x=\begin{pmatrix}()--b()\\ \frac{d \ln p(r, , r)}{d }\\ \end{pmatrix}$$ where $b()$ denotes the column matrix of estimator biases. To derive the Cramr-Rao bound, evaluate $(xx^T)$ . $$(xx^T)=\begin{pmatrix}M-bb^T & I+\frac{d b}{d }\\ (I+\frac{d b}{d })^T & F\\ \end{pmatrix}$$ where $\frac{d b}{d }$ represents the matrix of partial derivatives of the bias $\frac{\partial^1{b}_i}{\partial {}_j}$ and the matrix $F$ is the Fisher information matrix

Returning to the derivation, the matrix $(xx^T)$ is non-negative definite because it is a correlation matrix. Thus, for any column matrix, $$ , the quadratic form $^T(xx^T)$ is non-negative. Choose a form for $$ that simplifies the quadratic form. A convenient choice is $$=\begin{pmatrix}\\ -(F^{(-1)}(I+\frac{d b}{d })^T)\\ \end{pmatrix}$$ where $$ is an arbitrary column matrix. The quadratic form becomes in this case $$^T(xx^T)=^T\begin{pmatrix}M-bb^T-(I+\frac{d b}{d })F^{(-1)}(I+\frac{d b}{d })^T\\ \end{pmatrix}$$ As this quadratic form must be non-negative, the matrix expression enclosed in brackets must be non-negativedefinite. We thus obtain the well-known Cramr-Rao bound on the mean-square error matrix.

This form for the Cramr-Rao Bound does not mean that each term in the matrix of squared errors is greater than the corresponding term in thebounding matrix. As stated earlier, this expression means that the difference between these matrices is non-negativedefinite. For a matrix to be non-negative definite, each term on the main diagonal must be non-negative. The elements of the maindiagonal of $(^T)$ are the squared errors of the estimate of the individualparameters. Thus, for each parameter, the mean-squared estimation error can be no smaller than $$((({}_i)-{}_i)^2)\ge b_i()^2+((I+\frac{d b}{d })F^{(-1)}(I+\frac{d b}{d })^T)_{i, i}$$

This bound simplifies greatly if the estimator is unbiased ( $b=0$ ). In this case, the Cramr-Rao bound becomes $$((({}_i)-{}_i)^2)\ge F^{(-1)}_{i, i}$$ Thus, the mean-squared error for each parameter in a multiple-parameter, unbiased-estimator problem can be no smallerthan the corresponding diagonal term in the inverse for the Fisher information matrix. In such problems, the estimate's error characteristicsof any parameter become intertwined with the other parameters in a complicated way. Any estimator satisfying the Cramr-Rao bound with equality is said to be efficient .