1.12 The hilbert space of random variables

Random variable spaces

Probability – notation primer

Definition 1 A random variable x is defined by a distribution function

The density function is given by

Definition 2 The expectation of a function $g\left(x\right)$ over the random variable $x$ is

Definition 3 Pairs of random variables $X,Y$ are defined by the joint distribution function

The joint density function is given by

The expectation of a function $g(x,y)$ is given by

A hilbert space of random variables

Definition 4 Let $\{Y_1,\cdots ,Y_n\}$ be a collection of zero-mean ( $E\left[Y_i\right]=0$ ) random variables. The space $H$ of all random variables that are linear combinations of those $n$ random variables $\{y_1,\cdots ,y_n\}$ is a Hilbert space with inner product

We can easily check that this is a valid inner product:

• $⟨x,x⟩=E\left[x\overline{x}\right]={\int }_{-\infty }^{\infty }{\left|x\right|}^2{f}_x\left(x\right)\text{d}x=E\left[{\left|x\right|}^2\right]\ge 0$ ;
• $⟨x,x⟩=0$ if and only if $f_X\left(x\right)=\delta \left(x\right)$ , i.e., if $X$ is a random variable that is deterministically zero (and this random variable is the “zero” of this Hilbert space);
• $⟨x,y⟩=\overline{⟨y,x⟩}$ ;
• $⟨x+y,z⟩=E\left[(x+y)\overline{z}\right]=E\left[x\overline{z}\right]+E\left[y\overline{z}\right]=⟨x,z⟩+⟨y,z⟩;$

Note in particular that orthogonality, i.e., $⟨x,y⟩=0$ , implies $E\left[x\overline{y}\right]=0$ , i.e., $x$ and $y$ are independent random variables. Additionally, the induced norm $\parallel X\parallel =\sqrt{⟨X,X⟩}=\sqrt{E\left[\right|x{|}^2]}$ is the standard deviation of the zero-mean random variable $X$ .

A hilbert space of random vectors

One can define random vectors $X$ , $Y$ whose entries are random variables:

For these, the following inner product is an extension of that given above:

The induced norm is

the expected norm of the vector $x$ .

Minimum mean square error estimation

In an MMSE estimation problem, we consider $Y=AX+N$ , where $X,Y$ are two random vectors and $N$ is usually additive white Gaussian noise ( $Y$ is $m\times 1$ , $A$ is $m\times n$ , X is $n\times 1$ , and $N\sim \mathcal{N}(0,{\sigma }^{2}I)$ is $m\times 1$ ). Due to this noise model, we want an estimate $\widehat{X}$ of $X$ that minimizes $E\left[\parallel X-,\widehat{X},{\parallel }^2\right]$ ; such an estimate has highest likelihood under an additive white Gaussian noise model. For computational simplicity, we often want to restrict the estimator to be linear, i.e.

where $K_i^H$ denotes the $i^{th}$ row of the estimation matrix $K$ and ${\widehat{X}}_i=K_i^{H}Y$ . We use the definition of the ${\ell }_2$ norm to simplify the equation:

Since the terms involved in the sum are independent from each other and nonnegative, this minimization can be posed in terms of $n$ individual minimizations: for $i=1,2,...,n$ , we solve

where the norm is the induced norm for the Hilbert space of random variables. Note at this point that the set of random variables ${\sum }_{i=1}^n\overline{K_{ij}}{Y}_j$ over the choices of $K_i$ can be written as $\mathrm{span}\left({\left\{Y_j\right\}}_{j=1}^m\right)$ . Thus, the optimal $K_i$ is given by the coefficients of the closest point in $\mathrm{span}\left({\left\{Y_j\right\}}_{j=1}^m\right)$ to the random variable $X_i$ according to the induced norm for the Hilbert space of random variables. Therefore, we solve for $K_i$ using results from the projection theorem with the corresponding inner product. Recall that given a basis $Y_i$ for the subspace of interest, we obtain the equation ${\beta }_i=G{\left(K_i^H\right)}^T=G\overline{K_i}$ , where ${\beta }_{i,j}=〈X_i,,,Y_j〉$ and $G$ is the Gramian matrix. More specifically, we have

Thus, one can solve for $\overline{K_i}=G^{-1}{\beta }_i$ . In the Hilbert space of random variables, we have

Here $R_Y$ is the correlation matrix of the random vector $Y$ and ${\rho }_{X_{i}Y}$ is the cross-correlation vector of the random variable $X_i$ and vector $Y$ . Thus, we have $\overline{K_i}=G^{-1}{\beta }_i=R_Y^{-1}{\rho }_{X_{i}Y}$ , and so $K_i^H={\rho }_{X_{i}Y}^T{R}_Y^{-1}$ . Concatenating all the rows of $K$ together, we get $K=R_{X,Y}{R}_Y^{-1}$ , where $R_{X,Y}$ is the cross-correlation matrix for the random vectors $X$ and $Y$ . We therefore obtain the optimal linear estimator $\widehat{X}=KY=R_{X,Y}{R}_Y^{-1}Y$ .

At first, there may be some confusion on the difference between least squares and minimum mean-square error. To summarize:

• Least Squares are applied when the quantities observed are deterministic (i.e., a “single draw” of data or observations).
• Minimum Mean Square Error Estimation are applied when random variables are observed under Gaussian noise; one must know a distribution over inputs, and the error must be measured in expectation.