1.1 The bayesian paradigm  (Page 2/2)

Note

• $\stackrel{}{A}$ is biased ( ).
• Although $(A)$ is MVUB, $\stackrel{}{A}$ is better in the MSE sense.
• Prior information is aptly described by regarding $A$ as a random variable with a prior distribution $U(-A_0, A_0)$ , which implies that we know $-A_0\le A\le A_0$ , but otherwise $A$ is abitrary.

The bayesian approach to statistical modeling

Elements of bayesian analysis

• (a)

  joint distribution $$p(x,)=p(, x)p()$$

• (b)

  marginal distributions $$p(x)=\int p(, x)p()\,d$$ $$p()=\int p(, x)p()\,d x$$ where $p()$ is a prior .

• (c)

  posterior distribution $$p(x, )=\frac{p(x,)}{p(x)}=\frac{p(, x)p()}{\int p(, x)p()\,d x}$$

Selecting an informative prior

Clearly, the most important objective is to choose the prior $p()$ that best reflects the prior knowledge available to us. In general, however, our prior knowledge is imprecise andany number of prior densities may aptly capture this information. Moreover, usually the optimal estimator can't beobtained in closed-form.

Therefore, sometimes it is desirable to choose a prior density that models prior knowledge and is nicely matched in functional form to $p(, x)$ so that the optimal esitmator (and posterior density) can be expressed in a simple fashion.

Choosing a prior

1. informative priors

• design/choose priors that are compatible with prior knowledge of unknown parameters

2. non-informative priors

• attempt to remove subjectiveness from Bayesian procedures
• designs are often based on invariance arguments

Conjugate priors

Idea

Given $p(, x)$ , choose $p()$ so that $(p(x, ), p(, x)p())$ has a simple functional form.

Conjugate priors

Choose $p()\in$ , where $$ is a family of densities ( e.g. , Gaussian family) so that the posterior density also belongsto that family.

conjugate prior

$p()$ is a conjugate prior for $p(, x)$ if $p()\in \implies p(x, )\in$

The Gaussian prior is also conjugate to the Gaussian likelihood $$p(A, x)=\frac{1}{(2\pi ^2)^{\left(\frac{N}{2}\right)}}e^{\frac{-1}{2^2}\sum_{n=1}^N (x_n-A)^2}$$ so that the resulting posterior density is also a simple Gaussian, as shown next.

First note that $$p(A, x)=\frac{1}{(2\pi ^2)^{\left(\frac{N}{2}\right)}}e^{\frac{-1}{2^2}\sum_{n=1}^N x_n}e^{\frac{-1}{2^2}(NA^2-2NA\stackrel{-}{x})}$$ where $\stackrel{-}{x}=\frac{1}{N}\sum_{n=1}^N x_n$ .

Interpretation

• When there is little data $({}_A^2, \frac{^2}{N})$ $$ is small and $(A)=$ .
• When there is a lot of data $({}_A^2, \frac{^2}{N})$ , $\approx 1$ and $(A)=\stackrel{-}{x}$ .

Interplay between data and prior knowledge

Small $N\to (A)$ favors prior.

Large $N\to (A)$ favors data.

The multivariate gaussian model

The multivariate Gaussian model is the most important Bayesian tool in signal processing. It leads directly tothe celebrated Wiener and Kalman filters.

Assume that we are dealing with random vectors $x$ and $y$ . We will regard $y$ as a signal vector that is to be estimated from an observation vector $x$ .

$y$ plays the same role as $$ did in earlier discussions. We will assume that $y$ is p1 and $x$ is N1. Furthermore, assume that $x$ and $y$ are jointly Gaussian distributed $$(\begin{pmatrix}x\\ y\\ \end{pmatrix}, (\begin{pmatrix}0\\ 0\\ \end{pmatrix}, \begin{pmatrix}{R}_{\mathrm{xx}} & R_{\mathrm{xy}}\\ R_{\mathrm{yx}} & R_{\mathrm{yy}}\\ \end{pmatrix}))$$ $(x)=0$ , $(y)=0$ , $(xx^T)=R_{\mathrm{xx}}$ , $(xy^T)=R_{\mathrm{xy}}$ , $(yx^T)=R_{\mathrm{yx}}$ , $(yy^T)=R_{\mathrm{yy}}$ . $$R\equiv \begin{pmatrix}{R}_{\mathrm{xx}} & R_{\mathrm{xy}}\\ R_{\mathrm{yx}} & R_{\mathrm{yy}}\\ \end{pmatrix}$$