12.2 Linear regression  (Page 2/2)

Distribution of Example 5 From "random vectors and matlab" and Example 12 From "function of random vectors"

The pair $\{X,Y\}$ has joint density ${\displaystyle f_{XY}(t,u)=\frac{6}{37}(t+2u)}$ on the region $0\le t\le 2$ , $0\le u\le max\{1,t\}$ (see Figure [link] ). Determine the regression line of Y on X . If the value $X\left(\omega \right)=1.7$ is observed, what is the best mean-square linear estimate of $Y\left(\omega \right)$ ?

ANALYTIC SOLUTION

$$E\left[X\right]=\frac{6}{37}{\int }_0^1{\int }_0^1(t^2+2tu)dudt+\frac{6}{37}{\int }_1^2{\int }_0^t(t^2+2tu)dudt=50/37$$

The other quantities involve integrals over the same regions with appropriate integrands, as follows:

Quantity	Integrand	Value
$E\left[X^2\right]$	$t^3+2t^{2}u$	779/370
$E\left[Y\right]$	$tu+2u^2$	127/148
$E\left[XY\right]$	$t^{2}u+2t{u}^2$	232/185

Then

$$\mathrm{Var}\left[X\right]=\frac{779}{370}-{\left(\frac{50}{37}\right)}^2=\frac{3823}{13690}\mathrm{Cov}[X,Y]=\frac{232}{185}-\frac{50}{37}·\frac{127}{148}=\frac{1293}{13690}$$

and

$$a=\mathrm{Cov}[X,Y]/\mathrm{Var}\left[X\right]=\frac{1293}{3823}\approx 0.3382,b=E\left[Y\right]-aE\left[X\right]=\frac{6133}{15292}\approx 0.4011$$

The regression line is $u=at+b$ . If $X\left(\omega \right)=1.7$ , the best linear estimate (in the mean square sense) is $\widehat{Y}\left(\omega \right)=1.7a+b=0.9760$ (see [link] for an approximate plot).

APPROXIMATION

tuappr Enter matrix [a b]of X-range endpoints [0 2] Enter matrix [c d]of Y-range endpoints [0 2] Enter number of X approximation points 400Enter number of Y approximation points 400 Enter expression for joint density (6/37)*(t+2*u).*(u<=max(t,1)) Use array operations on X, Y, PX, PY, t, u, and PEX = total(t.*P) EX = 1.3517 % Theoretical = 1.3514EY = total(u.*P) EY = 0.8594 % Theoretical = 0.8581VX = total(t.^2.*P) - EX^2 VX = 0.2790 % Theoretical = 0.2793CV = total(t.*u.*P) - EX*EY CV = 0.0947 % Theoretical = 0.0944a = CV/VX a = 0.3394 % Theoretical = 0.3382b = EY - a*EX b = 0.4006 % Theoretical = 0.4011y = 1.7*a + b y = 0.9776 % Theoretical = 0.9760 Got questions? Get instant answers now!

An interpretation of ρ ²

The analysis above shows the minimum mean squared error is given by

If $\rho =0$ , then $E\left[{(Y-\widehat{Y})}^2\right]={\sigma }_Y^2$ , the mean squared error in the case of zero linear correlation. Then, ρ ² is interpreted as the fraction of uncertainty removed by the linear rule and X . This interpretation should not be pushed too far, but is a common interpretation, often found in the discussion of observations orexperimental results.

More general linear regression

Consider a jointly distributed class. $\{Y,X_1,X_2,\cdots ,X_n\}$ . We wish to deterimine a function U of the form

If U satisfies this minimum condition, then ${\displaystyle E\left[\right(Y-U\left)V\right]=0}$ , or, equivalently

To see this, set $W=Y-U$ and let $d^2=E\left[W^2\right]$ . Now, for any α

If we select the special

This implies $E{\left[WV\right]}^2\le 0$ , which can only be satisfied by $E\left[WV\right]=0$ , so that

On the other hand, if $E\left[\right(Y-U\left)V\right]=0$ for all V of the form above, then $E\left[{(Y-U)}^2\right]$ is a minimum. Consider

Since $U-V$ is of the same form as V , the last term is zero. The first term is fixed. The second term is nonnegative, with zero value iff $U-V=0\mathrm{a}.\mathrm{s}.$ Hence, $E\left[{(Y-V)}^2\right]$ is a minimum when $V=U$ .

If we take V to be $1,X_1,X_2,\cdots ,X_n$ , successively, we obtain $n+1$ linear equations in the $n+1$ unknowns $a_0,a_1,\cdots ,a_n$ , as follows.

1. $E\left[Y\right]=a_0+a_{1}E\left[X_1\right]+\cdots +a_{n}E\left[X_n\right]$
2. $E\left[Y{X}_i\right]=a_{0}E\left[X_i\right]+a_{1}E\left[X_1{X}_i\right]+\cdots +a_{n}E\left[X_n{X}_i\right]\text{for}1\le i\le n$

For each $i=1,2,\cdots ,n$ , we take $\left(2\right)-E\left[X_i\right]·\left(1\right)$ and use the calculating expressions for variance and covariance to get

These n equations plus equation (1) may be solved alagebraically for the a _i .

In the important special case that the X _i are uncorrelated (i.e., $\mathrm{Cov}[X_i,X_j]=0$ for $i\ne j$ ), we have

and

In particular, this condition holds if the class $\{X_i:1\le i\le n\}$ is iid as in the case of a simple random sample (see the section on "Simple Random Samples and Statistics ).

Examination shows that for $n=1$ , with $X_1=X$ , $a_0=b$ , and $a_1=a$ , the result agrees with that obtained in the treatment of the regression line, above.