1.10 Signal approximation in a hilbert space

We will now revisit “The Fundamental Theorem of Approximation” for the extremely important case where our set $A$ is a subspace. Specifically, suppose that $H$ is a Hilbert space, and let $A$ be a (closed) subspace of $H$ . From before, we have that for any $x\in H$ there is a unique $\widehat{x}\in A$ such that $\widehat{x}$ is the closest point in $A$ to $x$ . When $A$ is also a subspace, we also have:

The orthogonality principle

$\widehat{x}\in A$ is the minimizer of $∥x,-,\widehat{x}∥$ if any only if $\widehat{x}-x\perp A$ i.e., $〈\widehat{x},-,x,,,y〉=0$ for all $y\in A$ .

1. Suppose that $\widehat{x}-x\perp A$ . Then for any $y\in A$ with $y\ne \widehat{x},$ ${∥y,-,x∥}^2={∥y,-,\widehat{x},+,\widehat{x},-,x∥}^2$ . Note that $y-\widehat{x}\in A$ , but $\widehat{x}-x\perp A$ , so that $〈y,-,\widehat{x},,,\widehat{x},-,x〉=0$ , and we can apply Pythagoras to obtain ${∥y,-,x∥}^2={∥y,-,\widehat{x}∥}^2+∥\widehat{x},-,x∥$ . Since $y\ne \widehat{x}$ , we thus have that ${∥y,-,x∥}^2>{∥\widehat{x},-,x∥}^2$ . Thus $\widehat{x}$ must be the closest point in $A$ to $x$ .

   

2. Suppose that $\widehat{x}$ minimizes $∥x,-,\widehat{x}∥$ . Suppose for the sake of a contradiction that $\exists y\in A$ such that $∥y∥=1$ and $〈x,-,\widehat{x},,,y〉=\delta \ne 0$ .

Let $z=\widehat{x}+\delta y$ .

Thus $∥x,-,z∥\le ∥x,-,\widehat{x}∥$ , contradicting the assumption that $\widehat{x}$ minimizes $∥x,-,\widehat{x}∥.$

This result suggests a that a possible method for finding the bestapproximation to a signal $x$ from a vector space $V$ is to simply look for a vector $\widehat{x}$ such that $\widehat{x}-x\perp V$ . In the coming lectures we will show how to do this, but it will require a brief review of some concepts from linear algebra.