4.3 Instance-optimal guarantees revisited

We now briefly return to the noise-free setting to take a closer look at instance-optimal guarantees for recovering non-sparse signals. To begin, recall that in Theorem 1 from "Noise-free signal recovery" we bounded the ${\ell }_2$ -norm of the reconstruction error of

as

when $\mathcal{B}\left(y\right)=\{z:\Phi z=y\}$ . One can generalize this result to measure the reconstruction error using the ${\ell }_p$ -norm for any $p\in [1,2]$ . For example, by a slight modification of these arguments, one can also show that ${∥\widehat{x},-,x∥}_1\le C_0{\sigma }_K{\left(x\right)}_1$ (see  [link] ). This leads us to ask whether we might replace the bound for the ${\ell }_2$ error with a result of the form ${∥\widehat{x},-,x∥}_2\le C{\sigma }_K{\left(x\right)}_2$ . Unfortunately, obtaining such a result requires an unreasonably large number of measurements, as quantified by the following theorem of  [link] .

(theorem 5.1 of [link] )

Suppose that $\Phi$ is an $M\times N$ matrix and that $\Delta :{\mathbb{R}}^M\to {\mathbb{R}}^N$ is a recovery algorithm that satisfies

for some $K\ge 1$ , then $M>\left(1,-,\sqrt{1-1/C^2}\right)N$ .

We begin by letting $h\in {\mathbb{R}}^N$ denote any vector in $\mathcal{N}\left(\Phi \right)$ . We write $h=h_{\Lambda }+h_{{\Lambda }^c}$ where $\Lambda$ is an arbitrary set of indices satisfying $\left|\Lambda \right|\le K$ . Set $x=h_{{\Lambda }^c}$ , and note that $\Phi x=\Phi h_{{\Lambda }^c}=\Phi h-\Phi h_{\Lambda }=-\Phi h_{\Lambda }$ since $h\in \mathcal{N}\left(\Phi \right)$ . Since $h_{\Lambda }\in {\Sigma }_K$ , [link] implies that $\Delta \left(\Phi x\right)=\Delta (-\Phi h_{\Lambda })=-h_{\Lambda }$ . Hence, ${∥x,-,\Delta ,\left(,\Phi ,x,\right)∥}_2={∥h_{{\Lambda }^c},-,(-h_{\Lambda })∥}_2={∥h∥}_2$ . Furthermore, we observe that ${\sigma }_K{\left(x\right)}_2\le {∥x∥}_2$ , since by definition ${\sigma }_K{\left(x\right)}_2\le {∥x,-,\tilde{x}∥}_2$ for all $\tilde{x}\in {\Sigma }_K$ , including $\tilde{x}=0$ . Thus ${∥h∥}_2\le C{∥h_{{\Lambda }^c}∥}_2$ . Since ${∥h∥}_2^2={∥h_{\Lambda }∥}_2^2+{∥h_{{\Lambda }^c}∥}_2^2$ , this yields

This must hold for any vector $h\in \mathcal{N}\left(\Phi \right)$ and for any set of indices $\Lambda$ such that $\left|\Lambda \right|\le K$ . In particular, let ${\left\{v_i\right\}}_{i=1}^{N-M}$ be an orthonormal basis for $\mathcal{N}\left(\Phi \right)$ , and define the vectors ${\left\{h_i\right\}}_{i=1}^N$ as follows:

We note that $h_j={\sum }_{i=1}^{N-M}⟨e_j,v_i⟩v_i$ where $e_j$ denotes the vector of all zeros except for a 1 in the $j$ -th entry. Thus we see that $h_j=P_{\mathcal{N}}{e}_j$ where $P_{\mathcal{N}}$ denotes an orthogonal projection onto $\mathcal{N}\left(\Phi \right)$ . Since ${∥P_{\mathcal{N}},e_j∥}_2^2+{∥P_{\mathcal{N}}^{\perp },e_j∥}_2^2={∥e_j∥}_2^2=1$ , we have that ${∥h_j∥}_2\le 1$ . Thus, by setting $\Lambda =\left\{j\right\}$ for $h_j$ we observe that

Summing over $j=1,2,...,N$ , we obtain

and thus $M\ge \left(1,-,\sqrt{1-1/C^2}\right)N$ as desired.

Thus, if we want a bound of the form [link] that holds for all signals $x$ with a constant $C\approx 1$ , then regardless of what recovery algorithm we use we will need to take $M\approx N$ measurements. However, in a sense this result is overly pessimistic, and we will now see that the results we just established for signal recovery in noise can actually allow us to overcome this limitation by essentially treating the approximation error as noise.

Towards this end, notice that all the results concerning ${\ell }_1$ minimization stated thus far are deterministic instance-optimal guarantees that apply simultaneously to all $x$ given any matrix that satisfies the restricted isometry property (RIP). This is an important theoretical property, but as noted in "Matrices that satisfy the RIP" , in practice it is very difficult to obtain a deterministic guarantee that the matrix $\Phi$ satisfies the RIP. In particular, constructions that rely on randomness are only known to satisfy the RIP with high probability. As an example, recall Theorem 1 from "Matrices that satisfy the RIP" , which opens the door to slightly weaker results that hold only with high probability.