Introduction to concise signal models

Overview

In characterizing a given problem in signal processing, one is often able to specify a model for the signals to be processed. This model may distinguish (either statistically or deterministically)classes of interesting signals from uninteresting ones, typical signals from anomalies, information from noise, etc.

Very commonly, models in signal processing deal with some notion of structure, constraint, or conciseness. Roughly speaking, one often believes that a signal has “few degrees of freedom”relative to the size of the signal. This notion of conciseness is a very powerful assumption, and it suggests the potential for dramatic gains via algorithms that capture and exploit the true underlyingstructure of the signal.

In these modules, we survey three common examples of concise models: linear models, sparse nonlinear models, and manifold-based models. In each case, we discuss an important phenomenon:the conciseness of the model corresponds to a low-dimensional geometric structure along which the signals of interest tend to cluster. This low-dimensional geometry again has important implicationsin the understanding and the development of efficient algorithms for signal processing.

We discuss this low-dimensional geometry in several contexts, including projecting a signal onto the model class (i.e., forming a concise approximation to a signal), encoding such an approximation(i.e., data compression), and reducing the dimensionality of signals and data sets. We conclude with an important and emerging application area known as Compressed Sensing (CS), which is a novel methodfor data acquisition that relies on concise models and builds upon strong geometric principles. We discuss CS in its traditional, sparsity-based context and also discuss extensions of CS to otherconcise models such as manifolds.

General mathematical preliminaries

Signal notation

We will treat signals as real- or complex-valued functions having domains that are either discrete (and finite) or continuous (andeither compact or infinite). Each of these assumptions will be made clear as needed. As a generalrule, however, we will use $x$ to denote a discrete signal in ${\mathbb{R}}^N$ and $f$ to denote a function over a continuousdomain $\mathcal{D}$ . We also commonly refer to these as discrete- or continuous- time signals, though the domain need not actually be temporal in nature.

Lp and lp norms

As measures for signal energy, fidelity, or sparsity, we will employ the $L_p$ and ${\ell }_p$ norms. For continuous-time functions, the $L_p$ norm is defined as

Linear algebra

Let $A$ be a real-valued $M\times N$ matrix. We denote the nullspace of $A$ as $\mathcal{N}\left(A\right)$ (note that $\mathcal{N}\left(A\right)$ is a linear subspace of ${\mathbb{R}}^N$ ), and we denote the transpose of $A$ as $A{}^T$ .

We call $A$ an orthoprojector from ${\mathbb{R}}^N$ to ${\mathbb{R}}^M$ if it has orthonormal rows. From such a matrix we call $A^{{}^T}A$ the corresponding orthogonal projection operator onto the $M$ -dimensional subspace of ${\mathbb{R}}^N$ spanned by the rows of $A$ .