2.7 Eigendecomposition of linear operators

Definition 1 A scalar $\lambda$ is an eigenvalue of $A\in B(X,X)$ if there exists a vector $e$ (dubbed the eigenvector for $\lambda$ ) such that $Ae=\lambda e$ .

Multiples of eigenvectors are also eigenvectors, as shown below:

Definition 2 The eigenspace of A corresponding to $\lambda$ is defined by ${ϵ}_{\lambda }=\{e\in X:Ae=\lambda e\}$ .

For example, if a given eigenvalue $\lambda$ has two eigenvectors, the eigenspace is given by $\left[\{e_1,e_2\}\right]$ where $A{e}_1=\lambda e_1$ and $A{e}_2=\lambda e_2$ .

Definition 3 An operator $A\in B(X,X)$ is said to be self-adjoint if $A^{*}=A$ , i.e., $⟨Ax,y⟩=⟨x,Ay⟩$ for all $x,y\in X$ .

If $X={\mathbb{R}}^N$ , a self-adjoint operator corresponds to a symmetric matrix.

Theorem 1 All eigenvalues of a self-adjoint operator are real.

Let $\lambda$ be a complex eigenvalue of a self-adjoint operator $A$ . Then $Ae=\lambda e$ for some $e$ . For such an $e$ , we have

Therefore we know that lambda is the same as its complex conjugate ( $\lambda =\overline{\lambda }$ ). The only way for this to be possible is if the imaginary part of $\lambda$ is zero, and therefore $\lambda \in \mathbb{R}$ .

Theorem 2 If ${\lambda }_1,{\lambda }_2$ are distinct eigenvalues of a self-adjoint operator $A\in B(X,X)$ , then ${ϵ}_{{\lambda }_1}\perp {ϵ}_{{\lambda }_2}$ .

Assume we pick some arbitrary $e_1\in {ϵ}_{{\lambda }_1}$ and $e_2\in {ϵ}_{{\lambda }_2}$ . Then,

Therefore we know that ${\lambda }_1⟨e_1,e_2⟩={\lambda }_2⟨e_1,e_2⟩$ . Since we chose two distinct eigenvalues, we know that ${\lambda }_1\ne {\lambda }_2$ . Therefore we must have $⟨e_1,e_2⟩=0$ . This implies $e_1\perp e_2$ . Since $e_1$ and $e_2$ were chosen arbitrarily, this implies ${ϵ}_{{\lambda }_1}\perp {ϵ}_{{\lambda }_2}$ .

Theorem 3 (Schur's Lemma) Let $X$ be an N-dimensional Hilbert space and $A\in B(X,X)$ . Then there exists an orthonormal basis $\{{\phi }_1,{\phi }_2,...,{\phi }_N\}$ for $X$ and a set of coefficients ${\left\{A_{ij}\right\}}_{i,j=1}^N$ such that $A{\phi }_j=\sum _{i=1}^j{A}_{ij}{\phi }_i$ for $j=1,2,...N$ .

An important note: since $\left\{{\phi }_j\right\}$ makes an orthonormal basis for the domain $X$ and range $X$ , there exist coefficients $a_i$ such that $A{\phi }_j=\sum _{i=1}^N{a}_i{\phi }_i$ for $j=1,2,....N$ .

Example 1 Recall the matrix representation of an operator: given $A\in B(X,X)$ and an orthonormal basis ${\left\{{\phi }_i\right\}}_{i=1}^N$ for $X$ , we can write the matrix representation $\tilde{A}$ of the operator $A$ with entries

We can represent $\tilde{A}$ as an upper-triangular matrix (where $x$ represents a non-zero entry in the matrix):

This shows that there exists an orthonormal basis $\phi$ for which the matrix representation of $A$ is an upper-triangular matrix.

Theorem 4 If $X$ is an $N$ -dimensional space and $A\in B(X,X)$ , then there exists an orthonormal basis ${\left\{{\phi }_i\right\}}_{i=1}^N$ such that $A{\phi }_i={\lambda }_i{\phi }_i$ for a set of scalars ${\lambda }_1,{\lambda }_2,...{\lambda }_N$ . The matrix representation $\tilde{A}$ is a diagonal matrix whose entries are the eigenvalues; that is, $\tilde{A}=\mathrm{diag}\left(\left\{{\lambda }_i\right\}\right)$ .

Note that, according to the theorem, we can fully represent the operator $A$ by the aforementioned orthonormal basis $\left\{{\phi }_i\right\}$ and the diagonal matrix $\tilde{A}$ .

Pick the orthonormal basis $\{{\phi }_1,{\phi }_2,...{\phi }_N\}$ specified by Schur's Lemma. For $i<j$ , we have:

For $k<j$ , the term in the sum is equal to zero. We then have:

Thus, the only non-zero entries of the representation matrix $\tilde{A}$ are the diagonal entries. Furthermore, these entries are eigenvalues of $A$ .

Recall that to compute $Ax$ using its matrix representation $\tilde{A}$ , there are three steps:

1. Represent $x$ using the orthonormal basis $\left\{{\phi }_i\right\}$ and collect the coefficients into a vector $c$ .
2. Perform the matrix-vector product $d=\tilde{A}c$ .
3. Then obtain $Ax=b=\sum _{i=1}^N{d}_i{\phi }_i$ .

A matrix-vector product with a dense matrix is computationally intensive. If we can diagonalize $A$ , we can find the matrix-vector product $Ax$ by performing the operation $\tilde{A}c$ , where $\tilde{A}$ is a diagonal matrix, making the operation more efficient.