2.9 Singular value decomposition

Singular value decomposition (SVD) can be thought of as an extension to eigenvalue decomposition for non-symmetric matrices. Consider an $m\times n$ matrix $X$ . The following two matrices are symmetric and so have eigenvalue decompositions

where $X{X}^H$ is an $m\times m$ matrix and $X^{H}X$ is an $n\times n$ matrix. It turns out that we can therefore decompose the matrix $X$ as $X=U\Sigma V^H$ , where $\Sigma$ is an $m\times n$ “diagonal” matrix: ${\Sigma }_{i,i}={\sigma }_i$ are the singular values of $X$ , and ${\Sigma }_{i,j}=0$ for $i\ne j$ . The pseudoinverse of the matrix can then be written as $X^{†}=V{\Sigma }^{†}{U}^H$ , where ${\Sigma }_{i,i}^{†}=1/{\sigma }_i$ and ${\Sigma }_{i,j}^{†}=0$ for $i\ne j$ .

Principal component analysis

Principal component analysis can be thought of as KLT for sampled data. Assume that $\{x_1,x_2,\cdots x_L\}\subseteq {\mathbb{R}}^n$ is a zero-mean dataset, and collect it into a matrix $X=[x_1{x}_2\cdots x_L]\subseteq {\mathbb{R}}^{nxL}$ . Next, compute the SVD $X=U\Sigma V^T$ with the corresponding eigenvalue decomposition $X{X}^T=U\Lambda U^T$ . The matrix $U$ is known as the principal component analysis (PCA) matrix of $X$ ; its columns $U_1,U_2,...U_n$ are known as principal components, and its PCA coefficients are given by $Y=U^{T}X=\Sigma V^T$ . The matrix $Y$ contains the “scores” of all data points in the columns of $X$ against the principal components $U_i$ . One can show that the principal components in the matrix $U$ follow the formulation

In words, $u_j$ is the direction in which the projections of the data has the largest variance while being orthogonal to $\{u_1,u_2,...u_{j-1}\}$ .