by Suraj Rampure (suraj.rampure@berkeley.edu)

In discussion for Data 100, I ignored the difference between eigenvalues and singular values. They are actually very different, and we'll look at the formal definitions of both and the relationships between them here.

Eigenvalues

Suppose $A \in \mathbb{M}^{n \times n}$ is an $n \times n$ square matrix, and $\textbf{v} \in \mathbb{R}^n$ is an $n$-element column vector. Then, if we can say

$$A\textbf{v} = \lambda \textbf{v}$$

this implies $\textbf{v}$ is an eigenvector with corresponding eigenvalue $\lambda$.

Note, eigenvectors and eigenvalues exist only when $A$ is a square matrix. Let's take a look at why.

Suppose $A \in \mathbb{M}^{m \times n}$, meaning $A$ is a matrix with $m$ rows and $n$ columns. If we want to multiply $A$ on the right by a column vector $\textbf{v}$, that vector must have $n$ elements in order for the dimensions of $A$ and $\textbf{v}$ to allow multiplication. However, when we multiply an $m \times n$ matrix by a $n \times 1$ vector, the result will be an $m \times 1$ vector.

For example, suppose $A = \begin{bmatrix} 2 & 3 & 4 & 5 & 6 \\ 10 & 2 & -3 & 1 & 0\end{bmatrix}$ and $\textbf{v} = \begin{bmatrix} 1 \\ 2 \\ 0 \\ -1 \\ 3 \end{bmatrix} \in \mathbb{R}^5$. The result of this vector multiplication will be $\begin{bmatrix} 21 \\ 13 \end{bmatrix} \in \mathbb{R}^2$.

The only way that $A\textbf{v}$ can have the same dimensions as $\textbf{v}$ is if $m = n$, i.e. $A$ is square.

Singular Values

To find the singular values of $A$, we first begin by finding the eigenvalues of $A^TA$. If $A \in \mathbb{M}^{m \times n}$, then $A^TA$ will be an $n \times n$ symmetric matrix. Since $A^TA$ is square, it has eigenvalues, and furthermore, all of $A^TA$'s eigenvalues will be non-negative*.

Suppose $\lambda_1, \lambda_2, ..., \lambda_n$ are the $n$ eigenvalues of $A^TA$. The singular values of $A$, then, are $\sigma_1 = \sqrt{\lambda_1}, \sigma_2 = \sqrt{\lambda_2}, ..., \sigma_n = \sqrt{\lambda_n}$. In other words, the singular values of $A$ are the square roots of the eigenvalues of $A^TA$. Notice, since $\lambda_i$ is non-negative, we can always take the square root.

When are they the same?

There is a very special case in which the singular values of a matrix are the same as the eigenvalues of a matrix.

Claim: If $A$ is a symmetric matrix, i.e. $A = A^T$, then the singular values of $A$ are equal to the absolute values of the eigenvalues of $A$. In other words, if $A$ is symmetric, then if $\lambda_1, \lambda_2, ..., \lambda_n$ are the eigenvalues of $A$, then $\sigma_1 = |\lambda_1|, \sigma_2 = |\lambda_2|, ..., \sigma_n = |\lambda_n|$.

Proof: First, we'll show that if $\lambda$ is an eigenvalue of $A$, then $\lambda^2$ is an eigenvalue of $A^2$ for any symmetric matrix $A$.

$$\begin{align*} A\textbf{v} &= \lambda \textbf{v} \\ A^TA\textbf{v} &= A^T \lambda \textbf{v} \\ &= \lambda A^T\textbf{v} \\ &= \lambda (A\textbf{v}) \\ &= \lambda^2 \textbf{v} \end{align*}$$

Now, we've shown that the eigenvalues of $A^TA$ are of the form $\lambda^2$. The singular values of $A$ are simply the square roots of the eigenvalues of $A^TA$, i.e. $\sqrt{\lambda^2}$. $\lambda$ could have originally been negative, so we must say $\sqrt{\lambda^2} = |\lambda|$.

This proves that if $\lambda$ is an eigenvalue of a symmetric matrix $A$, then $|\lambda|$ is a singular value of $A$.

Furthermore, if $A$ is positive semi-definite, meaning it is symmetric AND all of its eigenvalues are non-negative, we can remove the absolute value symbol, and simply state that the eigenvalues and singular values of $A$ overlap.

Summary

In short:

• Eigenvalues are only defined for square matrices, whereas singular values are defined for all matrices
• Even when a matrix is square, there is no direct relationship between its singular values and eigenvalues
• If a matrix is symmetric (also meaning it is square), then its eigenvalues and singular values are closely related by $\sigma = |\lambda|$

For the purposes of our course, this is relevant when looking at PCA. We find the directions in which our data vary the most by determining and ranking the singular values of our data matrix $A$ (the "directions" that we choose are actually the eigenvectors of $A^TA$). All of this, of course, is done after the columns of $A$ are de-meaned.

Look at this note if you're interested in reading more about singular values and the singular value decomposition (SVD).