Eigenvalues and Diagonalization

Plot size against rent for a few hundred flats and the dots form a long cloud tilted up and to the right. The direction the cloud stretches along, and how far it stretches, are an eigenvector and an eigenvalue. Principal component analysis, a standard way to shrink many features into a few, is built on exactly this.

The rule

An eigenvector keeps its direction; its eigenvalue says how much it stretches.

Most vectors change direction when a matrix acts on them. A few special ones only get longer, shorter or flipped. For those, $Av = \lambda v$: $v$ is the eigenvector and the number $\lambda$ is its eigenvalue.

Take $A = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$:

• $v = [1, 1]$: $Av = [3, 3] = 3v$. Same direction, stretched by 3, so $\lambda = 3$.
• $v = [1, -1]$: $Av = [1, -1]$. Same direction, stretched by 1, so $\lambda = 1$.
• $v = [1, 0]$: $Av = [2, 1]$. The direction changed, so this is not an eigenvector.

Diagonalization rewrites the matrix using its eigenvectors: $A = P D P^{-1}$, where the columns of $P$ are the eigenvectors and $D$ is diagonal with the eigenvalues. In eigenvector coordinates the matrix is just "stretch each axis". That makes powers easy: $A^{10} = P D^{10} P^{-1}$, and a diagonal matrix is raised to a power one entry at a time.

Covariance matrices, which describe how features vary together, are symmetric. Symmetric matrices always have real eigenvalues and perpendicular eigenvectors, which is why PCA can use them as new axes: the eigenvector with the largest eigenvalue points where the data spreads most.

The shortcut

For a 2 × 2 matrix: the eigenvalues add up to the trace and multiply to the determinant. The trace is the sum of the diagonal. Find two numbers with that sum and product.

For a diagonal or triangular matrix it is even faster: the eigenvalues are the diagonal entries.

Drills

Answer fast — trust the rule.

Remember

• An eigenvector keeps its direction; its eigenvalue says how much it stretches.
• Shortcut for 2 × 2: eigenvalues add to the trace and multiply to the determinant.
• Trap: assuming any vector is an eigenvector; check that the direction really stays put.