The long-awaited^[1] sequel to my "Six (and a half) intuitions for KL divergence" is finally here!

Thanks to the following people for feedback: Denizhan Akar, Rudolf Laine, Simon Lermen, Aryan Bhatt, Spencer Becker-Kahn, Arthur Conmy, and anonymous members of my ARENA cohort (-:

If you want to 80-20 this post (and you already know what SVD is), then just read the "Summary" section below.

Summary

The SVD is a matrix decomposition $M = U S V^{T}$ , where U and V are orthogonal, and S is diagonal (with non-negative diagonal elements, which we call singular values). The six intuitions are:

1. Rotations and Scalings (geometric picture)

Orthogonal matrices $U$ and $V$ can be thought of as rotations of our basis, and $S$ as a scaling of our new basis. These operations are easier to geometrically visualise than the entire operation of a matrix, with all its nuances (e.g. things like shear).

2. Best rank- $k$ Approximations

Truncating the SVD sum gives us in some sense the "best way" to approximate $M$ as a sum of rank-1 matrices. In other words, the first few matrices in the SVD sum are the "most important parts of $M$ ".

3. Least Squares Regression

If we're trying to minimize $∥ M x - b ∥$ (either with a constraint on the norm of $x$ or with no constraint), we can express the solution for $x$ in terms of the SVD of $M$ . When $∥ x ∥$ is constrained, it becomes most important to get the components of $x$ right in the directions of the largest singular values of $M$ ; the smaller singular values matter less.

4. Input and Output Directions (like MLPs!)

An MLP can be thought of as a collection of neurons, where each neuron has an input direction (thought of as detecting a feature) and an output direction (outputting a result if that feature is observed). $v_{i}$ and $u_{i}$ are like the input and output directions (minus the nonlinearity in the middle), and furthermore each "neuron" in this case is completely independent (orthogonal).

5. Lost & Preserved Information

SVD tells you what information (if any) is lost when you use $M$ as a linear map. It also tells you which bits of information are easier / harder to recover.

6A. Principal Component Analysis

If $M$ is actually a matrix of data, we can interpret the SVD matrices in terms of that data. $V$ tells you whic

...

a zero

a zero