Singular Value Decomposition
Every matrix, however ugly, is a rotation then a stretch then another rotation. Step through the three and watch it happen.
Overview
What every matrix does
Take the unit circle and apply any 2×2 matrix. The result is always an ellipse. Not sometimes — always, for every matrix there is.
That is a strong claim, and the SVD is the reason it holds. Every matrix can be written as three operations in sequence:
A = U Sigma V'Read right to left, which is the order they apply:
V' rotates. A rotation does nothing to a circle, which is why stage 2 in the visualisation looks identical to stage 1 — but watch the two spokes, which do move. It is choosing which directions are about to be stretched.
Sigma stretches along the axes, by the singular values. This is where the circle becomes an ellipse.
U rotates the ellipse into its final orientation.
Step through the four stages above and the claim stops being abstract. Whatever you set the four matrix entries to, the shape at stage 3 is an axis-aligned ellipse, every time.
Singular Value Decomposition
Worth knowing
A = U Σ V': rotate, stretch along the axes, rotate again.Singular Value Decomposition
The factorisation that works on every matrix, and the one worth knowing if you only learn one.
The singular values
They are the lengths of the ellipse's semi-axes, always non-negative, and conventionally listed largest first.
The largest is how much the matrix can stretch any vector — its gain in the worst case.
The smallest is how much it can squash one.
Their ratio is the condition number, and it is the single most useful number in the readout. A condition number near 1 means the matrix treats all directions roughly alike. A large one means it nearly flattens some direction, and solving anything involving that matrix will amplify error along it. Drag the entries until the two singular values are far apart and watch the ellipse become a sliver.
A zero singular value means the matrix genuinely flattens a direction: the ellipse degenerates to a line segment, the matrix is singular, and it has no inverse.
Why not eigenvectors
The [eigendecomposition](eigenvalues_and_eigenvectors.html) is the more famous factorisation, and the SVD is the more useful one, for three reasons.
It always exists. Eigendecomposition requires a square matrix, and even then not every square matrix has one. The SVD exists for every matrix of every shape, including a 1000×3.
Its bases are orthonormal. Eigenvectors need not be perpendicular, and when they are nearly parallel the decomposition is numerically fragile. U and V are [orthonormal](basis_span_and_orthogonality.html) by construction.
Its values are real and non-negative. Eigenvalues of a real matrix can be complex. Singular values never are.
For a symmetric positive-definite matrix the two coincide, which is why they are so often confused.
Low-rank approximation
The property that makes the SVD indispensable: keep only the largest *k* singular values, set the rest to zero, and multiply back. The result is provably the best rank-*k* approximation of the original matrix, in the least-squares sense. Not a good one — the best.
That single fact underwrites a surprising amount:
PCA is the SVD of the mean-centred data matrix. The principal components are the right singular vectors and the explained variances are the squared singular values, which is why any PCA implementation you look inside is calling an SVD.
Image compression keeps the top singular values of the pixel matrix.
Latent semantic analysis factors a term-document matrix and reads the retained dimensions as topics.
Recommender systems factor a sparse user-item matrix into a low-rank product.
Noise reduction works on the assumption that signal concentrates in the large singular values and noise spreads through the small ones.
Singular values by hand, from A'A
You rarely compute an SVD on paper, but seeing where the singular values come from removes the mystery. The singular values of A are the square roots of the eigenvalues of A'A. Take A = [[3, 0], [4, 5]]:
A'A = [[9 + 16, 20], [20, 25]] = [[25, 20], [20, 25]].
A symmetric 2×2 [[a, b], [b, a]] has eigenvalues a + b and a − b, here 45 and 5. So the singular values are:
| singular value | from |
|---|---|
| σ₁ = √45 ≈ 6.708 | the largest stretch A applies |
| σ₂ = √5 ≈ 2.236 | the smallest |
Their ratio, √(45/5) = 3, is the condition number: A stretches its most sensitive direction three times more than its least. That is why the unit circle becomes an ellipse with semi-axes 6.708 and 2.236, and why the singular values are always real and non-negative — they are square roots of eigenvalues of A'A, which is symmetric positive semi-definite for any A. Eigenvalues of A itself could be complex or negative; the SVD's numbers never are, which is half the reason it is the factorisation to reach for.
This is the machinery under PCA, which is nothing more than the SVD of the mean-centred data: the principal directions are the right singular vectors and the variance each explains is its singular value squared. Keeping only the largest few — setting the small singular values to zero and multiplying back — gives the provably best low-rank approximation of the data, which is the same operation behind image compression, latent-semantic analysis and recommender factorisations. The small singular values are where noise concentrates, so dropping them often sharpens the signal rather than degrading it.
Where it goes wrong
Forgetting to centre before PCA. Without subtracting the mean, the first component points at the mean rather than at the direction of greatest variance.
Reading a condition number as an error. It is a sensitivity: it says how much input error can be amplified, not how much has been.
Assuming singular values are eigenvalues. They are the eigenvalues of A'A, square-rooted. Not the same object, and not equal except in special cases.
Computing the full SVD when you want the top few. For a large sparse matrix a truncated solver is orders of magnitude cheaper.
Rebuild a matrix from its biggest pieces
SVD splits any matrix into a stack of rank-one layers ordered by importance. Keeping only the first few is what compression and PCA actually do.
Check yourself
0 of …Answer without scrolling back up.
What shape does the unit circle become under any 2x2 matrix?
What does a large condition number tell you?
Why is the SVD preferred over an eigendecomposition?
Singular Value Decomposition
V' rotates. A rotation does nothing to a circle, which is why stage 2 in the visualisation looks identical to stage 1 — but watch the two spokes, which do move. It is choosing which directions are about to be stretched.