Singular values, and the honest measure of importance

The factorisation that always exists

Eigenvectors need a square matrix and still sometimes fail. The singular value decomposition has no such conditions. Every matrix — rectangular, singular, whatever — can be written as

A = U S V^T

where U and V are orthogonal (rotations, possibly with a reflection) and S is diagonal with non-negative entries down it. Those diagonal entries are the singular values, conventionally listed largest first.

The geometric reading is clean and worth memorising: any linear transformation, however complicated it looks, is rotate, stretch along the new axes, rotate again. That is all a matrix can do. The singular values are the stretch factors.

For an m × n matrix, there are min(m, n) singular values. The number of non-zero ones is the rank. The largest divided by the smallest is the condition number. Three concepts from earlier lessons all fall out of one decomposition.

Reading the decay

The practically useful thing is not the decomposition but the list of singular values, because it tells you how much of the matrix lives in how few directions.

import numpy as np
s = np.linalg.svd(W, compute_uv=False)
energy = np.cumsum(s**2) / np.sum(s**2)
print("dims for 90% of the energy:", np.searchsorted(energy, 0.90) + 1)

Squared singular values sum to the total squared magnitude of the matrix, so the cumulative fraction tells you what proportion you keep. If 90 per cent sits in the first 50 of 768 directions, the matrix is close to rank 50 and you can compress it hard. If you need 700 directions to reach 90 per cent, you cannot, and no amount of tuning will change that.

This is the check to run before attempting low-rank compression, not after it disappoints.

Truncation, and the guarantee behind it

Keep only the largest k singular values, zero the rest, and multiply back. The result is the best possible rank-k approximation of the original matrix, in the sense of squared error. This is the Eckart–Young theorem, and it is unusual in being both a real theorem and directly practical: no cleverer rank-k approximation exists, so if truncated SVD is not good enough, low-rank approximation is not good enough.

The error is exactly computable in advance:

squared error of rank-k truncation = sum of the squares of the discarded singular values

You know what compression will cost before you compress. That is rare, and it is why SVD is the right first tool.

Worked, in storage terms. A 4096 × 4096 matrix truncated to rank 256:

original:  4096 * 4096       = 16,777,216 numbers
rank 256:  4096*256 + 256 + 256*4096 = 2,097,408 numbers
saving:    8x, and the error is the tail of the singular values

Where SVD earns its place

• PCA is SVD. Centre your data matrix and take its SVD; the right singular vectors are the principal components and the squared singular values are proportional to the variance explained. Any implementation of PCA you will use calls an SVD routine underneath, because doing it that way avoids forming the covariance matrix and is numerically better.
• Latent semantic analysis, the 1990s ancestor of embeddings, is a truncated SVD of a term–document matrix. The idea that meaning lives in a low-dimensional space of a big sparse count matrix is thirty years older than the transformer.
• Least squares, robustly. np.linalg.lstsq uses SVD, which is why it returns a sensible answer even when the closed form would divide by something close to zero. The pseudoinverse is defined by inverting only the non-negligible singular values.
• Recommenders. Matrix factorisation for ratings is SVD's idea, adapted because the actual matrix is mostly missing and plain SVD needs a complete one.
• Diagnosing a trained model. The singular value spectrum of a layer tells you whether that layer is using its full width. A rapidly decaying spectrum is a layer that could be smaller.

The cost, and how to avoid paying it

A full SVD of an n × n matrix costs about O(n^3). For 4096 that is roughly 7 × 10^10 operations — seconds on a laptop, not free but affordable. For a 50,000 × 768 embedding matrix, a full SVD is wasteful when you only want the top 100 components.

The answer is a truncated or randomised SVD, which computes only the leading components and costs a fraction. sklearn.decomposition.TruncatedSVD and scipy.sparse.linalg.svds both do this and both work on sparse matrices, which full SVD cannot. All free, all CPU.

The limitation to state plainly

SVD finds the best linear low-dimensional summary. If your data lies on a curved surface — a spiral, a sphere, most real embedding manifolds — the best linear approximation can be poor while a non-linear method does much better. That is why UMAP and t-SNE exist for visualisation and why autoencoders exist for compression.

There is a second, quieter limitation. SVD optimises squared reconstruction error, which is a statement about fidelity to the numbers, not about usefulness for your task. A direction with a small singular value can still be the one carrying the signal your classifier needs. Compress by SVD, then measure your actual task metric; do not assume that keeping 95 per cent of the energy keeps 95 per cent of the performance.

The rule to keep

Look at the singular values before you decide anything about a matrix's size. They tell you the rank, the conditioning, and exactly what compression will cost, and they are one function call away.