Undoing a matrix, and why you rarely should

The matrix that does nothing

The identity matrix has 1s down the diagonal and 0s everywhere else:

I = [ 1  0  0 ]
    [ 0  1  0 ]
    [ 0  0  1 ]

Multiply any vector by it and you get the vector back. Multiply any matrix by it and you get that matrix back. It is the matrix equivalent of the number 1, and it exists so that "undoing" has something to mean.

The inverse

The inverse of A, written A^-1, is the matrix that undoes A: A^-1 A = I. If A rotates, A^-1 rotates back. If A stretches by 3, A^-1 shrinks by 3.

For a 2×2 there is a formula worth seeing once, because it exposes the whole story:

A = [ a  b ]      A^-1 =    1     [  d  -b ]
    [ c  d ]              ad - bc [ -c   a ]

The quantity ad − bc is the determinant. When it is zero, the formula divides by zero and no inverse exists. Worked:

A = [ 2  1 ]   det = 2*3 - 1*1 = 5
    [ 1  3 ]
A^-1 = (1/5) [  3  -1 ]  =  [  0.6  -0.2 ]
             [ -1   2 ]     [ -0.2   0.4 ]
check: A A^-1 = [1 0; 0 1]   (multiply it out; it does)

When there is no inverse

A matrix has no inverse when it destroys information — when two different inputs produce the same output. Consider

A = [ 1  2 ]      A [1, 0] = [1, 2]     A [0, 1] = [2, 4]
    [ 2  4 ]

The second row is twice the first, so every output lands on a single line. Given an output you cannot recover which input produced it, because infinitely many did. Such a matrix is called singular, and its determinant is 1*4 − 2*2 = 0.

This matters more than it first appears. A layer whose weight matrix is singular has thrown information away permanently, and no later layer can retrieve it. Real weight matrices are almost never exactly singular, but many are close, which is the practically important case.

Nearly singular: the condition number

The useful measure is the condition number, roughly the ratio of the largest to the smallest singular value. A well-conditioned matrix has a condition number in the tens. A badly conditioned one has it in the millions, and it means a tiny change in the input produces an enormous change in the output — or, in the direction that hurts, that a tiny rounding error in your data becomes an enormous error in the solution.

A rough rule: with 32-bit floats you have about 7 decimal digits of precision, so a condition number of 10^6 leaves you roughly one meaningful digit. That single fact explains most of the "why is my linear solve producing nonsense" problems in practice.

Why you should not compute the inverse

Here is a habit that separates people who have been burnt from people who have not. To solve Ax = b, do not compute A^-1 and multiply.

import numpy as np
x = np.linalg.inv(A) @ b     # slower, less accurate, avoid
x = np.linalg.solve(A, b)    # do this instead

Three reasons. Forming the inverse costs roughly three times the work of solving directly. It is measurably less accurate, because it computes an intermediate object with its own rounding errors before using it. And it is unnecessary: solve uses LU decomposition, which never builds the inverse at all.

The rule generalises. In numerical code, the answer to "how do I undo this matrix" is almost always "solve the system", not "invert the matrix". Library authors know this, which is why np.linalg.solve exists and why the documentation for np.linalg.inv gently steers you away.

Where this shows up in machine learning

Explicit matrix inversion is rare in deep learning, which is worth saying plainly: nothing in transformer training inverts a matrix. It appears in three neighbouring places.

• Linear regression's closed form involves (X^T X)^-1, and this is exactly where the advice bites. When two features are highly correlated, X^T X is close to singular, the condition number explodes, and the fitted coefficients become huge with opposite signs. This is multicollinearity, and adding a ridge penalty — (X^T X + λI)^-1 — fixes it precisely by pushing the smallest singular values away from zero.
• Gaussian processes and Kalman filters invert covariance matrices, and practitioners of both spend real effort on numerical conditioning, usually via Cholesky decomposition rather than an inverse.
• Second-order optimisers would like the inverse Hessian and cannot afford it; every practical method, from L-BFGS to Adam's diagonal approximation, is a way of avoiding that inversion.

The rule to keep

solve, never inv. And when a solve returns absurd numbers, check the condition number before you check anything else — np.linalg.cond(A) is one call, and it usually names the problem immediately.