Projection, and the art of removing a direction

Splitting a vector in two

Take a vector a and a direction b. Every vector a can be written as the sum of two pieces: one lying along b, and one perpendicular to it. The first piece is the projection of a onto b, and it is one formula:

proj_b(a) = ( (a . b) / (b . b) ) * b

If b is already a unit vector, b . b = 1 and this simplifies to (a . b) * b. The scalar a . b is how much of b there is in a, and multiplying by b turns that scalar back into a vector.

Worked, with a = [3, 4] and b = [1, 0]:

a . b = 3
b . b = 1
proj_b(a) = 3 * [1, 0] = [3, 0]
the leftover: a - proj_b(a) = [0, 4]

The leftover, [0, 4], is perpendicular to b: their dot product is 0*1 + 4*0 = 0. That is the definition of orthogonal — a dot product of exactly zero, meaning the two directions carry no information about each other under this measure.

Where projection is already happening

Every row of a weight matrix is a direction, and multiplying by that row computes the scalar part of the projection: how much of that direction is present in the input. A linear layer with 768 inputs and 3,072 outputs is asking 3,072 projection questions at once. That is all it is doing.

In attention, the query and key matrices project each token's representation into a smaller space — often 64 dimensions per head — before the dot products are taken. The reason is that different heads can then pick different directions to compare along, so one head compares subject to verb and another compares a pronoun to its antecedent, without competing for the same numbers.

Removing a direction, and why it half works

Here is the operation that gets used for something more interesting. If you want the part of a that has nothing to do with b, subtract the projection:

a_perp = a - ( (a . b) / (b . b) ) * b

This is the arithmetic behind a family of attempts to remove an unwanted attribute from a representation. Find a direction in embedding space that seems to encode, say, a gender association — often by taking the average difference between paired words — then project every embedding onto the space perpendicular to it. Afterwards, the dot product of every vector with that direction is zero. The attribute appears to be gone.

It is a genuinely useful technique and its limits are well documented. Later work found that after such a projection, the information is frequently still recoverable: a classifier trained on the "debiased" vectors can still predict the removed attribute well above chance, because the attribute was never confined to one direction. It was spread across many correlated directions, and removing one of them leaves the rest.

This is worth holding onto as a general lesson about linear methods. Subtracting one direction removes exactly one direction. If the thing you dislike lives in a subspace of forty directions, or is encoded non-linearly, the arithmetic will report success while the property survives. The maths did what you asked; the ask was too small.

Orthogonality as a design goal

Orthogonal directions are convenient because they do not interfere. Change the amount of one and the others are untouched. Several pieces of architecture lean on this:

• Orthogonal initialisation starts a weight matrix with mutually perpendicular rows, so the initial signal is neither amplified nor collapsed as it passes through.
• Multi-head attention works better when heads attend to different things, which in geometric terms means their projection subspaces are close to orthogonal. Nothing enforces it; training tends to arrive there because redundant heads gain nothing.
• The residual stream in a transformer is often described as a shared workspace in which different features occupy near-orthogonal directions, so that many can be written and read without erasing one another.

That last description is a useful model, not an established fact. Current interpretability research argues that models pack far more features than they have dimensions, using directions that are almost orthogonal rather than exactly so — a phenomenon called superposition. It explains a great deal and it is still an active research question rather than settled ground. Treat it as the best current story.

The rule to keep

The projection formula answers "how much of this direction is in that vector", and its complement answers "what is left once I take that direction out". Both are one dot product and one subtraction. What neither can do is guarantee that a concept you care about was ever a single direction in the first place.