Cross-entropy and KL: the cost of believing the wrong distribution

Paying with the wrong code

Entropy is what you pay when your predictions match reality. Cross-entropy is what you pay when they do not. If outcomes really arrive with probabilities p but you built your code, or your model, around probabilities q, the average cost is

H(p, q) = − Σ p_i × log2(q_i)

The probabilities that decide how often you pay are the true ones, p. The probabilities that decide how much you pay each time are yours, q.

Take the reality from the last lesson, p = (0.5, 0.25, 0.125, 0.125), whose entropy is 1.75 bits. Suppose your model believes the four outcomes are equally likely, q = (0.25, 0.25, 0.25, 0.25). Every outcome now costs −log2(0.25) = 2 bits, so H(p, q) = 2 bits. You pay a quarter of a bit more per outcome than a perfect model would.

Now a model that is wrong in a different way, q = (0.7, 0.1, 0.1, 0.1):

0.5   × −log2(0.7) = 0.5   × 0.515 = 0.257
0.25  × −log2(0.1) = 0.25  × 3.322 = 0.830
0.125 × −log2(0.1) = 0.125 × 3.322 = 0.415
0.125 × −log2(0.1) = 0.125 × 3.322 = 0.415
                                     -----
H(p, q) = 1.918 bits

This model is overconfident in the first outcome and pays for it on the second, which it thought was rare and which happens a quarter of the time.

The gap has a name

The extra cost over the entropy is the Kullback-Leibler divergence:

KL(p ‖ q) = H(p, q) − H(p) = Σ p_i × log2(p_i / q_i)

For the uniform model it is 2 − 1.75 = 0.25 bits; for the overconfident one, 1.918 − 1.75 = 0.168 bits. KL is never negative, and it is zero only when q equals p exactly. It measures, in bits per outcome, how wrong a belief is.

That identity, H(p, q) = H(p) + KL(p ‖ q), is the most useful line in this module. When you train a model by minimising cross-entropy against data, H(p) is a property of the data and does not move. So minimising cross-entropy is minimising KL, and the loss can never go below the data's own entropy. A language model's loss plateaus above zero not because it has failed but because language is genuinely uncertain: the floor is real. The previous lesson computed that floor for a label column; the same logic applies to every token.

With one-hot labels

In classification the "true" distribution for a single example is a one-hot: probability 1 on the correct class, 0 elsewhere. Every term in the sum vanishes except one, and the cross-entropy collapses to −log q(correct): the surprise of the right answer. That is why the loss you see in a training loop is simply minus the log of the probability the model gave the label. The elaborate definition and the simple one are the same thing.

KL is not symmetric

Swap the roles and you get a different number. With p = (0.5, 0.25, 0.125, 0.125) and q uniform:

KL(p ‖ q) = 0.25 bits
KL(q ‖ p) = 0.25 × (log2(0.25/0.5) + log2(0.25/0.25) + 2 × log2(0.25/0.125))
          = 0.25 × (−1 + 0 + 2) = 0.25 bits

Equal in this case, by coincidence of the round numbers. Change p to (0.7, 0.1, 0.1, 0.1) and KL(p ‖ uniform) = 0.53 while KL(uniform ‖ p) = 0.64. The direction matters, and it matters most when one distribution puts probability near zero where the other does not.

The practical consequence: if q assigns probability 0 to something p can produce, KL(p ‖ q) is infinite, because you would pay −log 0 when it happened. A model trained against exact one-hot targets is therefore pushed to make the correct logit infinitely larger than the rest, and never gets there. Label smoothing, which replaces the target (1, 0, 0, 0) with something like (0.97, 0.01, 0.01, 0.01), exists to give the model a finite place to stop.

The direction also changes behaviour. Minimising KL(p ‖ q) over q makes q cover everywhere p has mass, even if it must spread thin. Minimising KL(q ‖ p) makes q sit on one mode of p and ignore the others. When a preference-trained language model is penalised for drifting from its reference model, the penalty is the second kind, taken over the model's own samples, which is one reason such models become narrower rather than broader.

Two things KL is not

It is not a distance: it is asymmetric and does not satisfy the triangle inequality, so "the KL between two embeddings" is not a meaningful phrase unless both are distributions. And it is not bounded: a single near-impossible event can make it arbitrarily large, so an average KL across a dataset can be dominated by one row.

The trap in the library

torch.nn.functional.kl_div(input, target) expects input to be log probabilities and target to be plain probabilities, and it computes KL(target ‖ exp(input)). Passing probabilities for both runs without error and returns nonsense. scipy.special.rel_entr(p, q) takes plain probabilities for both and returns the elementwise terms; sum them yourself. Compute one small example by hand, as above, and check the library agrees before trusting it on anything larger.