Entropy: the average surprise, computed by hand

The definition, and one calculation

Entropy is the average surprise of a distribution: what you expect to pay per outcome, in bits, if outcomes arrive according to that distribution.

H(p) = − Σ p_i × log2(p_i)

Do one by hand and it stops being a formula. Four outcomes with probabilities 0.5, 0.25, 0.125 and 0.125:

0.5   × 1 bit  = 0.500
0.25  × 2 bits = 0.500
0.125 × 3 bits = 0.375
0.125 × 3 bits = 0.375
                 -----
H = 1.75 bits

The surprise of each outcome, weighted by how often it happens. Note that the rare outcomes cost three bits each but only occur an eighth of the time, so they contribute less than the common one.

Now the same four outcomes, equally likely. Each costs log2(4) = 2 bits and H = 2 bits. That is the maximum: entropy is largest when every outcome is equally likely, and then it equals log2 of the number of outcomes.

And a nearly certain distribution, 0.97, 0.01, 0.01, 0.01:

0.97 × 0.044 = 0.043
0.01 × 6.644 = 0.066   (three times)
H = 0.043 + 0.199 = 0.242 bits

Rare outcomes are very surprising when they happen, 6.6 bits each, but they almost never happen, so the average is small. Entropy is zero only when one outcome has probability 1.

Effective number of choices

Two to the power of the entropy is the effective number of equally likely outcomes. The 1.75-bit distribution above behaves like 2^1.75 = 3.36 equally likely choices; the 0.242-bit one like 2^0.242 = 1.18. This number has a name when the distribution is a model's prediction of the next token: perplexity. The course how-llms-work treats what it means for a language model; here the point is only that it is 2^H, or e^H if H is in nats, and nothing more mysterious.

The number to compute before training anything

Suppose a binary label is 1 in ten per cent of rows and 0 in ninety. A model that knows nothing but that base rate predicts 0.1 every time, and its average loss is the entropy of the label:

H = −(0.1 × log2 0.1 + 0.9 × log2 0.9)
  = −(0.1 × −3.322 + 0.9 × −0.152)
  = 0.332 + 0.137 = 0.469 bits = 0.325 nats

That is the loss of a model that has learned nothing except the frequency. Any model reporting a cross-entropy above 0.325 nats on that data is worse than a constant. People routinely train for hours, see a loss of 0.40, and call it progress, because they never computed the number the constant predictor gets. For a balanced binary label the floor is ln 2 = 0.693 nats; for ten balanced classes it is ln 10 = 2.303. Compute it, write it on the plot, and judge the curve against it.

Where else entropy appears

• A model's own uncertainty. The entropy of the next-token distribution says how decided the model is at that position. Low entropy on a factual token and high entropy on a name it is inventing is a pattern worth looking for.
• Choosing a split. Decision trees pick the feature whose split most reduces the entropy of the labels; machine-learning-foundations works through it.
• How peaked a softmax is. The entropy of an attention row tells you whether a head is looking at one token or spreading over many.

What entropy is blind to

Entropy sees only the list of probabilities, not what they are attached to. A distribution split evenly between "cat" and "kitten" has exactly one bit of entropy, and so does one split between "cat" and "lorry", though the second is far more uncertain in any sense that matters. Entropy has no notion of two outcomes being close. If nearness matters, you need a metric on the outcomes as well, which is a different tool.

A second caution concerns continuous quantities. The entropy of a density, called differential entropy, can be negative and changes when you change units; a Gaussian with standard deviation σ has entropy ½ log2(2πe σ²), which is below zero once σ < 0.24. It is still useful for comparing two densities in the same units, but the intuition "entropy is a count of bits" holds only for discrete outcomes.

Doing it in code

import numpy as np
p = np.array([0.5, 0.25, 0.125, 0.125])
H = -(p * np.log2(p)).sum()      # 1.75

With zeros in p this gives a nan from 0 × log 0; mask them out, because the correct contribution of an impossible outcome is zero. That single line, applied to your label column, is the baseline that the next lesson turns into a floor no model can go beneath.