NaN, Inf, and the five numbers to print about any tensor

Two special values, with rules

The floating-point standard reserves bit patterns for two things that are not numbers. Inf, positive or negative, is what overflow produces, and what division of a non-zero number by zero produces. NaN, "not a number", is what you get when the arithmetic has no defensible answer:

0 / 0          NaN
Inf − Inf      NaN
0 × Inf        NaN
Inf / Inf      NaN
sqrt(−1)       NaN
log(−1)        NaN
log(0)         −Inf   (not NaN, and the distinction is useful)

Both propagate. Any operation with a NaN input gives NaN, and most operations with Inf give Inf or NaN. One NaN in a gradient becomes a NaN in every weight it touches after one optimiser step, and then in every activation, and then in the loss. A NaN in training is therefore rarely where it appears; it is downstream of its origin, and the work is tracing it back.

The rule that surprises everyone

NaN is not equal to anything, including itself:

x = float("nan")
x == x            # False

This is by design, and it is the standard test: x != x is True exactly when x is NaN. It also means x in some_list, a == filter in pandas, and a dictionary lookup all fail to find a NaN. np.isnan and torch.isnan are the honest tests; math.isnan for scalars.

Sorting puts NaN at the end. np.max of an array containing one NaN returns NaN; np.nanmax skips it. Pandas mean() skips NaN silently, which is the more dangerous behaviour: a column with 60 per cent missing values reports a confident mean of the other 40 per cent, and nothing says so unless you print count() beside it. An accuracy computed over a column with NaN labels is an accuracy over the rows that had labels, which may not be the rows you meant.

Where NaN comes from in training

Each of the operations above has a training-time source. The ones that account for most incidents:

• log of zero. A probability underflowed to 0, then −log(0) = Inf in the loss, then NaN in the gradient. Module 5's log-sum-exp is the fix; so is never computing log(softmax(x)) as two steps.
• Square root of a small negative. A variance computed by cancellation, as in the summation lesson, comes out at −1e-9; sqrt returns NaN; a normalisation layer spreads it everywhere. Clamp to zero, or compute the variance the stable way.
• Division by a zero norm. Normalising a vector that happens to be all zeros, a padding embedding, gives 0/0. Add an epsilon to the denominator.
• Overflow. A learning rate too high pushes a weight to 1e20, an activation to Inf, and Inf − Inf in the next layer norm gives NaN. Module 3's gradient-norm plot shows this coming several steps early; gradient clipping stops it.
• fp16. Everything above happens at 65,504 instead of 10^38, which is the half-precision lesson.

Finding the origin

torch.autograd.set_detect_anomaly(True)     # raises at the first backward op producing NaN

This slows training badly and names the operation, which is what you need once and then never again. Cheaper, and worth leaving on: assert after each step that the loss is finite, and log the step at which it stopped being so, because the first non-finite loss is usually a few steps after the first non-finite gradient, and the gradient-norm history around that step tells the story.

if not torch.isfinite(loss):
    raise RuntimeError(f"non-finite loss at step {step}")

The five numbers

For any tensor you are suspicious of, print five things:

def describe(t, name=""):
    t = t.detach().float()
    print(name, tuple(t.shape), t.dtype,
          f"mean={t.mean():.3g} std={t.std():.3g} maxabs={t.abs().max():.3g}",
          f"nonfinite={(~torch.isfinite(t)).sum().item()}")

Shape and dtype, because half of all bugs are a transposed matrix or an integer where a float was meant. Mean and standard deviation, because the variance lesson said what healthy looks like: activations with mean near 0 and spread near 1, weights at their initialisation scale, gradients with a spread that is stable across steps. Maximum absolute value, because an activation of 10^4 is the overflow of ten steps from now. And the count of non-finite entries, because one is already too many.

Call it on the input batch, the first layer's output, the logits and the loss, in that order, at the step things went wrong. The first tensor whose numbers look wrong is upstream of the fault.

Inf on purpose

−Inf is also a tool. Masking in attention sets the scores of forbidden positions to −Inf before the softmax, and exp(−Inf) = 0 exactly, which is cleaner than any large negative number. float("inf") is the right initial value for a running minimum. And a log-probability of −Inf for an impossible event is correct, not a bug, as long as nothing downstream subtracts two of them.

The rule

A NaN is never the problem; it is the problem's last symptom. Test with isnan, never with ==. Print the five numbers, walk upstream, and find the log of zero, the root of a negative, the division by nothing, or the step whose norm doubled three times in a row.