Curvature, and what the optimisers are actually doing

The second derivative, in one line

The first derivative says which way is downhill. The second says how the slope itself is changing — whether the surface curves sharply or gently. In many dimensions the second derivatives form a matrix called the Hessian, one entry for every pair of parameters.

You will never compute a Hessian for a real model. For 7 billion parameters it would have 49 × 10^18 entries. But you need one number from it, and it is the one that explains most optimisation trouble.

The number that explains the zigzag

The Hessian's eigenvalues are the curvatures along its eigen-directions. The ratio of the largest to the smallest is the condition number, and it decides how gradient descent behaves.

Take a simple quadratic bowl with curvature 100 along one axis and 1 along another — condition number 100. Gradient descent must use a learning rate small enough not to diverge along the steep axis, roughly 2/100 = 0.02. But along the shallow axis, a step of 0.02 makes progress at rate 0.02 × 1, so covering that direction takes about 100 times as many steps as the steep one.

That is the zigzag: fast oscillation across the narrow direction, crawling progress along the long one. The number of steps needed scales roughly with the condition number, and real loss surfaces have condition numbers in the thousands.

Momentum: remembering where you were going

The fix costs one extra buffer. Instead of stepping along the current gradient, keep a running average of recent gradients and step along that.

v  <-  beta * v + grad
w  <-  w - learning_rate * v

with beta typically 0.9.

Why it works is worth seeing concretely. Along the narrow direction the gradient alternates sign each step, so successive terms in the running average cancel. Along the long direction the gradient points the same way every step, so they accumulate. With beta = 0.9 the accumulated step approaches 1/(1 − 0.9) = 10 times a single gradient. The consistent direction gets amplified tenfold and the oscillating one gets damped — precisely the correction the geometry needed.

Adam: a different step size per parameter

Adam adds a second idea. Track not just the average gradient but the average squared gradient, per parameter, and divide by its square root:

m <- beta1 * m + (1 - beta1) * grad          (average gradient)
v <- beta2 * v + (1 - beta2) * grad^2        (average squared gradient)
m_hat = m / (1 - beta1^t)                    (bias correction)
v_hat = v / (1 - beta2^t)
w <- w - lr * m_hat / (sqrt(v_hat) + eps)

Defaults: beta1 = 0.9, beta2 = 0.999, eps = 1e-8.

The division is the substance. A parameter whose gradients are consistently large gets a large v, so its effective step is scaled down; a parameter with small gradients gets scaled up. Each parameter ends up with its own step size, adapted automatically. That is a crude, cheap, diagonal-only substitute for using the curvature.

Bias correction is not decoration. Both m and v start at zero, so early estimates are biased towards zero. At step 1 with beta2 = 0.999, the raw v is 0.001 times the true squared gradient; dividing by 1 − 0.999^1 = 0.001 restores it. Without this the first few hundred steps take wildly wrong step sizes, which shows up as an unstable start and is the reason warmup helps.

What it costs

Adam stores two extra numbers per parameter. For 7 billion parameters in 32-bit:

weights   :  7e9 * 4  = 28 GB
gradients :  7e9 * 4  = 28 GB
Adam m    :  7e9 * 4  = 28 GB
Adam v    :  7e9 * 4  = 28 GB
                        112 GB, before a single activation

This is the arithmetic behind the familiar claim that full fine-tuning of a 7B model needs far more memory than you would guess from its size, and behind the popularity of memory-efficient variants: 8-bit Adam quantises the two states, and SGD with momentum keeps only one.

Reading a loss curve for which problem you have

Three patterns, three causes:

• Loss spikes to NaN early. The learning rate exceeds the stability limit for the steepest curvature. Lower it, or add warmup so the first steps are small.
• Loss decreases very slowly and smoothly. The learning rate is too small, or the problem is badly conditioned. Try raising the rate first; it is one experiment.
• Loss falls then plateaus while the gradient norm stays high. Oscillation across a narrow valley. A learning-rate decay usually breaks it, which is the main reason cosine schedules are the default.

Log the gradient norm alongside the loss. Two curves distinguish these cases in seconds; one curve does not.

The honest position on optimisers

AdamW — Adam with weight decay applied separately from the gradient — is the default for transformer training and has been for years. The claim that a newer optimiser beats it is made regularly and survives independent replication rarely, in part because comparisons are extremely sensitive to how much tuning each method received. If you are choosing an optimiser, the evidence supports AdamW plus a careful learning rate over almost anything else, and the time is better spent on the learning rate than on the optimiser.

The rule to keep

Curvature differing across directions is why plain descent is slow. Momentum averages away the oscillation, Adam rescales each parameter by its own gradient history, and both are approximations chosen because the exact correction is unaffordable.