Counting the floating-point operations in a layer and a training run

One entry, one dot product

Module 2 showed that every entry of a matrix product is a dot product between a row of the left matrix and a column of the right. A dot product of length k is k multiplications and k additions: 2k floating-point operations, or FLOPs. Multiplying an m × k matrix by a k × n one produces m × n entries, so

FLOPs of (m × k) · (k × n) = 2 × m × n × k

That one line is most of the cost accounting of deep learning.

A linear layer taking a 4,096-dimensional vector to another 4,096-dimensional vector is a 4096 × 4096 matrix applied to one column:

2 × 4096 × 4096 × 1 = 33.5 million FLOPs

for one token. The matrix has 16.8 million parameters, and each parameter was used exactly once, for one multiplication and one addition. That generalises.

The rule: two FLOPs per parameter per token

In a forward pass, every weight in every linear layer touches each token once, multiplying and adding. So

forward FLOPs per token ≈ 2 × (number of parameters)

A seven-billion-parameter model spends about 14 billion FLOPs to process, or generate, one token. Embedding lookups cost nothing, being a table read, and the attention scores are an extra term the next lesson counts separately; for ordinary context lengths the rule above is within twenty per cent.

Training is three times a forward pass

Backpropagation, from module 3, computes two things per layer: the gradient with respect to the layer's input, to pass backward, and the gradient with respect to its weights. Each is a matrix multiply of the same size as the forward one. So the backward pass costs about twice the forward, and a full training step, forward plus backward, is about three times the forward:

training FLOPs ≈ 6 × parameters × tokens

Seven billion parameters on one trillion tokens:

6 × 7 × 10^9 × 10^12 = 4.2 × 10^22 FLOPs

That number is the one to carry. Everything about the cost of a training run follows from it and from how fast the hardware actually runs.

From FLOPs to time

A data-centre GPU of the A100 generation is rated at 312 trillion FLOPs per second in half precision. Nobody achieves that. Real training runs sustain around 40 per cent of the rating, for reasons the arithmetic-intensity lesson explains, so call it 125 trillion per second.

4.2 × 10^22 / 1.25 × 10^14 = 3.4 × 10^8 seconds ≈ 10.7 years on one GPU
                                             ≈ 3.9 days on 1,000 GPUs

That is roughly what a seven-billion-parameter model on a trillion tokens costs, and the arithmetic is two multiplications long. The course how-llms-work turns it into money; the point here is that you can do it yourself, from a parameter count and a token count, in under a minute.

On a laptop CPU, NumPy manages on the order of 10^11 FLOPs per second on a well-shaped matrix multiply. A forward pass of the seven-billion model, at 14 billion FLOPs per token, would take 0.14 seconds per token if FLOPs were the only limit. They are not, and the third lesson from here says what is, but the FLOP count sets the floor.

Batches do not change the count

Processing a batch of 32 tokens through the same layer is 2 × 4096 × 4096 × 32 FLOPs: thirty-two times the work, for thirty-two times the tokens. The FLOPs per token are unchanged. What changes with batching is efficiency, how close the hardware gets to its rating, and that is a story about memory, not arithmetic.

Where the rule misleads

• Attention is extra, and grows with context. The scores between every pair of tokens add 4 × L × d FLOPs per token per layer for context length L and width d. At L = 4096 and d = 4096 that is one sixth of the linear layers' cost; at L = 24,576 it equals them. The next lesson has the arithmetic.
• Mixture-of-experts models have more parameters than they use per token. Count the parameters that a token actually passes through, not the total; a model with 400 billion parameters that routes each token through 40 billion costs like a 40-billion model per token.
• FLOPs are not time. A layer that does 10^9 FLOPs on 10^9 bytes of weights is waiting for memory, and no FLOP count sees that.
• Rated FLOPs are for one shape. The 312 trillion figure applies to large half-precision matrix multiplies. Elementwise operations, small matrices and single precision get a fraction of it.

Do it once

import numpy as np, time
A = np.random.rand(4096, 4096).astype(np.float32)
x = np.random.rand(4096, 32).astype(np.float32)
t = time.perf_counter(); A @ x; dt = time.perf_counter() - t
print(2 * 4096 * 4096 * 32 / dt / 1e9, "GFLOP/s")

Run it on whatever you have. The number you get is the constant that turns every FLOP count in this module into seconds on your own machine.