Why a GPU idles: memory-bound versus compute-bound

Two ceilings, not one

A chip can only do so many floating-point operations per second: its compute ceiling. It can also only move so many bytes per second between memory and the units that do the arithmetic: its bandwidth ceiling. Every computation needs both, and it runs at the speed of whichever ceiling it hits first. FLOP counts, from three lessons ago, see only the first. Most of the surprises in ML performance come from the second.

The numbers for an A100-class GPU:

compute:    312 × 10^12 FLOP/s (half precision)
bandwidth:    2 × 10^12 bytes/s

For a laptop CPU, roughly 10^11 FLOP/s and 5 × 10^10 bytes/s.

Arithmetic intensity

Divide the FLOPs a computation performs by the bytes it must move, and you get its arithmetic intensity, in FLOPs per byte. Compare it with the chip's ridge point, the compute ceiling divided by the bandwidth ceiling:

A100 ridge:   312 × 10^12 / 2 × 10^12 = 156 FLOP/byte
laptop ridge: 10^11 / 5 × 10^10        =   2 FLOP/byte

Below the ridge, the computation is memory-bound: the arithmetic units wait for data and the achieved FLOP/s is the intensity times the bandwidth. Above it, compute-bound: the chip is doing arithmetic as fast as it can. Plot achieved performance against intensity and you get a rising line that flattens at the ridge, shaped like a roof, which is why this is called the roofline model.

Generating one token is memory-bound

A model generating text produces one token at a time, and for each token it multiplies every weight matrix by a single vector. Each weight, 2 bytes in fp16, is read from memory once and used for one multiply and one add: 2 FLOPs per 2 bytes, an intensity of 1.

On the A100 that is 156 times below the ridge. The chip is doing arithmetic less than one per cent of the time. What it is doing is streaming 14 GB of weights past the arithmetic units, and the time that takes is set by bandwidth alone:

14 GB / 2 TB/s = 7 ms per token  →  at most about 140 tokens per second

No FLOP count predicts that. On a laptop with 50 GB/s of bandwidth and a 4 GB int4 model, the same arithmetic gives 80 ms per token, about twelve tokens a second, which is what people see running a small model locally. Quantising to int4 speeds up generation not because the arithmetic is cheaper but because there are a quarter as many bytes to stream.

Batching is how you climb the roof

Process 64 sequences at once and each weight, still read once, is used 64 times: intensity 64. Process 256 and it is 256, above the ridge, and now the chip is compute-bound and the FLOP count starts to predict the time. This is why serving systems batch requests, why throughput per GPU rises almost linearly with batch size until the ridge, and why the latency for a single user is what it is regardless of how fast the chip's arithmetic is. Training uses batches of thousands of tokens against each weight and lives well above the ridge, which is why FLOP arithmetic works for training time and fails for single-stream inference.

The operations that never climb

Elementwise work has an intensity near a quarter: adding two tensors reads 8 bytes and writes 4 to do one FLOP. Activation functions, layer normalisation, dropout, residual additions are all like this, and all are memory-bound on every chip ever made. A transformer layer that spends 5 per cent of its FLOPs on them can spend 40 per cent of its time on them. The fix is fusion: combine several elementwise steps into one kernel so the data is read once and written once. Every compiled-model toolchain does this, and the reason is on this page.

Why training reaches 40 per cent of the rating

The 312 trillion is for one ideal operation running continuously. A real step alternates large matrix multiplies, which approach the rating, with attention blocks, normalisations, optimiser updates and communication between GPUs, none of which do. The average over a step is the utilisation figure, and 35 to 45 per cent is a good result. Quoting a job's cost from the rating alone underestimates it by that factor.

Measure the roof you own

import numpy as np, time
n = 4096
A = np.random.rand(n, n).astype(np.float32); x = np.random.rand(n, 1).astype(np.float32)
for cols in (1, 16, 256):
    X = np.repeat(x, cols, axis=1)
    t = time.perf_counter(); A @ X; dt = time.perf_counter() - t
    print(cols, round(2 * n * n * cols / dt / 1e9), "GFLOP/s")

The one-column case is a matrix-vector product with intensity near 0.5 and runs at bandwidth speed. The 256-column case is compute-bound. The ratio between the two lines is your machine's version of the gap this lesson is about, and it will be somewhere between ten and a hundred.

What the model leaves out

The roofline has two ceilings; real chips have more. Cache levels give a small working set a higher effective bandwidth. Communication between GPUs in a cluster is a third, lower ceiling that bounds large training runs. And the model says nothing about latency between dependent operations, which matters for small models where launching a kernel takes longer than running it. It is still the first tool to reach for, because the single most common performance mistake is counting FLOPs for a computation that was never going to be limited by them.