Bytes per parameter, and the arithmetic of what fits on a card

A model is a number of bytes

Before a model can do anything it has to fit in memory, and the arithmetic for that is a multiplication: parameters times bytes per parameter. The bytes per parameter depend on the number format, which module 8 opens up; for now the four sizes are enough.

float32   4 bytes
bf16/fp16 2 bytes
int8      1 byte
int4      0.5 byte

A seven-billion-parameter model:

float32  28 GB
fp16     14 GB
int8      7 GB
int4    3.5 GB

Set those against the memory you have. A consumer graphics card has 8, 12, 16 or 24 GB. A laptop with a shared memory pool has 8 to 32 GB, of which the model can use perhaps three quarters. A phone has 6 to 12 GB shared with everything else. So the seven-billion model runs on an 8 GB card in int4, on a 16 GB card in fp16, and on a phone only in int4 with the operating system complaining. A seventy-billion model at int4 is 35 GB and needs either two cards or a machine built for it. None of this requires a benchmark; it is the multiplication.

Room to think: the KV cache

At inference a transformer keeps, for every token in the context, the key and value vectors of every layer, so that it need not recompute them for each new token. Per token that is

2 (key and value) × layers × width × bytes
= 2 × 32 × 4096 × 2 = 524,288 bytes ≈ 0.5 MB

for a typical seven-billion model in fp16. A 4,096-token context is therefore 2.1 GB on top of the weights, and a 32,000-token context is 17 GB, more than the weights themselves. Serving eight such conversations at once needs eight caches. When a hosted model charges more for long contexts, this is what it is charging for. The course how-llms-work covers the engineering around the cache; the size is this one product.

Training is far larger than the model

To train, you hold more than the weights. In the usual mixed-precision recipe with Adam, per parameter:

weights in fp16              2 bytes
master copy of weights fp32  4 bytes
gradient fp16                2 bytes
Adam first moment fp32       4 bytes
Adam second moment fp32      4 bytes
                            16 bytes per parameter

Seven billion parameters times sixteen is 112 GB, before a single activation is stored. That is why full fine-tuning of a seven-billion model needs a data-centre card or several consumer ones, though inference needs one.

Then the activations. Backpropagation, from module 3, must keep every layer's inputs until the backward pass. For a transformer block the saved tensors come to about 34 bytes per hidden unit per token per layer in mixed precision, once the attention scores are handled in blocks. For a 2,048-token sequence, width 4,096, 32 layers:

2048 × 4096 × 32 × 34 ≈ 9.1 GB per sequence

A batch of eight is 73 GB. Activation memory scales with batch and sequence length, weight memory does not, and this is why the batch that fits is decided by activations. Gradient checkpointing throws most of them away and recomputes them during the backward pass, trading about a third more compute for a fivefold memory reduction; it exists because of this arithmetic.

Why adapters made fine-tuning affordable

Freeze the base model and train small added matrices, the LoRA recipe: the 16 bytes per parameter apply only to the adapter's parameters, perhaps 0.5 per cent of the total. The base sits at 2 bytes per parameter, read-only:

base 7B × 2 bytes         = 14 GB
adapters 35M × 16 bytes   = 0.56 GB
activations, batch 1      ≈ 9 GB (less with checkpointing)

About 24 GB, which is one high-end consumer card, against 120 GB or more for full fine-tuning. Quantise the frozen base to int4 and it is under 15 GB. The technique has a name and a paper, but the reason it works on a laptop is that most of the sixteen bytes were optimiser state, and optimiser state exists only for what you train.

The arithmetic of the data

The same habit applies to datasets. A text corpus of a billion tokens at two bytes per token id is 2 GB; as raw UTF-8 text at four characters a token, about 4 GB. An image dataset of a million 224 × 224 colour images as raw uint8 pixels is 10^6 × 224 × 224 × 3 = 150 GB, which will not fit in RAM and must stream; as JPEGs, about a tenth of that on disk, decoded on the fly. Embeddings for ten million documents at 768 float32 dimensions are 30 GB, which is the reason the search lesson exists.

Three cautions

Peak memory is not average memory: a temporary buffer during a large matrix multiply, or the moment when an optimiser step holds old and new weights together, can add half again. Frameworks reserve memory in blocks, so the number in the monitoring tool is above what the arithmetic says. And a model that fits with 200 MB to spare will fail at the first long input. Leave a fifth free, and when a thing does not fit, do the multiplication again before buying anything.