Quantisation is rounding with a scale: int8, int4 and the outlier problem

The whole idea in one line

To store a float weight as an 8-bit integer, pick a scale and round:

q = round(w / scale)          an integer from −127 to 127
w ≈ q × scale                 to recover it

The scale is chosen so that the largest weight in the tensor lands on 127: scale = max|w| / 127. That is symmetric int8 quantisation, and it is most of what the word means. The course running-models-yourself covers the tools; this lesson is the arithmetic those tools perform.

One weight, by hand

A tensor whose largest weight is 0.02 has scale = 0.02 / 127 = 0.000157. A weight of 0.0034:

q = round(0.0034 / 0.000157) = round(21.6) = 22
w ≈ 22 × 0.000157 = 0.003465
error = 0.000065, about 2 per cent of the weight

The error of any weight is at most half a scale step, 0.0000787 here. For weights near the maximum that is 0.4 per cent; for weights a tenth the size it is 4 per cent; for a weight of 0.00005 it rounds to zero and the error is 100 per cent. Quantisation is exact for the large values and brutal for the small ones, because the step size is set by the largest.

The outlier problem

Now suppose that same tensor contains one weight of 5.0. The scale becomes 5 / 127 = 0.039. The weight of 0.0034 is round(0.086) = 0. So is every weight below 0.02: the whole tensor apart from the outlier rounds to zero or to one step. One number has destroyed a million.

This is not hypothetical. Large language models develop a few channels with activations tens or hundreds of times larger than the rest, and naive int8 quantisation of those activations wrecks the model while the weights alone quantise fine. Three responses, all arithmetic:

1. Per-channel scales. One scale per row or column instead of per tensor, so the outlier only sets the step for its own row. Costs one float per row; nearly free.
2. Group-wise scales. One scale per block of 64 or 128 weights. An int4 model with a 16-bit scale per 128 weights spends 4 + 16/128 = 4.125 bits per weight, and the outlier's damage is confined to its block of 128.
3. Move the outlier. Multiply an activation channel by a constant and divide the matching weight row by the same constant; the product is unchanged, and the difficulty has been shifted from the activations, which are hard to quantise, to the weights, which are easy. This is what the smoothing methods do.

Asymmetric, and the zero-point

If the values are not centred, an activation after a ReLU is never negative, half the integer range is wasted. The asymmetric form adds an offset:

q = round(w / scale) + zero_point

so that the minimum maps to 0 and the maximum to 255 in uint8. The zero-point is chosen so that a real zero maps exactly onto an integer, because padding and ReLU produce exact zeros in quantity and rounding them to something else adds a bias everywhere.

Four bits

Int4 has sixteen levels. With a per-tensor scale that is hopeless; with group-wise scales it works for weights, and int4 weights with fp16 activations is the standard recipe for running a large model on a small device. The arithmetic of why: weights are static, so they can be quantised carefully, once, with all the tricks above; activations change with every input and must be quantised on the fly, which is why they are usually left at higher precision.

The error introduced is measurable and it is not zero. A model at int4 typically loses a few tenths of a point of perplexity relative to fp16; at int3 it loses several points and at int2 it stops working. The loss is larger for smaller models, because they have fewer redundant weights to absorb the noise, and larger in the layers nearest the output. Anyone claiming int4 is free has not measured it on their own task; the honest statement is that it is cheap and the price is known.

Why it is faster, and when it is not

Module 7's arithmetic-intensity lesson said single-token generation is bound by streaming the weights. Int4 weights are a quarter of the bytes, so generation is up to four times faster, before any change in the arithmetic. For large-batch throughput, which is compute-bound, the gain depends on whether the hardware has integer matrix units; if it must convert int4 back to fp16 before multiplying, there is no speed-up at all, only memory saved.

Doing it yourself

import numpy as np
w = np.random.randn(4096).astype(np.float32) * 0.01
w[17] = 5.0                                   # one outlier
scale = np.abs(w).max() / 127
q = np.round(w / scale).astype(np.int8)
print((q == 0).mean())                        # fraction of weights that became zero: nearly all
row_scale = np.abs(w[:2048]).max() / 127      # a group without the outlier
q2 = np.round(w[:2048] / row_scale)
print((q2 == 0).mean())                       # a few per cent

Run it, then delete the outlier line and run it again. The difference between the two fractions is the whole reason quantisation papers exist.