Why averages look normal, and when they refuse to

Watch it happen

The central limit theorem says that the average of enough independent draws is approximately normal, whatever shape the draws came from, provided their variance is finite. Rather than prove it, watch it.

import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(0)
for n in (1, 2, 10, 50):
    avg = rng.integers(1, 7, size=(100_000, n)).mean(axis=1)   # average of n dice
    plt.hist(avg, bins=60, alpha=0.5, label=f"n={n}")
plt.legend(); plt.show()

One die is flat: six equal bars. The average of two is a triangle, peaked at 3.5. The average of ten is already a recognisable bell. At fifty it is a bell to the eye and to any test you care to run, and its spread is 1.71 / √50 = 0.24, exactly what the last lesson predicted.

Now change the source to something skewed:

avg = rng.exponential(1.0, size=(100_000, n)).mean(axis=1)

The exponential distribution starts at zero and has a long right tail. The average of five is still visibly lopsided. At thirty it is close to a bell; at a hundred, indistinguishable. Skew slows the convergence, and the rule of thumb that thirty is enough is a rule for mild skew, not a law.

Finally, a heavy tail:

avg = (rng.pareto(1.5, size=(100_000, 10_000)) + 1).mean(axis=1)

The Pareto with tail exponent 1.5 has a finite mean but infinite variance. The average of ten thousand draws is still wildly skewed, with a long tail of averages that were dragged by a single enormous value. There is no n at which this becomes a bell, because the theorem's one condition is not met.

Why adding things makes a bell

The distribution of a sum of two independent variables is the convolution of their two distributions, which is a kind of smoothing: every bump in one is smeared by the whole shape of the other. Do that repeatedly and bumps wash out. The normal distribution is the shape that convolution leaves unchanged, apart from widening; it is the fixed point of the process, so everything with finite variance drifts toward it. Heavy tails escape because a single draw can be larger than the sum of all the others, and no amount of smoothing removes a spike that keeps arriving.

What the theorem is about, and what it is not

The theorem describes the distribution of the average, across repeated samples. It says nothing about the data. Two confusions follow from mixing those up.

"My data are not normal, so I cannot compute a confidence interval for the mean." Wrong, usually. The interval is about the average, and the average of fifty non-normal values is close to normal unless the tail is heavy.

"The average is normal, so the data must be." Also wrong. Salaries are skewed, and the average of a thousand salaries is bell-shaped anyway.

The legitimate worry is the third case: "my metric is the mean of heavily skewed latencies, so fifty may be too few for the bell to have formed." There the honest move is to simulate, as above, with your own data, or to report a percentile, whose sampling distribution does not depend on the tail.

A rule that replaces the thirty

For a proportion, the average of zeros and ones, the bell needs both outcomes to have appeared a reasonable number of times. The usual condition is np ≥ 10 and n(1 − p) ≥ 10. An event with p = 0.01 therefore needs n ≥ 1000 before the normal approximation to its rate is trustworthy, not thirty. Rare-event rates are exactly where people apply the thirty rule and get intervals that include negative numbers.

Where the bell appears inside a model

The same theorem is why so much of deep learning is analysed with Gaussians. A unit in a layer computes a sum of hundreds of products of weights and inputs; that sum is near-normal by the theorem, which is what the initialisation arithmetic in module 8 relies on. A mini-batch gradient is an average over the batch, so its noise around the full gradient is near-normal, which is why the noise in stochastic gradient descent is modelled as Gaussian. And a batch loss is a sum over examples. None of these are assumptions; they are the theorem applied.

The tails converge last

One limit to state plainly. The theorem promises the middle of the distribution of the average becomes a bell. The far tails take much longer. The probability that the average of thirty skewed values sits more than five spreads from its mean is not the Gaussian's 3 × 10^-7; it can be a thousand times larger. This matters precisely when you are estimating something rare, a failure rate or a worst case, and it is why the next lessons treat rare rates with their own arithmetic rather than a bell.