An estimator is a recipe, and why the variance divides by n − 1

A number computed from a sample

An estimator is a rule: give it a sample, it gives back a number that stands in for something about the whole population. The sample mean estimates the population mean. The fraction of test items a model gets right estimates its accuracy on everything it will ever see. The rule is the estimator; the number it returns on one particular sample is the estimate.

Because the sample is random, the estimate is too. Draw a different sample and you get a different number. So an estimator has properties of its own, separate from any single result. Two matter most.

• Bias: if you repeated the sampling forever and averaged all the estimates, would you land on the truth? If yes, the estimator is unbiased.
• Variance: how far do the estimates scatter from one sample to the next?

The sample mean is unbiased for the population mean. The most common recipe for the variance is not, and the reason is worth seeing once with numbers.

Why dividing by n undercounts

Take a population of two values, 0 and 2. Its mean is 1 and its variance, the average squared distance from the mean, is 1. Now draw samples of size two with replacement. There are four equally likely samples:

sample   sample mean   Σ(x − mean)²   ÷ n   ÷ (n−1)
(0, 0)       0              0          0       0
(0, 2)       1              2          1       2
(2, 0)       1              2          1       2
(2, 2)       2              0          0       0
                          average:    0.5      1.0

Divide by n and the estimates average to 0.5, half the true variance. Divide by n − 1 and they average to exactly 1. The n − 1 is not a fudge; it is the correction that makes the average of the estimates equal the truth.

The mechanism: the formula measures distances from the sample mean, and the sample mean was chosen to be as close to the sample points as possible. Module 2 showed that the mean is the point minimising the sum of squared distances. So distances from it are systematically smaller than distances from the true mean, which the sample does not know. One degree of freedom was spent estimating the mean, and n − 1 is what is left.

At n = 1000 the two recipes differ by a tenth of a per cent and nobody cares. At n = 5 they differ by 25 per cent, and small-sample statistics is full of places where they matter.

The trap in the libraries

The same column gives two different variances depending on which library you ask:

import numpy as np, pandas as pd, torch
x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
np.var(x)                      # 8.25   divides by n
pd.Series(x).var()             # 9.17   divides by n − 1
torch.tensor(x, dtype=torch.float).var()   # 9.17   n − 1 by default
np.var(x, ddof=1)              # 9.17   ddof = 'delta degrees of freedom'

The ratio is n / (n − 1), here 10 / 9. The standard deviation differs by the square root of that. If a number you computed disagrees with a colleague's by a few per cent, check ddof before anything else.

Unbiased is not the same as best

Bias is one property. An estimator's overall error combines both:

mean squared error = bias² + variance

A slightly biased estimator with much lower variance can be more accurate on any single sample. Dividing by n is biased but has lower variance than dividing by n − 1, and for some purposes it wins. The whole of regularisation, later in this module, is accepting a little bias to buy a lot less variance.

Two more surprises of the same kind. The sample standard deviation, the square root of the n − 1 variance, is still biased, low, because the square root of an average is less than the average of square roots. And the sample maximum is always a biased estimate of the population maximum: the largest of ten draws is never larger than the largest thing there is, and is usually smaller.

A useful estimator you were not taught

That last fact has a famous fix. Suppose items are numbered 1 to N, you see k of them at random, and the largest number you saw was m. The best estimate of N is not m but

N ≈ m + m/k − 1

Seeing five items with a maximum of 60 suggests N ≈ 60 + 12 − 1 = 71. Allied statisticians used this on the serial numbers of captured German tanks and estimated monthly production at 246 against a true figure of 245, while intelligence reports said 1,400. The same recipe estimates how many distinct records a system holds from a few sampled ids, or how many bugs a codebase has from the ones found so far. An estimator is a recipe, and some recipes are much better than the obvious one.