Standard error for a mean and a proportion, by hand

Two formulas, and the numbers they give

The standard error is the spread of an estimate across repeated samples: σ/√n from two lessons ago, with the unknown σ replaced by something you can compute.

For a mean, use the sample standard deviation s:

SE(mean) = s / √n

For a proportion p, such as an accuracy, the standard deviation of a 0/1 variable is √(p(1 − p)), so

SE(p) = √( p (1 − p) / n )

A model scores 85 per cent on 200 items:

SE = √(0.85 × 0.15 / 200) = √0.000638 = 0.0252

About 2.5 points. The rough 95 per cent range is two standard errors either side: 80 to 90. The same 85 per cent on 50 items gives SE = 0.0505, five points, and a range of 75 to 95. Twenty percentage points wide, from a number that was reported to one decimal place.

The table to carry in your head

The product p(1 − p) is largest at p = 0.5, where it is 0.25, so the worst-case standard error is 0.5/√n. Two of those, for the 95 per cent half-width:

n =    100  →  ±10 points
n =    400  →  ± 5 points
n =  1,600  →  ±2.5 points
n = 10,000  →  ± 1 point

Quadrupling the items halves the width. At p = 0.95 the factor √(0.95 × 0.05) = 0.218 is under half the worst case, so the widths shrink accordingly, but the table is the bound to use when you do not yet know p.

The difference between two models

Two models score 85 and 88 per cent, each on its own 200 items. Standard errors of independent estimates combine by adding their squares:

SE(A) = 0.0252,   SE(B) = √(0.88 × 0.12 / 200) = 0.0230
SE(B − A) = √(0.0252² + 0.0230²) = 0.0341

The gap is 3 points and the standard error of the gap is 3.4 points. The gap is smaller than its own uncertainty. Nothing has been shown.

Now suppose both were scored on the same 200 items. Item difficulty is then shared: a hard item is hard for both, and that shared part cancels in the difference. What remains is the items on which the two models disagree. Say there are 30 such items, and model B wins 18 of them to A's 12. The difference is (18 − 12)/200 = 0.03, as before, but its standard error is now approximately

SE(paired difference) ≈ √(18 + 12) / 200 = √30 / 200 = 0.0274

Smaller, because 170 items that both got right or both got wrong contribute nothing to the noise of the comparison. It is still only 1.1 standard errors, still nothing shown, but the paired design was closer. This is why comparing on shared items is worth so much more than comparing on separate sets, and the course evaluating-ai builds the practice on it. The arithmetic is: count the discordant items.

Standard error is not standard deviation

The standard deviation describes the spread of the data. The standard error describes the uncertainty of an estimate computed from the data. At n = 100 the second is a tenth of the first. A plot with SE error bars looks ten times tighter than one with SD bars, and papers switch between them without saying which. When you read "mean ± 2.3", ask which. When you write one, say which.

A useful check: standard deviation does not shrink as you collect more data, because the data are as spread as they are. Standard error does. If a reported uncertainty gets smaller as n grows, it is a standard error.

Three conditions, each already met or not

The proportion formula assumes the items are independent; the design-effect lesson said what to do when they are not. It assumes p is not near 0 or 1, or n is large enough that np and n(1 − p) both exceed about 10; the next lesson shows what breaks when that fails. And it assumes the quantity is a mean. A median, a 95th percentile, an F1 score or an AUC is not a mean, and the formula does not apply; for those, the bootstrap, resampling the items and recomputing, is the tool, and evaluating-ai covers its pitfalls.

On paper, then in code

import math
k, n = 170, 200
p = k / n
se = math.sqrt(p * (1 - p) / n)
print(p, se, p - 2*se, p + 2*se)     # 0.85  0.0252  0.80  0.90

The whole computation fits on the back of an envelope, and doing it there, once, before quoting any accuracy, is the habit this lesson exists to install. An accuracy without an n is a rumour.