What a 95 per cent interval promises, and where the textbook one breaks

The promise is about the procedure

A 95 per cent confidence interval is a recipe with a guarantee: if you drew sample after sample and ran the recipe each time, 95 per cent of the intervals it produced would contain the true value. The guarantee is about the long run of the procedure. It is not a statement that the truth has a 95 per cent chance of being inside the one interval in front of you; that reading belongs to the Bayesian interval two lessons on, and it needs a prior to earn it.

You can check the promise directly.

import numpy as np
rng = np.random.default_rng(1)
true_p, n, hits = 0.7, 200, 0
for _ in range(10_000):
    k = rng.binomial(n, true_p)
    p = k / n
    se = np.sqrt(p * (1 - p) / n)
    hits += (p - 1.96 * se) <= true_p <= (p + 1.96 * se)
print(hits / 10_000)            # about 0.95

The recipe here is the textbook one: estimate plus or minus 1.96 standard errors. At p = 0.7 and n = 200 it keeps its promise. The value 1.96 is the point on the bell curve that leaves 2.5 per cent in each tail, and the recipe leans on the central limit theorem to justify the bell.

Where the textbook interval breaks

Run the same check with true_p = 0.98 and n = 50. The coverage is not 95 per cent. It is 63.5 per cent. The recipe misses the truth more than a third of the time while claiming to miss it one time in twenty.

The mechanism is visible in one example. A model scores 49 of 50:

p = 0.98,   SE = √(0.98 × 0.02 / 50) = 0.0198
interval = 0.98 ± 1.96 × 0.0198 = 0.941 to 1.019

An upper bound above 1 for a proportion. And a model scoring 50 of 50 gives SE = 0, so the interval is [1.00, 1.00]: a claim of certainty from fifty observations. The standard error was computed from the estimate, and near the boundary the estimate is a poor stand-in for the truth, so the width is wrong. This is not a small-sample curiosity. At p = 0.9 and n = 100, the coverage is 93 per cent, still short, and it oscillates unpredictably as n changes.

The interval that keeps the promise

The Wilson interval fixes the mechanism by solving for the values of the true p under which the observation would be plausible, instead of assuming the estimate is the truth. With z = 1.96 and z² = 3.84:

centre     = (p + z²/2n) / (1 + z²/n)
half-width = z × √( p(1−p)/n + z²/4n² ) / (1 + z²/n)

For 49 of 50:

centre     = (0.98 + 0.0384) / 1.0768 = 0.946
half-width = 1.96 × √(0.000392 + 0.000384) / 1.0768 = 0.0507
interval   = 0.895 to 0.997

For 50 of 50 it gives 0.929 to 1.000. Both intervals stay inside [0, 1], both are asymmetric, and both are wider than the textbook version on the side that matters. The centre is pulled toward 0.5 by the z²/2n term, which acts like adding two successes and two failures to the count; that pull is the correction.

from statsmodels.stats.proportion import proportion_confint
proportion_confint(49, 50, method="wilson")     # (0.895, 0.997)

If statsmodels is not to hand, the two lines above are the whole formula. Use Wilson whenever np or n(1 − p) is under about 10, which for a rare failure rate is nearly always.

Means with small samples

For a mean rather than a proportion, the small-sample problem is different: s is a noisy estimate of σ, and the 1.96 should widen to account for it. The replacement is the t-distribution's quantile, which depends on n − 1 degrees of freedom: 2.26 at n = 10, 2.09 at n = 20, 2.01 at n = 50, and 1.96 in the limit. At n = 10 the interval is 15 per cent wider than the textbook one, which is the honest width.

Three things an interval cannot tell you

Bias. An interval quantifies sampling noise only. A test set drawn from the wrong population gives a tight interval around the wrong number, and nothing in the width warns you. Module 4 and evaluating-ai both return to where the items came from; no interval formula does.

Which values inside are likely. The interval is a range with a coverage guarantee, not a distribution over the range. The truth is not more likely to be at the centre in any sense the recipe licenses.

That 95 is special. It is a convention. A 90 per cent interval is narrower by a factor of 1.645/1.96 = 0.84, a 99 per cent one wider by 2.576/1.96 = 1.31. Report which one you used, and prefer showing the interval to a bare yes or no about whether it excludes some number.

The habit

An accuracy is a proportion; compute Wilson. A mean over fewer than fifty items; use t. Anything else, resample. And before quoting any of them, ask whether the items were independent, because that assumption is the one no formula on this page can check.