Checking a number without looking it up

The cheapest verification there is

Many wrong numbers can be caught in ten seconds, without a source, using arithmetic you already have. This matters because looking things up is expensive and you will not do it every time, whereas a sanity check costs almost nothing and catches the large errors — which are the ones that embarrass you.

Language models are unreliable with numbers in a specific way. They handle small arithmetic reasonably, degrade on multi-step calculation, and are worst at quantities where the plausible-looking answer and the correct answer differ by a factor of ten or a thousand. A wrong digit count produces text that reads perfectly.

Six checks

Order of magnitude. Round everything to one digit and a power of ten. India's population is about 1.4 × 10⁹. A claim that a national scheme reached "340 million households" should stop you: at roughly 4.5 people per household there are about 300 million households in total, so the claim asserts more than universal coverage. You did not need a source.

Per person. Divide the aggregate by the population. A government programme costing ₹90,000 crore across 1.4 billion people is about ₹640 each. Now you know whether the number is large or is merely long. Almost every very large figure in the news becomes interpretable this way, and many become unremarkable.

Units and definitions. Watts against watt-hours is the commonest error in energy writing — one is a rate, the other a quantity, and confusing them makes a claim meaningless. Similarly: revenue against profit, users against accounts, cases against deaths, crore against million, gross against net. When a number seems surprising, suspect the unit before suspecting the world.

Percentage against percentage point. A rise from 2% to 3% is a one percentage point rise and a 50% increase. Both are correct and they sound completely different, which is why the choice between them is a standard persuasive move.

Base rate. A test that is "99% accurate" for a condition affecting 1 in 10,000 people produces, in a million tests, about 100 true positives and about 10,000 false ones. Ninety-nine per cent of the positives are wrong. This arithmetic is behind an enormous share of bad decisions about screening, fraud detection and face recognition, and it takes one minute on the back of an envelope.

Consistency inside the passage. Do the parts sum to the whole? Does the growth rate reproduce the end figure from the start figure? Does the total across categories match the stated total? Fabricated tables very often fail this, because each number was generated to look right individually.

A worked case

Suppose a model tells you: "The plant produces 500 megawatts and supplies electricity to 2 million homes."

Check it. 500 MW running continuously for a year is 500 × 8,760 hours, which is 4.38 million megawatt-hours, or 4.38 billion kilowatt-hours. Divide by 2 million homes: about 2,190 kWh per home per year. That is a plausible figure for a European or Indian urban household and low for a US one — and the plant will not run continuously, so the real capacity factor pulls it down by a third or more. So the claim is roughly right for some countries and optimistic for others. That is a useful answer, and it took two multiplications.

Notice what happened: you did not verify the claim, you established the range in which it could be true. That is usually enough to decide whether to spend twenty minutes finding the real figure.

Making the model do it properly

If a calculation matters, do not ask a language model to perform it in prose. Ask it to write the calculation and run it. Every serious assistant can produce a few lines of Python, and Python is free, exact, and available in your browser through any notebook service, or on your machine with one install.

hours = 8760
mw = 500
capacity_factor = 0.55
kwh = mw * 1000 * hours * capacity_factor
print(kwh / 2_000_000)   # kWh per home per year

Code is checkable in a way that prose arithmetic is not: you can read the formula and see whether it is the right formula, which is a different and easier task than checking whether a stated result is right.

The general rule: use the model for the setup, never for the sum. It is good at knowing that annual energy is capacity times hours times capacity factor. It is unreliable at multiplying them.