Asking three times, and reading the disagreement

The confidence signal you are not being shown

The one thing you most want to know is which parts of an answer the model was sure about. No consumer product tells you. There is no confidence figure on the screen, and the writing is equally assured throughout.

There is a way to get a rough version of it, and it costs three times as much and about two minutes.

Ask the same question three times, in three separate, fresh conversations, and compare the answers.

Why fresh conversations, specifically

This matters, so do not shortcut it. Pressing "regenerate" or asking again in the same chat does not give you an independent sample — the earlier answer is sitting in the context window, influencing the next one. You get variations on a theme rather than three draws from the underlying distribution.

Three new chats, identical prompt pasted into each. Then compare, not on the prose, but on the specifics: names, numbers, dates, section references, the direction of the recommendation.

Reading the result

Where the three agree, the model's distribution was concentrated. The answer is stable.

Where they differ, the distribution was flat and it was effectively choosing. Those are the load-bearing points to check, and they are usually a small fraction of the text.

A worked example. You ask three times what notice period a supplied contract requires, and get 30 days, 30 days, 90 days. You now know precisely which clause to read yourself, and you know it in two minutes rather than by rereading the whole agreement. This is the technique's best use: not to produce an answer, but to aim your attention.

The limitation that must not be skipped

Agreement is not evidence of truth.

If the training data consistently contained a wrong thing, or if your question strongly cues one continuation, all three runs converge on the same wrong answer with the same confidence. Three identical answers tell you the model is stable. Stability and correctness are different properties, and nothing in this procedure connects them.

So the rule is asymmetric, and worth stating carefully:

• Divergence is informative. It reliably marks a soft spot.
• Convergence is not reassurance. It removes one reason to worry and supplies no positive evidence.

Anybody who tells you that three matching answers means an answer is verified has swapped a consistency check for a truth check.

Two models beat one model three times

A stronger version: ask two different providers' models the same question. Their errors are less correlated, because they were trained on different data with different methods, so a disagreement is more likely to reflect genuine difficulty than a quirk of one model's training.

The free path here is good. Several providers offer a free tier sufficient for occasional cross-checks, and a local model through Ollama gives you a genuinely independent second opinion that costs nothing per query and sends nothing anywhere. A local seven-billion model is weaker than a frontier model, which for this purpose does not matter much: you are not asking it to be right, you are asking whether it says something different.

When it is worth the cost

Three runs cost three times as much and take three times as long, so this is not a default. Reserve it for:

• A number or a date that will be acted on.
• A claim you intend to put in front of somebody senior.
• An answer that surprised you.
• Anything where you noticed yourself thinking "that seems convenient".

For the routine rewrite of an email, one run is fine, and the consequences of a flat distribution are a slightly different adjective.

Majority voting, and its ceiling

The automated version of this — run five times, take the most common answer — appears in the research literature as self-consistency, and on multi-step arithmetic and reasoning benchmarks it produces real gains. It works there because the task has one correct answer that wrong paths scatter away from, so the correct answer is the only one that accumulates.

It works far less well on questions of fact about the world, where a wrong answer that the model reliably prefers wins the vote every time. Voting is not a truth procedure; it is a way of finding the model's favourite answer, which is only useful when the model's favourite answer is right for structural reasons.

Use disagreement to find where to look. Do not use agreement to decide you no longer need to.