The outcomes you never get to see

You only learn about the people you said yes to

Here is the problem that quietly invalidates a large share of published model evaluations, and it has nothing to do with algorithms.

A lender approves an applicant and later observes whether they repaid. A lender rejects an applicant and observes nothing, ever. So the training data contains outcomes only for approvals. The same holds for hiring — you learn how the people you hired performed, never how the rejected candidates would have. For bail, parole, admissions, insurance underwriting, content moderation appeals. In every one of these, the label exists only for the cases the previous system let through.

This is the selective labels problem, and it has three consequences that compound.

Your accuracy figure is measured on a biased sample. The model looks good on approvals because approvals are where it was confident. Its performance on the population it rejected is unmeasured and unmeasurable from this data.

Errors in one direction are invisible. A false negative — a good applicant rejected — never generates a record. False positives, by contrast, arrive as defaults on a report. So the loss function the organisation feels is asymmetric, and it drifts towards rejecting more, forever, with no counter-signal.

The next model trains on this. Yesterday's decisions become today's training data, so the new model learns to replicate the old one's blind spots, and its confidence in them grows because they are now confirmed by data.

The loop closes

Put those together and you get a self-reinforcing system that looks, from inside, like it is improving.

Predictive policing is the sharpest case. Send more officers to a district, and more incidents are recorded there — not because more crime occurs but because recording requires an officer present. Those records train the next model, which sends more officers. The loop has no external correction, and what the model has learned to predict is police deployment. A 2016 RAND evaluation of Chicago's Strategic Subject List found the programme had no detectable effect on the outcome it was meant to reduce, while people on the list were more likely to be arrested — which is what a system that directs attention rather than prevents harm should be expected to produce.

Recommendation systems run the same loop more benignly. A model shows you what it predicts you will click, you click it, and the model's confidence rises. You never see the material it did not show, so neither does it. Whether that is a problem depends on the stakes, but the structure is identical.

How to break it

The fix requires giving up something, which is why it is rare.

Random exploration. Approve a small random fraction of applicants the model would have rejected — one or two per cent — and record what happens. This is the only method that produces genuine information about the rejected region, and it is what a scientist would do. It costs money, in expected defaults, and it needs to be signed off as a deliberate research expense rather than a bug. Some lenders do exactly this and call it a champion-challenger or holdout portfolio.

Exploit natural experiments. If cases are assigned to different decision-makers with different strictness, the lenient ones effectively supply the exploration. Economists use this heavily; the leading study of bail decisions did exactly this, comparing outcomes across judges who differed in release rates.

Compare against the previous decision-maker on the region where they disagreed. If the model and a human reviewer disagree on 8% of cases, route those to a fuller process and record the result. It is cheaper than random exploration and it only illuminates the boundary, not the deep interior.

At minimum, say so. If none of the above is possible, the evaluation report must state that performance is measured only on the accepted population and that the rejection error rate is unknown. That sentence is the difference between an honest evaluation and a misleading one, and it costs nothing but nerve.

The general shape

Whenever a system's decisions determine what data it later sees, you have a loop, and the loop will make it more confident rather than more correct. Ask two questions of any deployed model: what does this system prevent me from observing? and what would it take to observe it anyway?

Those questions have nothing to do with machine learning. They are the reason clinical trials randomise, and the reason a shop that only surveys its current customers never learns why anyone left.