Measuring a gap you are not allowed to record

The awkward prerequisite

Everything in this module requires knowing which group each person belongs to. In many real settings you do not, and often you are not permitted to ask. European data protection law treats racial or ethnic origin, religion and health as special categories with a high bar for processing. Some jurisdictions prohibit collecting ethnicity data in employment altogether. In India, asking an applicant's caste in a private hiring process is socially and legally fraught. And people decline to answer.

So the practitioner is stuck between two duties: do not collect sensitive data, and do not run a system with unmeasured disparate impact. This lesson is about how that is actually handled, and where each approach breaks.

Route one: ask, but keep it apart

The cleanest answer is voluntary self-identification, collected separately from the decision process, stored by a different team, and used only for aggregate monitoring. This is what mature equal-opportunity monitoring looks like in the UK and elsewhere: the recruiting manager never sees it, the analyst never sees the individual decision-maker's identity, and the output is a quarterly table.

Its weaknesses are practical rather than conceptual. Response rates are partial and non-random — people who fear discrimination are less likely to disclose, which biases the very measurement meant to protect them. And separation has to be enforced by architecture, not by policy, because a field in the same database as the application is a field somebody will eventually join to it.

Route two: infer it, carefully

Where self-identification is impossible, some regulators use statistical imputation. The best-documented method is Bayesian Improved Surname Geocoding: combine the probability distribution of race given a surname, from census surname tables, with the distribution given a residential location, and update one with the other. The US Consumer Financial Protection Bureau has used BISG for fair-lending analysis, and published its methodology.

Be precise about what this gives you. It produces a probability, not a fact, and it is accurate enough for population aggregates and unreliable for individuals. Its accuracy varies sharply by group — it works better where surnames are distinctive and worse where they are shared across communities. In India, surname-based inference of caste or religion is technically feasible and enormously error-prone, and the social consequences of a wrong inference attached to a person are severe.

The rule that keeps this ethical: imputed group membership is for measuring aggregates and must never re-enter the decision pipeline, be stored against an individual, or be shown to anyone who makes decisions about that person. A system that infers religion in order to check for discrimination, and then leaks that inference into a customer record, has created the harm it was auditing for.

Route three: measure the proxies instead

Sometimes the honest move is to abandon the protected characteristic and monitor what you can observe. Selection rates by postcode, by school type, by whether the applicant's address is rural, by whether their name is transliterated, by the language of the submitted document. None of these is race or caste, and all of them are correlated enough to reveal a structural gap.

This is weaker evidence and it is unlikely to satisfy a regulator. It is also achievable today, using data you already hold, and it catches the large problems. A hiring funnel with a fifteen-point gap by school type has something to look at, whatever the protected-characteristic figures would have shown.

Route four: audit the model rather than the outcomes

You can also test the system directly, without any real people's data. Generate paired applications identical in every substantive respect and differing only in the signal you want to test — the name, the pronoun, the mention of a particular university or a career gap. Run them through and compare the scores.

This is the AI version of the correspondence-audit method that has been used in labour economics for decades, sending matched CVs with different names to real employers. Applied to your own system it is fast, cheap, requires nobody's sensitive data, and gives a clean causal answer about the model's behaviour rather than a correlational one about outcomes. Its limit is that it tells you about the model in isolation, not about the whole process the model sits inside, where humans, sourcing channels and self-selection also act.

The position to hold

"We cannot measure it because of privacy" is sometimes true and is very often a convenient stopping point. Before accepting it, work through the four routes. At least one of them is available in almost every setting, and the last one requires no personal data at all.

And if you genuinely cannot measure disparity in a system that decides about people, that is not a neutral state. It is a known unknown, and it belongs in writing, in the risk register, next to the decision to deploy anyway.