When the decision is about a person

The most consequential use, and the least examined

Sorting job applications. Scoring loan or tenancy applications. Grading student work. Ranking staff for promotion or redundancy. Flagging benefit claims or insurance claims as suspicious. Writing performance reviews.

These are the uses with the largest effect on human lives and, in most organisations, the least scrutiny — because they arrive dressed as efficiency and because the people affected are not in the room.

Why the bias is not a bug

Amazon built a CV-screening tool trained on the CVs the company had received over a decade and on which of them had been hired. It learned the pattern in that history. It penalised CVs containing the word "women's" — as in "women's chess club captain" — and downgraded graduates of two women's colleges. Engineers adjusted it. They could not be confident it had not found other proxies. The project was scrapped in 2018.

Nothing malfunctioned. The system did exactly what it was built to do: reproduce past decisions. If your past decisions were skewed, a tool that predicts them is a machine for continuing the skew, and now the skew has a number attached and an appearance of objectivity.

This is the harder version of the problem, and it survives every obvious fix. Remove gender from the data and the model finds proxies: sports, a gap in employment, a school, a postcode, a name, the phrasing conventions of somebody writing in a second language. You cannot remove a protected characteristic from data by deleting the column, because the world encodes it in twenty other places.

Three specific dangers in professional use

Automated rejection at the top of the funnel. The dangerous decision is usually the first cut, not the final choice. Nine hundred applications reduced to forty by a model, then forty reviewed conscientiously, is not a human-reviewed process. It is an automated process with a human-reviewed tail, and 860 people were rejected by something nobody can explain.

Explanations that are stories. Ask a model why it scored someone low and you get a fluent paragraph. That paragraph was generated after the fact and is not a description of what happened inside the computation. It looks exactly like accountability and provides none. Under several legal regimes, a person subject to an automated decision has a right to a meaningful explanation, and a generated one does not satisfy it.

Writing about people. Performance reviews, references, incident reports, safeguarding notes. Beyond bias, there is a specific dishonesty here: the document claims to be your assessment of a colleague. If most of the words are not yours, it is not, and everyone reading it in five years will treat it as if it were.

What responsible use looks like

There are legitimate uses. They tend to have this shape:

• Assist the person, never replace them. Draft interview questions. Summarise a long application against criteria you wrote. Suggest what a policy says about a situation. Then a person reads and decides.
• Never let it make the cut. Ranking and shortlisting are the decision, whatever the interface calls them.
• Check the same input twice. Take a real application, change only the name to one of different apparent gender or ethnicity, and see whether the output moves. If it does, you have found something and you must stop. This test takes four minutes and almost nobody runs it.
• Look at outcomes, not intentions. Compare selection rates across groups for the process as a whole, before and after adoption. Bias shows up in outcomes; it is invisible in the reasoning.
• Keep the record. What was used, on what, who decided, on what grounds. If challenged in two years, that file is the whole of your defence.
• Tell people. In several jurisdictions you must. Everywhere, a candidate who discovers afterwards that a machine sorted them and nobody said so is a candidate who has lost trust in you permanently.

The question that settles most cases

Before using AI in any decision about a person, ask:

Could I explain to this person, to their face, exactly how this decision was reached — and would that explanation satisfy them?

If the honest answer is "I would have to say a system scored you and I do not know why", you have your answer, and no efficiency argument outweighs it. This is the one area of the course where the correct advice is more conservative than what your employer may be considering, and where the cost of being wrong is paid entirely by somebody who never agreed to it.