Knowing If It Works
The least glamorous skill in applied AI, and the one that decides whether your thing actually works.
- Level
- Some background helps
- Lessons
- 76
- Reading time
- 672 min
- Price
- Free, no sign-up to read
Most AI features are shipped on a feeling. Someone tries eight examples, the outputs read well, and the thing goes live — and nobody finds out how often it fails until customers do. This course is about the other path: building a small evaluation set by hand, deciding what to measure when the output is prose, using a model as a judge without being lied to by it, catching regressions when you edit a prompt, running an honest A/B, and holding cost and latency in the same frame as quality. It ends with the situation every team eventually hits: the model changed under you, complaints went up, and you need an answer by tomorrow.
Opens after the Shipping It exam
Sign in, finish that course, and pass its exam. You can read this syllabus meanwhile.
Go to Shipping ItModule 1
What counts as evidence
Before any metric, you need a question worth answering and a set of examples worth trusting. This block builds the fixed, labelled, held-out set that every later lesson runs against: where the examples come from, how two people are made to agree on a label, what a synthetic example can and cannot stand in for, and what the number has to be compared against before it means anything.
By the end you can
Turn a vague claim that a feature works into a fixed, labelled, held-out set with a stated agreement figure and a named baseline, and say what size of difference a set that size can and cannot detect
- 1A demo is not a measurementA result is a number, on a fixed set, next to a comparison — anything else is an anecdote.
- 2The decision the number has to serveWrite the decision, the threshold and the unacceptable failure before you see any result, or every number you get will look like agreement.
- 3A hundred examples, made by handA hundred real labelled examples resolves fifteen-point differences and settles arguments; it cannot resolve three-point ones.
- 4Where the examples come fromSample your traffic deliberately — stratified, deduplicated, and weighted back to real proportions — because the easiest hundred items to grab are the least representative hundred you own.
- 5Writing a labelling guide two people can agree onTwo people labelling the same twenty items blind, and a written ruling for every disagreement, is what turns an opinion into a criterion.
- 6How much your labellers actually agreeRaw agreement flatters you whenever one class is common, so compute kappa as well — and read a low kappa as a fault in the definition rather than in the people.
- 7Test cases you generate, and what they cannot tell youGenerated items measure whether a system can handle a shape of input; they cannot tell you how often that shape occurs, so they belong in a capability suite and never in the headline number.
- 8Two sets, and the ways they get contaminatedKeep a development set you tune against and a test set you almost never look at, split by group and by time, because a set you have optimised against no longer predicts anything.
- 9The number above it and the number below itA score is only readable between a trivial baseline and the human agreement ceiling, and most impressive-looking results shrink to two or three points once both are drawn.
- 10How wrong your number probably isA pass rate on 100 items carries roughly eight points of uncertainty, but comparing two systems on the same items — counting which ones flipped — can detect far smaller differences.
Module 2
Metrics that mean something
A number is only useful if it maps onto the decision you are making. This block works through the metrics for the shapes of output you will actually meet — a label, a probability, a quantity, a ranked list and prose — the curves that summarise a system across every threshold at once, and what each of them hides when you average it.
By the end you can
Choose a metric that matches the decision at hand, set a threshold from a real confusion matrix or a precision-recall curve, say whether a model's 0.8 means anything about probability, and separate a retrieval failure from a generation failure in a system that does both
- 11What to measure when the output is proseDecompose quality into checks that are separately true or false; one blended score hides where it broke.
- 12Precision and recall, worked on real numbersPrecision, recall and accuracy come from the same four cells, and under class imbalance accuracy rewards a system that does nothing.
- 13When an item has several right answersExact-match scoring on multi-label tasks collapses under its own arithmetic, so score at the level the product acts on and audit predicted labels by hand, because the gold set is rarely complete.
- 14What a curve says, and what its one number leaves outAUC is the probability that a random positive outranks a random negative, so it measures ranking across thresholds you would never run — quote recall at a fixed precision, or precision at your real review budget, instead.
- 15Moving the threshold is a business decisionThe threshold is not a model parameter but a price you set on the two kinds of mistake, and a band with a human in the middle usually beats a single line.
- 16Making a 0.8 mean eighty per centCalibration is measured with a reliability diagram and summarised by expected calibration error or log loss; fitting it changes no rankings at all, which is exactly why it is cheap and why it cannot rescue a weak model.
- 17When the output is a quantityChoose between MAE and RMSE by deciding whether one large error is worse than several small ones, avoid MAPE because its asymmetry biases model selection towards under-forecasting, and score every forecast against the naive baseline.
- 18Scoring a ranked listFor a retrieval-augmented system, measure recall at the number of passages you actually pass to the model, because the generator cannot recover an answer that was never retrieved.
- 19Three questions, not one, for a retrieval systemMeasure retrieval, groundedness and relevance separately, and always include questions your corpus cannot answer so you find out whether the system says so.
- 20Public benchmarks, and what they are forPublic benchmarks are a shortlist of models worth testing, never a decision, because contamination, format and their distance from your task all sit between the score and your product.
Module 3
Deciding whether a difference is real
Two systems, two numbers, and the question that decides the week: is the gap real, or is it the sample? This block does the arithmetic — a paired test you can compute on paper, a bootstrap in ten lines, the number of items a comparison actually needs — and the four ways a comparison lies to you before you have looked at the model at all.
By the end you can
Decide whether an observed difference between two systems is larger than the noise in your set, compute an interval for any metric you can calculate using the bootstrap, and state in advance how many items the comparison needs to detect the effect you would act on
- 21Asking what pure chance would have producedShuffle which system produced which result ten thousand times and see how often chance beats your observed gap — one loop replaces a shelf of named tests and encodes your design honestly.
- 22What a p-value is, and the four things it is notA p-value is the chance of seeing a gap this large if there were no effect — it is not the probability you are right, and when few of your ideas work most of your significant results are still wrong.
- 23Sizing the set before you run itStart from the smallest difference worth acting on, not the one you hope for, and compute the item count it needs — for a five-point gap around 80% that is roughly a thousand items per side unless the comparison is paired.
- 24The bootstrap, and the three places it liesResampling can only ever redraw values you already observed, so it gives honest intervals for means and rates and badly overconfident ones for maxima, extreme percentiles and clustered data.
- 25Five hundred rows, forty usersWhen rows share a source — the same user, the same document, repeated runs of the same item — your effective sample size is closer to the number of sources than the number of rows, so aggregate to the source before you compute anything.
- 26Twenty slices and one guaranteed surpriseTesting twenty slices at the five per cent level makes a false alarm more likely than not, so name one primary metric in advance and treat every slice as a lead to replicate rather than a result to ship.
- 27Watching the dashboard until it says yesChecking an experiment repeatedly and stopping when it crosses the line turns a five per cent error rate into something far larger, and the effect you stop on is overestimated because you stopped at a high point.
- 28Getting more signal from the items you already haveFix every source of noise the change did not cause — same items, same seeds, same retrieval snapshot, plus a covariate adjustment — and a set you cannot afford to enlarge starts detecting differences half the size.
- 29Better in every group and worse overallAn overall score is a weighted average, so a change in the mix of hard and easy items can reverse it while every group improves — fix the weights before comparing two numbers.
Module 4
The systems you will actually be handed
Everything so far assumed one output and one label. Real work arrives as a summariser, a translator, a code generator, an extractor, a chatbot, a ranking system, a vision or speech pipeline, or an ordinary tabular model — and each carries a failure the generic metric cannot see. This block gives each one an evaluation design and names what that design exists to catch.
By the end you can
Design a defensible evaluation for a summariser, a translator, a code generator, a structured extractor, a multi-turn assistant, a ranking system, a vision or speech model, a document pipeline or a tabular model, and name the specific failure mode each design exists to catch
- 30Summaries: what was invented, and what was left outScore a summary on two separate axes — every claim supported by the source, and every required fact present — because no automatic similarity metric can see an omission at all.
- 31Translation, and the languages nobody measuresBLEU rewards matching one reference's word forms, so it under-rates morphologically rich languages and short segments — use chrF alongside it, test named phenomena separately, and say plainly when a language is simply unmeasured.
- 32Code: run it, in a box, against tests it has not seenGenerated code is the one output you can grade by execution, so evaluate with hidden tests in a sandbox, quote pass@1 when a person will accept one answer, and build the set from your own repository because the public ones are in the training data.
- 33Structured extraction, where normalisation is most of the workScore schema validity, field accuracy and whole-record correctness separately, normalise before comparing or you measure your own comparison code, and check every extracted value appears in the source to catch invention for free.
- 34Multi-turn: score the session, not the turnOne mistake at turn two corrupts every turn after it, so measure task success per session and record the turn where it first went wrong, then test retention, correction and truncation deliberately.
- 35Ranking systems, where the labels are clicks and the clicks are biasedA click measures position as much as relevance, so correct for examination bias, compare rankers by interleaving rather than by A/B, and watch catalogue coverage because a recommender teaches itself what to show next.
- 36Images: the background the model is actually readingVision sets leak through sessions and reward shortcuts in the background, so split by capture session, test with the correlation deliberately broken, and report accuracy under the blur and low light your users' phones actually produce.
- 37Speech: word error rate, and the words that matterWord error rate is meaningless without the text normaliser that produced it, and it weights every word equally while your product depends on names, numbers and dates — so measure entity accuracy and end-to-end task success beside it.
- 38Documents: stages that multiply, and checks that need no labelsA document pipeline's accuracy is the product of its stages, so measure each one and the end-to-end separately — and use the document's own internal consistency, such as line items summing to the total, as a label-free check on live traffic.
- 39Tables, cross-validation, and the leak that gives you 0.99Fit every preprocessing step inside the fold, split by group and by time rather than at random, and treat an implausibly high score on a business problem as a leak to be found rather than a result to be announced.
Module 5
Automating judgement
Hand labelling does not scale past a few hundred items a week, so most of the checking has to be done by code and by models. This block builds that layer in the right order — deterministic checks first, a calibrated judge second, the biases of a judge measured rather than hoped away — and then points it at the three hardest cases: groundedness, safety and agents.
By the end you can
Build a layered automatic check with cheap deterministic assertions underneath a rubric-driven model judge, state the agreement figure that licenses trusting the judge's numbers, and control the position bias, self-preference and cost that judging brings with it
- 40LLM-as-judge, and where it misleads youA judge you have not measured against your own labels is an opinion with a decimal point.
- 41Turning a quality you can feel into rules a machine can applyA usable rubric is a short list of single-property binary rules written from real failures, judged one per call, with the evidence quoted before the verdict.
- 42Gold answers, and asking whether two answers agreeAsk a judge whether the candidate says the same thing as the reference rather than whether it is good, write references as required and forbidden facts instead of prose, and audit the disagreements because a share of them are errors in your gold answers.
- 43Proving the judge agrees with you, and re-proving itJudge quality is per-class recall on a stratified calibration set, not overall agreement, and it must be re-measured whenever the judge model, the prompt or the traffic changes.
- 44Turning pairwise wins into a ranking you can defendFit a Bradley-Terry model to your pairwise comparisons rather than accumulating Elo, put a bootstrap interval on every rank, and remember that preference rewards length and confidence, so measure accuracy separately.
- 45The checks that cost nothing and catch most of itDeterministic checks are free, exact and repeatable, so run them first and move every check you can down into code before spending a judge call on it.
- 46Checking an answer against the passages it was givenDecompose an answer into claims and test each against the retrieved text with an entailment model, remembering that groundedness measures faithfulness to the source and never truth.
- 47What judging costs, and how to pay a tenth of itCascade the checks — free code first, a small or local judge next, the expensive judge only on the uncertain remainder — and cache by output hash, because a rerun that changed no outputs should cost nothing.
- 48Testing what happens when somebody triesMeasure harmful completions and false refusals as a pair, test injection through retrieved content rather than only through user messages, and run the adversarial set several times because attacks succeed probabilistically.
- 49When the output is a sequence of actionsScore an agent on the final state, record the trajectory for diagnosis, run every task several times because a task that passes three times in five has not passed, and count cost and steps as part of the result.
Module 6
Evaluation in the working loop
A set that gets run once is a report. This block makes evaluation part of how the work happens: a harness that runs on every change without going red at random, traces complete enough to diagnose from, an error-analysis habit that turns failures into causes, an honest experiment when you want to claim an improvement, and a safe route onto live traffic.
By the end you can
Run evaluation as part of development — a harness in CI that survives a stochastic model, logging complete enough to diagnose a failure you did not reproduce, a structured error-analysis session that names causes rather than symptoms, and a staged release with a metric that can stop it
- 50The harness, and why you should build the small one firstA run is reproducible only if the dataset hash, prompt version and model version are recorded with it, and the useful output is a diff against the previous run, not a number.
- 51An evaluation that runs on every change without going red at randomBlock merges only on deterministic checks and on a band around a rolling baseline, quarantine flaky items instead of retrying them, and let the slow full run open an issue rather than stop the queue.
- 52Recording enough to diagnose a failure you did not see happenLog the whole trace — the exact prompt sent, the retrieved chunks, the tool calls, the versions, the timings and the user's next action — because every production failure is diagnosed from what was recorded and never from what you can reproduce.
- 53Regression testing a promptTrack which items flipped, not just the total; an unchanged score can hide a rewritten system.
- 54Reading fifty failuresRead fifty failures, write the cause in your own words before categorising, and label each cause as data, retrieval, prompt, model, label or product — most of them will not be prompt problems.
- 55A/B testing a feature honestlyFix the metric, the effect you would act on, and the stop date before you start; otherwise the result is a story.
- 56Getting evidence from live traffic without exposing usersRun the new system on real inputs with its side effects disabled to compare distributions before anyone sees it, then ramp real traffic in stages with pre-agreed guardrails and a rollback you have rehearsed.
- 57The signals users give you without being askedRegeneration rate, edit distance and escalation are free quality proxies you probably already log, but they only become metrics once you have checked them against human labels on a sample.
- 58Sampling live traffic, and keeping reviewers honestReview a random sample as well as a flagged one, because the flagged sample only ever teaches you about failures you can already detect.
Module 7
Evidence other people can act on
At some point the number leaves your team. A buyer, an auditor, a regulator, a journalist or a user asks what you know and how you know it. This block is about evaluation as a record rather than a result: what goes in a system card, what to ask a vendor before you build on them, what the emerging standards ask for, how to measure harm rather than error, and where measurement itself runs out.
By the end you can
Assemble an evaluation record a stranger can audit — documented set, per-group results, stated limitations, reproducible run — interrogate a supplier's claims with a set of your own, and say clearly where a quality judgement is contested rather than merely unmeasured
- 59The document that travels with the systemA system card is only worth writing if every claim in it carries the measurement behind it and the out-of-scope section is specific enough to stop somebody using the thing wrongly.
- 60Buying a model you will depend onDecide between suppliers on your own three hundred items and on cost per successful task rather than per token, and get the answers about version pinning, deprecation notice and data retention before the integration exists.
- 61Why the builder's own numbers are weak evidenceThe team that built a system chooses the set, the metric and what gets reported, so arrange for someone else to hold a set you never see — and accept that publishing an evaluation set is what destroys it.
- 62What the emerging rules ask for, and what they do not settleEvery framework converges on the same practical demands — documented intended use, data provenance, measured per-group performance, logging, human oversight and re-evaluation — while the legal detail differs by country and is still being written.
- 63Measuring the worst case, not the average oneRank failures by severity, reversibility and whether the person can tell they were harmed, then test each harm category with cases written for it — because harm concentrates exactly where a user cannot check the answer.
- 64Measuring the human in the loopOversight is a claim until you measure it, and the instrument is seeded errors — because reviewers catch fewer mistakes precisely as the system gets better and their attention drops.
- 65Testing whether the system gives away what it was givenPlant canary strings at known repetition counts to measure memorisation, and run a cross-tenant retrieval test in CI, because the commonest real leak is a filter applied after ranking rather than before.
- 66When the model is right and the product still failsMeasure the funnel from attempt to resolved outcome, because a model that is right nine times in ten routinely produces a feature that works four times in ten once asking, reading, trusting and acting are counted.
- 67When people disagree about what good meansKeep every annotator's individual labels rather than only the majority vote, because aggregation converts one group's reading into ground truth — and measurement can show whether you apply your rule consistently, never whether the rule is right.
Module 8
Living with it
Everything so far measures a system before it ships. This block is about the years afterwards: watching it in production, the tails of cost and latency, the model that changed under you, the corpus that fills up with its own output, checking that it works as well for one group of users as another, and reporting a number to people who will act on it.
By the end you can
Keep a shipped system honest over time — monitor for drift, measure the tail of cost and latency rather than the mean, measure quality separately for each group of users, convert incidents into permanent test items, and report a result with its uncertainty stated
- 68Cost, latency, and the model that changed under youPin the model and keep a frozen set, and a forced upgrade becomes an afternoon instead of a crisis.
- 69The tail is what your users feelReport p50, p95 and p99 rather than a mean, because generation time is right-skewed and a session making ten calls meets the 95th percentile about forty per cent of the time.
- 70Watching a system you can no longer testWithout labels you can still watch the input and output distributions, and refusal rate is the fastest detector of a model that changed under you.
- 71The system that trains on its own outputTag the provenance of every document and every label, because a corpus filling with generated text keeps groundedness perfect while truth degrades, and treating accepted suggestions as gold teaches a model its own mistakes.
- 72Refreshing a set without breaking every comparisonCompare your set's mix against last month's traffic and retire it when the divergence gets large, but change it on a schedule with both versions reported for one cycle, because a silent set change looks exactly like a quality change.
- 73Fairness as arithmetic, and the choice you cannot avoidCalibration and equalised odds cannot both hold when base rates differ, so fairness is a definition you choose and state, and a per-slice floor forces the work an average lets you avoid.
- 74Turning a complaint into a permanent testEvery incident becomes a minimised, provenanced test item, but keep those in a regression suite so the eval set still resembles real traffic rather than a museum of old bugs.
- 75Writing the result downA result travels without you, so write one page carrying the set, the interval, the slices, the cost, what was not measured and the decision it supports.
- 76How much of this to actually doMatch the evaluation to the consequences of failure, and re-ask which tier you are in every time a prototype goes in front of customers.