The study where they were slower and felt faster

A result worth sitting with

In 2025 the research organisation METR ran a randomised controlled trial with sixteen experienced open-source developers working on their own repositories — codebases they had contributed to for years. Across 246 real tasks, each was randomly assigned to allow or disallow AI tools.

Before starting, the developers expected AI to speed them up by about 24%. Afterwards, they estimated it had sped them up by about 20%. The measured result was that tasks with AI allowed took 19% longer.

They were slower, and they believed they were faster, after doing it.

This is a small study on a specific population — expert developers on mature codebases they know intimately — and the authors are careful about generalising. Other studies in other settings find large genuine gains. The point is not that AI does not help. It is that the feeling of speed and the fact of speed came apart, and the people experiencing it could not tell.

Where the time actually goes

The measured explanation is mundane and generalises far beyond programming. Time saved on producing a first draft is spent on:

Reading and understanding output you did not write. Comprehension of unfamiliar work is slower than production of familiar work, for most experienced people.

Verification. Checking whether it is right, which for anything consequential means checking every specific.

Repair. Fixing the parts that are subtly wrong, which is often harder than writing it correctly, because you must first locate the error inside something plausible.

Re-prompting. Adjusting, retrying, negotiating. Individually small and collectively substantial.

Context switching. Moving between writing, reviewing and instructing has a cost each time.

Why does it feel faster? Because the visible, effortful part — the blank page — disappears, and effort is what people use to estimate time. The remaining work is reading and checking, which feels like less work even when it takes longer.

The jagged frontier

The other essential result comes from a 2023 field experiment with 758 consultants at Boston Consulting Group. On tasks inside the model's capability, consultants using AI completed 12% more tasks, 25% faster, with quality rated 40% higher. On a task deliberately designed to sit just outside it — where the correct answer required combining data and interview material in a way the model got wrong — those using AI were about 19 percentage points less likely to reach the right answer.

The authors called it a jagged technological frontier: the boundary between what these systems do well and badly is irregular, unmarked, and not predictable from how difficult a task looks to a human. Two tasks that seem equally hard sit on opposite sides.

That jaggedness is why intuition about where to trust the tool is unreliable, and why the gains and losses can be large in both directions in the same job.

What to do about it

Measure end to end, not on the draft. The honest unit is time from starting the task to having something you would sign. Teams that measure generation time record enormous gains and cannot explain why nothing finishes sooner.

Time a few tasks both ways. Two hours of self-experiment beats a year of impressions. Do the same class of task with and without, and record the total.

Map your own frontier. Keep a running note of task types where it reliably helps and where it reliably wastes time. This is personal and stable, and it is the highest-return document most professionals could keep.

Notice the fluency trap. The output that is easiest to accept is well-formatted, confident and complete-looking, which is a property of the writing rather than of the correctness.

Watch for the expertise inversion. The productivity studies consistently find the largest gains for less experienced workers and small or negative effects for the most experienced — which fits both this and the customer-support results in the next lesson.

The honest summary

Across the credible studies, gains are real and highly variable: large for novices and for routine writing, smaller or negative for experts on complex work in domains they know well. Nobody should conclude either that these tools do not work or that a stated percentage applies to them.

What everyone can conclude is that self-report is not evidence here. Your sense of how much time this saved you has been measured, at least once, against a stopwatch, and lost.