ISSUE 007
They were 19 percent slower and believed they were 20 percent faster
The last entry in this pillar reported a randomized trial finding AI made developers 26 percent more productive. Here is another randomized trial, run by an independent nonprofit, finding the opposite. I spent a while trying to work out which one to believe before realising that was the wrong question.
An idea worth thinking about
Nobody in the room could tell which way it went
Before the study began, METR asked the developers to forecast the effect. They predicted AI would cut their completion time by 24 percent. Outside experts were more bullish still: economists predicted a 39 percent speedup, machine learning experts 38 percent.
Then the developers did 246 real tasks, half with AI allowed and half without. Afterwards, having lived through it, they estimated AI had made them about 20 percent faster. The measured result was that tasks took 19 percent longer.
The forecast missed by more than forty percentage points and got the direction wrong. More striking, the after-the-fact estimate also got the direction wrong — by people who had just done the work, in code they had maintained for years on average.
Set aside every argument about model versions and task types for a moment, because this part doesn't depend on them. If experienced professionals cannot tell whether a tool helped or hurt them on work they know intimately, then "it feels faster" is not evidence of anything. And "it feels faster" is the basis on which almost every small business in the country is currently deciding what AI is worth.
A tool I disappeared into
A paper that spends most of its length trying to explain away its own result
The full METR paper is free, and the thing that makes it worth opening rather than reading a summary is the middle section. Having found a result that surprised them, the authors collected and evaluated evidence for twenty different properties of their setup that could plausibly have produced the slowdown — project size, quality standards, the developers' prior familiarity with the repositories, how the tools were used.
That is what confidence in a result is supposed to look like: not defending it, but systematically listing the ways you might be wrong and checking each one. They also collected screen recordings, so the analysis rests on watching what people actually did rather than on what they reported.
Read it as a model for evaluating any tool in your own business. Before concluding a change helped, write down five reasons the improvement might be something else — seasonality, a different mix of work, one unusually good month — and see which ones you can rule out. Most of us skip straight to the conclusion, which is exactly the habit the developers in this study fell into.
Learn something
Bounded tasks and open-ended work are different questions
Put the randomized trials side by side and a pattern appears that neither one states on its own. An earlier controlled experiment from Microsoft Research and MIT found developers 55.8 percent faster — on a bounded task, specified in advance, verified by automated tests. The firm experiments in the last entry found 26 percent faster on ordinary task completion across thousands of developers. METR found 19 percent slower on open-ended work in mature repositories the developers already knew deeply.
The variable that moves is the shape of the task, not the quality of the tool. Where the goal is clearly specified, the context is small enough to hand over, and correctness is checkable, the gains are large. Where the work is open-ended, the relevant context lives in a codebase and a set of standards the person carries in their head, the handover cost exceeds the help.
The translation to a small business is direct and doesn't require any software at all. Drafting a first version of a standard letter is a bounded task with a checkable result. Rewriting the pricing on your own service agreement, where the relevant context is eleven years of what clients have asked for, is not. Expect a different answer from the same tool on those two jobs, and don't let the result on one set your expectations for the other.
What we’re seeing
Sixteen developers, 246 real tasks, and a 19 percent slowdown nobody noticed
METR — a nonprofit that evaluates AI systems, with no product to sell here — recruited 16 experienced open-source developers and randomized 246 of their real tasks: bugs, refactors, feature work in mature repositories they had contributed to for about five years on average, some with over a million lines of code. On a random half, AI tools were allowed. On the rest they weren't. With AI allowed, tasks took 19 percent longer.
Three caveats belong immediately next to that number, and the first comes from METR itself. The study tested tools available between February and June 2025 — mainly Cursor Pro running Claude 3.5 and 3.7 Sonnet — and METR now labels the result historical, saying it doesn't necessarily reflect current tools or current workflows. Anyone quoting this as evidence about today's models is using it against the authors' own instruction.
Second, sixteen people is a small sample, however many tasks they completed. Third, and most important, the finding is specific: experienced developers, in codebases they know well, doing open-ended work. It does not show AI slows down beginners, or new projects, or anything outside software.
So how does this sit with the 26 percent gain in the last entry? They aren't in contradiction once you notice they measured different things. That study covered 4,867 developers across large firms, on ordinary task completion, and found the gains concentrated among the least experienced. This one covered deep experts on work where their own accumulated knowledge was the main asset. The consistent story across both is that AI substitutes for knowledge you don't have and interferes with knowledge you already carry.
Which leaves the perception result as the one I'd actually act on, because it survives every caveat above. These developers were wrong about the direction of the effect after experiencing it. If you're deciding whether an AI tool is earning its subscription, your impression is not evidence. Time something. Count something. Pick one recurring task, measure two weeks with and two weeks without, and find out — because the people best positioned to know, in the best-designed study available, could not tell.
One question
Have you ever actually timed yourself with and without AI?
Not estimated — timed. Reply with what you found, or reply that you haven't. I expect almost nobody has, including me.
More in this pillar as it's written.
— Shana
Sources link to the original publisher, never to a summary.