It feels like progress
At eleven at night I cannot measure what a session is worth. I can only feel it. One study shows how far that feeling can be from the result.
The trial
In 2025 the research group METR ran a randomized trial with 16 experienced open-source developers.1 They worked on 246 real issues in their own repositories. Each issue was randomly assigned: AI tools allowed, or not.
With AI allowed, the issues took 19% longer.
The slowdown is not what I find interesting. Before the study, the developers expected AI to make them 24% faster. After it, having done the work, they still believed they had been about 20% faster.
They were slower, and they went home sure they had been faster.
Do not read it as “AI makes you slower”
METR says so itself. In February 2026 the group wrote that the result is out of date and that developers are probably sped up more by now.2 They also explained why they could not simply run the study again: 30 to 50% of the developers held back tasks they did not want to do without AI. People no longer accept working without the tools.
It was sixteen people with the tools of early 2025. What I take from it is the distance between the two numbers, between what the session felt like and what it delivered. I have no reason to think newer tools closed that distance.
Where the feeling comes from
The Berkeley researchers I wrote about in where did the evening go saw the same thing from the outside.3 Having an AI partner gave people a sense of momentum. What the researchers observed during that momentum was attention switching, frequent checking of what the AI had produced, and a growing number of open tasks.
A group at IBM describes the cost on the other side.4 From interviews with developers and researchers, they describe a tiredness that comes from steering, reading and verifying AI output without a break. They call it vibing fatigue. One of their three suggestions is to build nudges into the tools, to step away when a session has stopped adding value. It is an article based on interviews, without numbers, and the authors say that testing the remedies is still ahead.
Eleven at night
Put the two together. The feeling of progress is strongest while I am in the session. The judgement of whether it was worth it needs distance. Late in the evening I have plenty of the first and none of the second, and I am tired, which helps neither.
That is the state I described in my own story: it felt like progress, and a lot of it was fear of falling behind.
Move the judgement
I do not trust myself to judge a session at night, so I stopped asking. The question goes to the morning.
In the app I built, an idea that arrives after closing time is spoken in and put away. I do not see it again that night. In the morning it is on the screen with one question: does this still matter? Then I keep it or drop it.
I have no numbers on how many ideas survive that question. The app does not collect anything, from me or from anyone else. What I can say is that the question is easier to answer after a night than in the middle of a session.
“One more prompt” becomes one last prompt.
One Last Prompt is an iPhone app, and it is out now on the App Store.
Download on the App StoreZsolt Csizmadia builds One Last Prompt.
Sources
- Joel Becker, Nate Rush, Beth Barnes and David Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, METR, 10 July 2025. metr.org
- METR, “We are Changing our Developer Productivity Experiment Design”, 24 February 2026. metr.org
- Laura Counts, “AI promised to free up workers’ time. UC Berkeley Haas researchers found the opposite”, Berkeley Haas Newsroom, 18 February 2026. newsroom.haas.berkeley.edu
- J. He, N. Fulton and H. Strobelt, “Vibing fatigue: Navigating the hidden cognitive costs of vibe coding”, IBM Think Insights, 11 September 2026. ibm.com