Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR’s results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to “settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn.” While those factors may not apply in “many realistic, economically relevant settings” involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.
This article confirms my own experiences with AI. I spend a lot more time reviewing, reprompting, and tweaking than I save on coding. Having to double check or fight it to get what I want is not a time saver. Not to say that it doesn’t save time when it is right, but the thing that I never seem to get across to proponents of AI is that anytime I need to reprompt or refine, I have lost. I have officially wasted time at this point compared to simply referencing the documentation. Unless I’m generating a significant portion of code which only needs minor tweaks. I’m generally not saving time.