METR AI Coding Research Inconclusive Because Dev Participants Refused to Complete Tasks Without AI (metr.org)
from brianpeiris@lemmy.ca to programming@programming.dev on 12 Mar 04:18
https://lemmy.ca/post/61665147

Selected developer quotes:

“I’m torn. I’d like to help provide updated data on this question but also I really like using AI!” — a developer from the original study early-2025 when asked to participate in the late-2025 study.

“I found I am actually heavily biased sampling the issues … I avoid issues like AI can finish things in just 2 hours, but I have to spend 20 hours. I will feel so painful if the task is decided as AI-disallowed.” — a developer from the new study noting selection effects when choosing what tasks to include in the study.

“my head’s going to explode if I try to do too much the old fashioned way because it’s like trying to get across the city walking when all of a sudden I was more used to taking an Uber.” — a developer from the new study noting selection effects when choosing what tasks to include in the study.

#programming

threaded - newest

hperrin@lemmy.ca on 12 Mar 05:35 next collapse

That’s fucking bleak. Maybe this is the dev equivalent of AI psychosis. AI paralysis?

[deleted] on 12 Mar 06:14 next collapse
.
maegul@lemmy.ml on 12 Mar 06:17 next collapse

I mean, it makes sense that it’s addictive, right?

I also suspect it’s one of those things that just naturally splits people. For some, the addictiveness and appeal just don’t make sense. For others it’s irresistible.

It’s part of the reason why I’m so doomer on the state of things, from a generally anti-AI/sceptical perspective. There’s just something compulsive that this kind of tool triggers in many people.

pupbiru@aussie.zone on 12 Mar 10:09 next collapse

the other insidious part about it is that pretty much everyone agrees: experience is critical to ensuring AI isn’t just producing slop… 30% of the time you get something that’s not just working, but well architected

now when you get AI to do things, even if you go with the assertion that it’s quicker (which a lot of the time i doubt: task choice is also critical for effectively using AI to generate useful outputs), you’re grinding down on your experience… not only are you learning less, but you’re also letting your reasoning skills degrade because you’re not using them (this is a pretty well-documented effect in standard neuroscience afaik)

imo, only use AI in situations where you’d put a library in, because the level of abstraction from the problem solving is similar

amju_wolf@pawb.social on 12 Mar 14:27 collapse

Our brains are wired to seek out efficiency, so yeah, it’s not really surprising.

pixxelkick@lemmy.world on 12 Mar 06:39 next collapse

More like “why the fuck would I walk all the way across the city now that I own a car”

Once you find out how a bunch of boring bulk tasks can be automated away and 20 hours of work turns into 20 minutes, you really dont wanna go back to the old way.

If someone asks me to code in C# without my IDE in notepad, can I do it? Sure

But it fuckin sucks losing all your hotkeys and refactor quick actions and auto complete and lsp error checking…

Would you find it weird for someone to state they’d rather use an IDE than not when coding, because it saves so much time/effort?

hperrin@lemmy.ca on 12 Mar 06:46 next collapse

More like, “why would I, an avid runner, who has trained in running for over a decade, run five blocks when I can just drive”.

spartanatreyu@programming.dev on 12 Mar 06:52 next collapse

More like “why the fuck would I walk all the way across the city now that I own a car”

That’s a bad analogy.

Using an LLM for coding gives you an initial speed up with the trade off being skill atrophy, and a build up of cognitive debt in the project.

A better analogy would be the Greek government before their national debt crisis. It would have been better to invest in themselves, not lie about their own finances, and not kick the can down the road. But they kept lying and kicking the can down the road because it was easier in the short term. Of course, we all know how that turned out in the end.

pixxelkick@lemmy.world on 12 Mar 14:39 collapse

You only skill atrophy if you go and perk off playing video games while the agents cook.

If you actually are productive and spend that freed up time working on tasks the agents cant do fast and easy, aka, the hard stuff, you instead will improve your skill even faster as now you are spending most of your time on the important tasks and not wasting 95% of your workday on easy boilerplate stuff anyone with 2 braincells can pump out.

Senal@programming.dev on 12 Mar 16:07 collapse

What you’re describing is skill atrophy, it’s just in skills you don’t value.

It’s also skill acquisition/reinforcement for skill you do value.

rimu@piefed.social on 12 Mar 07:01 next collapse

20 hours of work turns into 20 minutes

The gains, where they exist, are nowhere near that much. In some cases, it makes developers slower (even though they think they’re a bit faster):

we find that when developers use AI tools, they take 19% longer than without - AI makes them slower.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

pixxelkick@lemmy.world on 12 Mar 14:36 collapse

Have you actually read the study? People keep citing this study without reading it.

To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work.

They grabbed like 8 devs who did not have pre-existing set up workflows for optimizing AI usage, and just throw them into it as a measure of “does it help”

Imagine if I grabbed 8 devs who had never used neovim before and threw them into it without any plugins installed or configuration and tried to use that as a metric for “is nvim good for productivity”

People need to stop quoting this fuckass study lol, its basically meaningless.

Im a developer using agentic workflows with over 17 years experience.

I am telling you right now, with the right setup, I weekly turn 20 hour jobs into 20 minute jobs.

Predominantly large “bulk” operations that are mostly just boilerplate code that is necessary, where the AI has an existing huge codebase to draw from as samples and I just give it instructions of “see what already exists? implement more of that following <spec>”

A great example is integration testing where like 99% of the code is just boilerplate.

Arrange the same setup every time. Arrange your request following an openapi spec file. Send the request. Assert on the response based on the openapi spec.

I had an agent pump out 120 integration tests based on a spec file yesterday and they were, for the most part, 100% correct, yesterday. In like an hour.

The same volume of work would’ve easily taken me way longer.

SpicyLizards@reddthat.com on 12 Mar 10:15 collapse

Well, i agree, but AI isn’t equal to an IDE. AI is dangerous, it will give credible results that look purposeful, but not necessarily meaningful. It’s a reviewers and thus a maintainers nightmare.

Also, as others have said, the very real risk of skill atrophy and real imposters who can build something but cannot understand how it works.

And just at a major environmental cost.

Luminous5481@anarchist.nexus on 12 Mar 13:39 collapse

Might have to do with them not wanting to work because they admit to cutting their wages to a third of what they were previously

thedeadwalking4242@lemmy.world on 12 Mar 10:49 next collapse

I think the change in 150hr to 50hr probably had a large impact on this study as well. I’m not sure why that’s not more focused on.

Luminous5481@anarchist.nexus on 12 Mar 12:54 collapse

Why would they want to talk about cutting wages lol

Luminous5481@anarchist.nexus on 12 Mar 12:50 collapse

“We cut their pay to a third of what it was previously and they refused to do any work themselves”

Lmao that’s some real capitalist bullshit right there