LLM Idolatrine: Why no one wants your agentic PRs
(blog.magosomni.com)
from moto@programming.dev to programming@programming.dev on 08 Mar 13:56
https://programming.dev/post/46873518
from moto@programming.dev to programming@programming.dev on 08 Mar 13:56
https://programming.dev/post/46873518
#programming
threaded - newest
With more and more places pushing LLM generated code I’m getting real review fatigue. Code reviews used to be very reasonable and you could be very mindful about other persons code. Now it’s starting to be just LLM generated code which makes reviewing tedious as you are reading the same sloppy stuff in hopes that you can build your mental model of the changes to codebase before new LLM slop bomb hits the PR queue.
Surely we can come up with networks of trust for this sort of thing, so that you don’t have to deal with PRs from people with no references.
Everyone starts off without references, and there’s already less of a pipeline from user to helpful contributor to fellow maintainer than most projects want without having to add more chokepoints. There isn’t a solution without downsides while there are people using LLMs.
That’s true, but as a maintainer you could encourage those helpful maintainers to triage issues from regular users.
I think the real benefit would come from taking a user’s reputation into account across projects.
At the end of the day you can’t have low effort pull requests, and expect maintainers to look at everything. It’s the same spam problem as in any other domain.
This works if you have the luxury to select the people whose PRs you review, but in a corporate environment you just don’t have that option. I would love to just reject obvious LLM code, but it’s not going to keep me employed. Instead I’m stuck at figuring out how to meaningfully review LLM changes and how to manage the mental model with these rapid changes.
One huge issue is that LLMs do weird and stupid things differently than how humans do them.
If you’ve developed an eye for reading human-made changes, you’re not necessarily going to recognize new and surprising failure modes as easily. It’s literally harder than regular code review.
Humans with modern tooling, for example, rarely hallucinate field/class/method/object names because non-spicy autocomplete keeps them on the rails. LLMs seem much more willing to decide the menu bar is .menuBar and not .topMenu, probably because their training corpus is full of the former.
Exactly.
Another problem with LLMs is that they are actually useful in some tasks and they can generate very good quality code if you’re diligent enough developer. I also have built personal tools with them, but I don’t have the knowledge of the code the LLM has hallucinated which means that before I would push this code forward I will have to basically familiarise myself with the code in a way how a code review works.
The knowledge you gain from this is also different from that of actually writing and running the code yourself. I have seen people who use LLMs to write commit messages which is the last thing you should do. Commit messages are probably the only places were we can meaningfully store the knowledge gathered during development and the more I see LLM commits the more I lose hope.
Humans are also lazy. I won’t make 5,000 line changes just because I feel like refactoring while adding a field to a query. A LLM might, and I don’t want to read all of that.
I’m afraid it’s worse than just moving the bottleneck to review/due-diligence.
C-suits don’t like to admit this, because it challenges the assumption that job roles are all about input and output with no messy internal state to worry about, but…
When you write some code, the code isn’t the only thing that you make. You also develop a mental model of how the code works, how you wish it would work, the distance between those two, and some sense of the possible paths that could get you there.
That mental model is ultimately the more important part for the long-term health of the project. Coding is more an activity of communication between people; having an artifact that tells the computer what to do is almost an incidental side-effect of successful communication.
Yeah that’s a real good point. I focused a lot on the short term issues of agentic slop but you’re right the long term impact of this is going to be staggering.
100%. Something I wanted to touch on in my post but cut because I couldn’t weave it in well was more on the relationship with “The Problem” and “The Code”. Pre-AI coding acted as a forcing function for the problem context. It’s really hard to effectively build software without understanding what you’re ultimately driving towards. With agentic coding we’re stripping that forcing function out.
Institutional knowledge is already something that’s been hard to quantify and value by C-suites and now you’re ripping out the crucial mechanism for building that.
I see a lot of memes with people being like “in 5 years we’re just gonna press yes and not understand what the agent is doing” and I keep thinking “why do people think this is funny?”
I understand why you cut it. There is a lot that you could dig into there, and you would really have to go into detail it you wanna convince the suits that can only think in terms of KPIs.
The forcing function is a good way of looking at it. We’re not really doing the same thing but faster. We’re doing less, and that naturally takes less time. AI made understanding optional.
I kinda wonder to what extent coding is actually a domain that AI is uniquely “good at” vs simply how well coding activities are able to survive on artifacts which are produced but not understood.
It seems like something we were already exceptionally capable of tolerating, compared to other sectors.
<img alt="" src="https://midwest.social/pictrs/image/56d7eee3-5bf6-4272-8d7b-d00b1ae94779.jpeg">
It looks like this was briefly touched in the article, but LLMs don’t learn shit.
If I tell you your use of a list is dumb and using a set changes the code from
O(n)toO(1)and cuts out 15 lines of code, you probably won’t use a list next time. You might even look into using a deque or heap.If your code was written by a LLM? You’ll “fix” it this time (by telling your LLM to do it) and then you’ll do it again next time.
I’m sorry, but in the latter case, not only are you mentally handicapping yourself, but you’re actively making the project worse in the long term, and you’ve got me sending out resumes because, and I mean this in the politest way possible, but go fuck yourself for wasting my time with that review.
I’ve seen lots of people describe LLMs as a junior developer, but for this very reason, they are worse than a junior. I can train a junior and build them into senior, so the time spent correcting mistakes is still worthwhile. I can’t train an LLM, so correcting mistakes is just pointless.
I was told I have to adopt agentic AI or I can’t be promised to stay around for too long.
Meanwhile I’m catching all the slop the other devs submit.
I don’t get it. This is will destroy people already bad at coding in the long run, then people like me who really don’t want to implement an agentic (whatever that is) but find all the flaws in the PR, get burned in the long run.
Find a company that values engineers. The one you’re at doesn’t, whether they understand that or not.