from HaraldvonBlauzahn@feddit.org to programming@programming.dev on 04 Jul 07:45
https://feddit.org/post/32163938
Mid-last year, I had GPT (maybe 5.0 or 5.1) try to find the source of a bug. Naturally, this code didn’t have tests and git bisect wouldn’t work, and it was a UI interaction bug for which I’m not even really qualified to write a test for, so I asked Codex to bisect between dates X and Y to find the commit that introduced this bug. Codex immediately told me the offending commit was after this date range (which couldn’t possibly be correct). On telling Codex this was wrong, it then told me some commit that was obviously also not the offending commit once or twice. On telling it those were wrong, it then told me the offending commit was some plausible looking commit. When I asked it to prove or disprove its theory, it told me that it wrote a test and confirmed that the alleged commit was the breaking commit.
I then asked it to show me by making a video with the full developer end-to-end stack in the normal browser test environment. It claimed that it didn’t have permissions to do that (which was a lie), but it could make video of the execution of the repro before and after the commit in playwright with the appropriate test code. The video was convincing and showed the feature working properly before the commit and failing to work after the commit. Something about this didn’t feel right, so I tried reproducing the issue by hand before and after the commit and found out that the whole thing was a fabrication. The video made it look like Codex had reproduced the bug, but it was an artificial browser environment that was designed to create a fake repro, not the real environment.
Like I said, because this was non-ironically such a great experience, I immediately thought to myself, “how can I get more of this?”
#programming
threaded - newest
So, do I get this right: People try it. It does not work better than a coin toss. But because it outputs human language - the hallmark of intelligence, until now -, people think there is still something onto it, get hooked, and continue to use it for code generation?
From the blog:
Similar to gambling, that might well apply to agentic coding as a whole.
Also, gambling is addictive, and addictive things distort reality.
The mistake is in assuming the AI is perfect and will be correct all the time.
If you’re relying on it to be correct and not verifying its output, you’re doing it wrong.
It’s like doing a search and finding posts in forums. Sometimes what you find is wrong or not appropriate for your situation.
AI doesn’t replace your need to do critical thinking.
I think unless you are a total beginner, proper verification will frequently take about as long, or longer than writing it yourself.
Like it’s harder to read even good and correct legacy code, than to write new code.
Codebase with no tests, check.
What the hell are they doing in bugfixing an UI bug, when they are “not qualified” to write a test for it. Anyhow, not competent enough for the codebase you’re working on - check.
So, instead of asking the LLM to e.g. create a proper reproduction as a test case, asking it to bisect, which the author claimed that I wasn’t possible, for some reason. So, also adding can’t bisect on his own, and can’t prompt properly, check and check.
So, the author realized it hallucinates. The author asks for video proof (instead of a fucking test, again). The author is surprised it generated him a video of exactly what they wanted to see, only creating it in a different way than they wanted to.
This reads like “I have close to zero clue what I’m doing, I also don’t really know how to achieve what I want properly, and now I’m making a salty blog post that my magical text microwave didn’t fix my half-assed description of a problem”. Like, honestly, what the hell was the expectation here?
This. If you’re at a point that you’re arguing with an LLM, you’ve already lost. Just start a new thread with a different approach, don’t make an article about your inability to use an LLM.
Does the name “Dan Luu” say anything to you? Do you know his blog ?
In general, for Dan Luu I wouldn’t assume he is not competent enough.
And besides that, what is the point of LLMs / GenAI if you need to be an expert in everything it touches to handle it correctly? If you are an expert, you can already do it yourself.
Also, if one needs to be an expert in the topic to get good results, this creates more doubt that the “ibtelligence”, “reasoning”, and “capabilities” of these things are not in reality the intelligence of the user, since he does the real work of discerning fabrication and accidental good output.
Reminds me on that old story of the smart horse “Hans” which could do math, indicating the result with is hooves. But it turned out he could do it only when his owner was around - the horse had learned when his owner agreed with the result and indicated that unconciously.
Reminder that “a coin toss” is only bad odds for problems with binary and equally likely outcomes. And that’s rarely the case for anything that an LLM is used for. A 50% chance of saving an hour of work a couple times a day are pretty good odds. If I have a problem which a candidate solution is easy to verify, it’s often more effective to let an LLM investigate it for some time before I do so, and only jump in if it fails.
There have been several little fixes I’ve done in minutes with an agent that would take me at least an hour to manually investigate, write a solution, test, and refactor. So yes, there is something to it, but you need to know how to use it. Keep arguing in a thread after noticing hallucinations is a clear sign the author doesn’t know how to use an LLM.
And another aspect is: You can, of course, engineer reliable things from unreliable components. Much of hardware works like that. Even my bicycle needs to have two brakes, for redundancy. Cloud computing and things like distributed databases and file systems works like that, at the price of massive complexity.
I can see that some intelligent people are attracted by the challenge. Like a juggler who tries to keep more balls in the air.
But for generating code and algorithms, and the price being intelligibility and maintainability - is this a good idea?
One key piece of getting good results from LLMs is not to have them do anything you can’t do yourself. I catch AI doing weird things all the time and just fix it or have AI fix it accordingly.
Left to its own devices, AI will generally produce bad output over a large enough size. This is why I argue AI will ultimately not replace developers. Even the best models I’ve seen just make more sophisticated errors. The product must be reviewed and fixed by someone who actually understands how to write it.
The question is more the threshold at which AI costs more than is gained in efficiency. As we’ve seen a lot of folks don’t gain efficiency, that’s obvious in some cases. Yet, other folks do see gains and the question is whether this is a domain issue or a technique issue.