It's totally plausible that AI codegen produces more bugs. It still seems important to familiarize yourself with these tools now though, because that bug count is only ever going to go down. These tools are here to stay.
I see this stranded claim get trotted out in the corporate world all the time as a refutation to "The AI broke". I fail to understand who the invisible audience for this is supposed to be.
Also, the FOMO argument for why one should use X as much as they can comes across as scammy. Not that I believe you're trying to scam anyone. Though the person you originally got this from may have been!
> Also, the FOMO argument for why one should use X as much as they can
Who said that you should use it as much as you can?
> Though the person you originally got this from may have been!
Borderline offensive to claim that I could only parrot someone else's sales pitch rather than come to my own conclusions based on technical merit and an evaluation of progress of the past few years.
I don't think that these specific tools are here to stay. Categorically, yes, but I expect there to be big changes in interfaces and how they work in the next five years.
I don't think spin up time on LLM technology requires as much investment as the hype claims, nor do I think that the current methodology will be as long lived as they think. Sitting out may be detrimental in the now, but I expect that developers that do so will be able to catch up just fine.
> I don't think that these specific tools are here to stay.
I agree, we're still in an experimental phase so these tools are definitely not in their final form, I meant more that the LLM core of the tool is here to stay, and familiarizing yourself with how to use LLMs to solve every day problems is a good time investment.
I've experimented mostly with the raw chat interfaces for programming, circuit design, querying documentation, and troubleshooting OS issues rather than spending time googling, and they've proven incredibly valuable even with the basic chat interface. I've also hit many of the issues/limitations other people have reported, sometimes wasting some time going down a rabbit hole, and sometimes I was convinced that the LLM was wrong in diagnosing some issue, but it turned out to be correct in the end.
Despite the difficulties, it remains the case that I wouldn't have started or finished some projects without them.
Did the models from 2 years ago produce more bugs, fewer bugs or the same bugs as today's models? Do you think next years AI models will produce the same number of bugs, more bugs, or fewer bugs?
> Did the models from 2 years ago produce more bugs, fewer bugs or the same bugs as today's models?
Is anyone actually tracking that with a methodology not prone to fine-tuning? Specifically, I know a lot of the tests have the problem that you can train the AI to pass the test, so a higher score is not indicative of overall higher performance. I'm not actually being rhetorical here to make a point; I'm genuinely interested if anyone has derived a methodology that gives confidence behind these claims.
(Aside: Its not a huge stretch to claim that they're getting better, but it mostly seems anecdotal from this point, or using methods that have the above problem I stated)
I'm assessing my own experience here. I occasionally check new models on some kinds of problems I'm familiar with but that are not common programming challenges, like arrow-based FRP abstractions but written in C# rather than Haskell. I've noticed considerable improvements on their ability to translate such abstractions idiomatically.
