> What about hallucinations? Honestly, since GPT-3.5, I haven’t noticed ChatGPT or Claude doing a lot of hallucinating.
See this is what I don't get about the AI Evangelists. Every time I use the technology I am astounded at the amount of incorrect information and straight up fantasy it invents. When someone tells me that they just don't see it, I have to wonder what is motivating them to lie. There is simply no way you're using the same technology as me with such wildly different results.
> There is simply no way you're using the same technology as me with such wildly different results.
Prompting styles are incredibly different between different people. It's very possible that they are using the same technology that you are with wildly different results.
I think learning to use LLMs to their maximum effectiveness takes months (maybe even years) of effort. How much time have you spent with them so far?
> I have to wonder what is motivating them to lie.
Most of these people who aren't salesmen aren't lying.
They just cannot tell when the LLM is making up code. Which is very very sad.
That or they could literally be replaced by a script that copy/pastes from stack-overflow. My friend did that a lot and it definitely helped features ship but doesn't make maintainable code.
> When someone tells me that they just don't see it, I have to wonder what is motivating them to lie. There is simply no way you're using the same technology as me with such wildly different results.
I don’t know what technology you are using but I know that I am getting very different results based on my own prompt qualities.
I also do not really consider hallucinations to be much of an issue for programming. It comes up so rarely and it’s caught by the type checker almost immediately. If there are hallucinations it’s often very minor things like imagining a flag that doesn’t exist.
A lot of you guys, including the author, will respond with these "you're holding it wrong" comments. But you never give examples of actual prompts that are somehow significantly different to what I use. The author gives a very small handful of his example prompts and I don't see anything in them that's fundamentally different to what I've tried. If anything his prompts are especially lazy compared to what I use and what I have read as best practice among "prompt engineers":
"is this idiomatic C?"
"not just “how does X work”, but follow-up questions like “how does X relate to Y”. Even more usefully, you can ask “is this right” questions"
"I’ll attach the entire file or files to Copilot chat, paste the error message, and just ask “can you help?”"
> A lot of you guys, including the author, will respond with these "you're holding it wrong" comments. But you never give examples of actual prompts that are somehow significantly different to what I use.
It’s hard to impossible to discuss these things without a concrete problem at hand. Most of the prompt is the context provided. I can only talk from my experience which is, that how you write the prompt matters.
If you share what you are trying to accomplish I could probably provide some more appropriate insights.
I've shared a few hundred examples of how I'm using this stuff, with full prompt and response transcripts. You can find them linked from items on these tags on my blog - they're usually gists.
A lot of times they probably are holding it wrong. These things aren’t mind readers. You have to provide proper context and clear asks. They are just another computer interface to learn to use.
And they aren’t perfect, but they sure can save a lot of time once you know how to use them and understand what they are and aren’t good at.
See this is what I don't get about the AI Evangelists. Every time I use the technology I am astounded at the amount of incorrect information and straight up fantasy it invents. When someone tells me that they just don't see it, I have to wonder what is motivating them to lie. There is simply no way you're using the same technology as me with such wildly different results.