LLMs have default personalities - shaped by RLHF and other post-training methods. There is a lot of variance to it, but variance from one LLM to another is much higher than that within the same LLM.
If you want an LLM to retain the same default personality, you pretty much have to use an open weights model. That's the only way to be sure it wouldn't be deprecated or updated without your knowledge.
I'd argue that's "underlying hidden authorial style" as opposed to what most people mean when they refer to the "personality" of the thing they were "chatting with."
Consider the implementation: There's document with "User: Open the pod bay doors, HAL" followed by an incomplete "HAL-9000: ", and the LLM is spun up to suggest what would "fit" to round out the document. Non-LLM code parses out HAL-9000's line and "performs" it at you across an internet connection.
Whatever answer you get, that "personality" is mostly from how the document(s) described HAL-9000 and similar characters, as opposed to a self-insert by the ego-less name-less algorithm that makes documents longer.
That's a lie people repeat because they want it to be true.
People evaluate dataset quality over time. There's no evidence that datasets from 2022 onwards perform any worse than ones from before 2022. There is some weak evidence of an opposite effect, causes unknown.
It's easy to make "model collapse" happen in lab conditions - but in real world circumstances, it fails to materialize.
What makes you look at existing AI systems and then say "oh, this totally isn't capable of describing a problem or figuring out what's actually wrong"? Let alone "this wouldn't EVER be capable of that"?
> What makes you look at existing AI systems and then say "oh, this totally isn't capable of describing a problem or figuring out what's actually wrong"?
I wouldn't say they're completely incapable.
* They can spot (and fix) low hanging fruit instantly
* They will also "fix" things that were left out there for a reason and break things completely
* even if the code base fits entirely in their context window, as does the complete company knowledge base, including Slack conversations etc., the proposed solutions sometimes take a very strange turn, in spite of being correct 57.8% of the time.
That's about right. And this kind of performance wouldn't be concerning - if only AI performance didn't go up over time.
Today's AI systems are the worst they'll ever be. If AI is already capable of doing something, you should expect it to become more capable of it in the future.
It's a self-evident truth. Even if today, at this very moment AI hits a hard plateau and there's nothing we can do to make AI better, ever, then this still holds true. It simply means we'll keep what we have right now. Any new model will be a step back and thus be discarded. So what we have today is the worst, and the best it will ever be. But barring that extremely unlikely scenario, like GPT-3 to GPT-4 and Claude 3 to Claude 4, we will see improvements (either incremental or abrupt) over the coming weeks/months/years. Any failed experiments will never see the light of day and the successful experiments will become Claude X or GPT X, etc.
By now, the main reason people expect AI progress to halt is cope. People say "AI progress is going to stop, any minute now, just you wait" because the alternative makes them very, very uncomfortable.
Well, to use the processor analogy, with models we reached the situations where the clocks can't do that much more. So the industry switched to multiplying cores etc. but you can actually see the slope plateauing. There are wild developments for the general public like the immediate availability of gpt-oss-120b that I'm running on my MBP right now, there is Claude Code that can work for weeks doing various stuff and being right half of the time, that's all great, but we can all see development of the SOTA models has slowed down and what we are seeing are very nice and useful incremental improvements, not great breakthroughs like we had 3-4 years ago.
(NB I'm a very rational person and based on my lifelong experience and on how many times life surprised me both negatively and positively, I'd say the chance of a great breakthrough occurring short term is 50%, but it has nothing to do or cannot be extrapolated from the current development as this can go any way actually. We already had multiple AI winters and I'm sure humanity will have dozens if not hundreds of them still.)
Plateauing? OpenAI's o1 is revolutionary, less than a year old, and already obsolete.
Are you disappointed that there's no sudden breakthrough that yielded an AI that casually beats any human at any task? That human thinking wasn't obsoleted overnight? That may or may not happen yet. But a "slow" churn of +10% performance upgrades results in the same outcome eventually.
There's only this many "+10% performance upgrades" left between ChatGPT and the peak of human capabilities, and the gap is ever diminishing.
I think the reason people feel it's plateauing is because the new improvements are less evident to the average person. When we saw GPT-4 I think we all had that "holy shit" moment. I'm talking to a computer, it understands what I'm saying, and responds eloquently. The Turing test, effectively. That's probably the most advanced benchmark humans can intuitively assess. Then there's abstract maths, which most people don't understand, or the fact that this entity that talks to me like an intelligent human being, when left to reason about something on its own devolves into hallucinations over time. All real issues, but much less tangible, since we can't relate it to behaviours we observe or recognize as meaningful in humans. We've never met a human that can write a snake game from memory in 20 seconds without errors, but can't think on its own for 5 minutes before breaking down into psychosis, which is effectively what GPT-4 was/is. After the release of GPT-4 we strayed well outside of the realm of what we can intuitively measure or reason about without the use of artificial benchmarks.
> By now, the main reason people expect AI progress to halt is cope. People say "AI progress is going to stop, any minute now, just you wait" because the alternative makes them very, very uncomfortable.
OK, so where is the new data going to come from? Fundamentally, LLMs work by doing token prediction when some token(s) are masked. This process (which doesn't require supervision hence why it scaled) seems to be fundamental to LLM improvement. And basically all of the AI companies have slurped up all of the text (and presumably all of the videos) on the internet. Where does the next order of magnitude increase in data come from?
More fundamentally, lots of the hype is about research/novel stuff which seems to me to be very, very difficult to get from a model that's trained to produce plausible text. Like, how does one expect to see improvements in biology (for example) based on text input and output.
Remember, these models don't appear to reason much like humans, they seem to do well where the training data is sufficient (interpolation) and do badly where there isn't enough data (extrapolation).
I'd love to understand how this is all supposed to change, but haven't really seen much useful evidence (i.e. papers and experiments) on this, just AI CEOs talking their book. Happy to be corrected if I'm wrong.
That's not true. And trust me, dude, it scares the living ** out of me, so I wish you were right. Next-token prediction is the AI-equivalent of a baby flailing its arms around and learning basic concepts about the world around it. The AI learns to mimic human behavior and recognize patterns, but it doesn't learn how to leverage this behavior to achieve goals. The pre-training is simply giving the AI a baseline understanding of the world. Everything that's going on now, getting it to think (i.e. talking to itself to solve more complex tasks), or getting it do do maths or coding, is simply us directing that inherent knowledge it's gathered from its pre-training and teaching the AI how to use it.
Look at Claude Code. Unless they hacked into private GitHub/GitLab repos... (which, honestly, I wouldn't put beyond these tech CEO's, see what CloudFlare recently found out about Perplexity as an example), but unless they really did that, they trained Claude 4 on approximately the same data as Claude 3. Yet for some reason its agentic coding skills are stupidly enhanced when compared to previous iterations.
Data no longer seems to be the bottleneck. Which is understandable. At the end of the day, data is really just a way to get the AI to make a predicion and run gradient descent on it. If you can generate for example a bunch of unit tests, you can let the AI freewheel its way into getting them to pass. A kid learns to catch a baseball not by seeing a million examples of people catching balls, but instead by testing their skills in the real world, and gathering feedback from the real world on whether their attempt to catch the ball was successful. If an AI can try to achieve goals and assess whether or not its actions lead to a successful or a failed attempt, who needs more data?
Fundamentally the bottleneck is on data and compute. If we accept as a given that a) some LLM is bad at writing eg rust code because there's much less of it on the Internet compared to say react js code but that b) the LLM is able to generate valid rust code and c) the LLM is able to "tool use"the rust compiler and a runtime to validate the rust it generates, and iterate until the code is valid, and finally d) use that generated rust code to train on, then it seems that barring any algorithmic improvements in training, that the additional data should allow later versions of the LLM to be better at writing rust code. If you don't hold a-d to be possible then sure, maybe it's just AI CEOs talking their book.
The other fundamental bottleneck is compute. Moore's law hasn't gone away, so if the LLM was GPT-3, and used 1 supercomputer's worth of compute for 3 months back in 2022, and the supercomputer used for training is, say, three times more powerful (3x faster CPU and 3x the RAM), then training on a latest generation supercomputer should lead to a more powerful LLM simply by virtue of scaling that up and no algorithmic changes. The exact nature of the improvement isn't easily back of the envelope calculatable, but even with a laymen's understanding of how these things work, that doesn't seem like an unreasonable assumption on how things will go, and not "AI CEOs talking their book". Simply running with a bigger context window should allow the LLM to be more useful.
Finally though, why do you assume that, absent papers up on arvix, that there haven't and won't be any algorithmic improvements to training and inference? We've already seen how allowing the LLM to take longer to process the input (eg "ultrathink" to Claude) allows for better results. It seems unlikely that all possible algorithmic improvements have already been discovered and implemented. Just because OpenAI et Al aren't writing academic papers to share their discovery with the world and are, instead, preferring to keep that improvement private and proprietary, in order to try and gain a competitive edge in a very competitive business seems like a far more reasonable assumption. With literal billions of dollars on the line, would you spend your time writing a paper, or would you try and outcompete your competitors? If simply giving the LLM longer to process the input before user facing output is returned, what other algorithmic improvements on the inference side on a bigger supercomputer with more ram available to it are possible? Deepseek seems to say there's a ton of optimization still as of yet to be done.
Happy to hear opposing points of view, but I don't think any of the things I've theorized here to be totally inconceivable. Of course there's a discussion to be had about diminishing returns, but we'd need a far deeper understanding is the state of the art on all three facets I raised in order to have an in depth and practical discussion on the subject. (Which tbc I'm open to hearing, though the comments section on HN is probably not the platform to gain said deeper understanding of the subject at hand).
We are nowhere near the best learning sample efficiency possible.
Unlocking better sample efficiency is algorithmically hard and computationally expensive (with known methods) - but if new high quality data becomes more expensive and compute becomes cheaper, expect that to come into play heavily.
"Produce plausible text" is by itself an "AGI complete" task. "Text" is an incredibly rich modality, and "plausible" requires capturing a lot of knowledge and reasoning. If an AI could complete this task to perfection, it would have to be an AGI by necessity.
We're nowhere near that "perfection" - but close enough for LLMs to adopt and apply many, many thinking patterns that were once exclusive to humans.
Certainly enough of them that sufficiently scaffolded and constrained LLMs can already explore solution spaces, and find new solutions that eluded both previous generations of algorithms and humans - i.e. AlphaEvolve.
That's exactly how it works. Every input of AI performance improves over time, and so do the outcomes.
Can you damage existing capabilities by overly specializing an AI in something? Yes. Would you expect that damage to stick around forever? No.
OpenAI damaged o3's truthfulness by frying it with too much careless RL. But Anthropic's Opus 4 proves that you can get similar task performance gains without sacrificing truthfulness. And then OpenAI comes back swinging with an algorithmic approach to train their AIs for better truthfulness specifically.
Like when a relationship is obviously over. Some people enjoy the ending fleeting moments while others delude themselves that they just have to get over the hump and things will go back to normal.
I suspect a lot of the denial is from the 30 something CRUD app lottery winner. One of the smart kids all through school, graduated into a ripping CRUD app job market and then if they didn't even feel the 2022 downturn, they now see themselves as irreplaceable CRUD app genius. Something understandable since the environment has never signaled anything to the contrary until now.
My psychological reaction to what's going on is somehow pretty different.
I'm a systems/embedded/GUI dev with 25 years of C++ etc., and nearly every day I'm happy and grateful to be the last generation to get really proficient before AI tools made us all super dependant and lazy.
Don't get me wrong, I'm sure people will find other ways to remain productive and stand out from each other - just a new normal -, but I'm still glad all that mental exercise and experience can't be taken away from me.
I'm more compelled to figure out how I can contribute to making sure younger colleagues learn all the right stuff and treat their brains with self-respect than I feel any need to "save my own ass" or have any fears about the job changing.
You made me think of the role of mental effort/exercise. In parts of the western world, we are already experiencing a large increase in dementia/alzheimer and related. Most of it is because we are doing better with other killers like heart etc, and many cancers also. But is said that mental activity is important to stave off degenerative diseases of the brain. Could widespread AI trigger a dementia epidemic? It would be 30 years out, but still...
What? LLMs do benefit from economies of scale. There are a lot of things like MoE sharding or speculative decoding that only begin to make sense to set up and use when you're dealing with a large inference workload targeting a specific model. That's on top of all the usual datacenter economies of scale.
The whole thing with "OpenAI is bleeding money, they'll run out any day now" is pure copium. LLM inference is already profitable for every major provider. They just keep pouring money into infrastructure and R&D - because they expect to be able to build more and more capable systems, and sell more and more inference in the future.
Single LLM company can't stop investing into better systems and marketing of them, because there is no moat and customers will flee to the ones who do invest. It's free after all. So it is a closed loop which can't be broken, companies can but won't switch to "just inference". And with investing, all of the LLM companies are losing money a lot (on the LLMs specifically).
> So for the most part access to AI is way cheaper than it will be in the next 5-10 years.
That's a lie people repeat because they want it to be true.
AI inference is currently profitable. AI R&D is the money pit.
Companies have to keep paying for R&D though, because the rate of improvement in AI is staggering - and who would buy inference from them over competition if they don't have a frontier model on offer? If OpenAI stopped R&D a year ago, open weights models would leave them in the dust already.
If you don't design your compressor to output data that can be compressed further, it's going to trash compressibility.
And if you find a way to compress text that isn't insanely computationally expensive, and still makes the compressed text compressible by LLMs further - i.e. usable in training/inference? You, basically, would have invented a better tokenizer.
A lot of people in the industry are itching for a better tokenizer, so feel free to try.
Major AI companies are not doing nearly enough to address the sycophancy problem.
I get that it's not an easy problem to solve, but how is Anthropic supposed to solve the actual alignment problem if they can't even stop their production LLMs from glazing the user all the time? And OpenAI is somehow even worse.
Clearly you missed the part about name-calling. Anyways, that's a false dichotomy since you can be a third option: someone that's skeptical of messing with an important energy consumption pathway in the human body.
The baseline of "energy consumption pathways in the human body" now is to be severely messed up.
Humans did not evolve for an environment where food is overly abundant and physical activity is optional. For almost the entire evolutionary history of humans, this just wasn't the case. But it is what humans are having to deal with today.
Now, take a look at the "metabolic syndrome" and its prevalence. Clearly, there's a lot of room for improvement.
By all accounts, this generation of GLP-1 agonists has found a meaningful way to improve on that baseline. The benefits are broad and the side effects are manageable. This isn't "surprising" as much as it is "long overdue".
If you want an LLM to retain the same default personality, you pretty much have to use an open weights model. That's the only way to be sure it wouldn't be deprecated or updated without your knowledge.