Words are cheap, and Altman is a salesman. He’s just staying whatever he needs to say to keep people off his back. I don’t put any stock into this statement whatsoever. He may slow development, or he may not, but whether he does will depend almost entirely on what’s best for him and his company, not humanity.
It can’t do what he’s trying to do. It can’t one shot a giant project. Not reliably. It still can’t. Not even Astra. Not even close.
You need to use LLMs to build the individual components and then put it together yourself. The human architect is still needed.
Just saying: “build this complete project” is not architecting. It’s more like wishing. You will find rare examples where someone’s LLM wish came true (more or less), but I think most of these people are just burning tokens.
It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.
The most charitable explanation I can think of for this is something like regression to the mean. When a model is first released, there'll be a subset of users who, just by chance, sample the highest quality band of the distribution that answers their query. Some of them will rush over to social media and post about how amazing a model is. Over time, those users' mental model of responses will converge but they'll perceive the model's return to typical performance as a downgrade.
This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included).
I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.
Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.
> I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.
IMO this is 90% of it (as someone who has a bit of a different interaction style and runs these things less autonomously, and hasn't generally seen the claimed regressions). Day 1: throw new stuff at it that failed badly, exciting to see something make more progress! Day n: reality sets in that it still wasn't perfect the first time.
To add onto this, if you use a shiny new model and it gives you a turd, you're not going to tweet about it ("hey guys, look what I made with Astra! Nothing!"), and even if you do nobody is going to interact with it so it does poorly in the algorithm, because it has to compete with all the people using the new model to make something that looks impressive. Then people get tired of the magic trick and the logic flips.
We've seen that some---gpt5 was considered pretty lackluster intially, in particular. Opus 4.7 and 5 vs 4.6 were also greeted with a lot more "meh" than 4.6 or Fable.
Yeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.
Can't really measure without being sure you aren't being messed around with, when it's a remote platform. Stupidly easy to detect when people run such benchmarks/tests against you as well.
Some people here have remarked previously that while reduced precision doesn't show up in quick prompts, it does severely impact these models' ability to perform long running tasks - to the point that running these big models with severe quantization might be counterproductive as smaller but less quantized ones perform better.
It’s all speculation (you too), but I think the effect you’re describing is instead getting calibrated to the model’s limits. Next time a new model comes out, wait a month before trying and see if you have the same feeling of rapid quality decline after a few days. I did after I jumped back into it mid 5.x or whatever ChatGPT after paternity leave. Blown away for a few days, worried about my job for a few days, then increasingly aware of its limits.
You would need a significant sample size to make any sort of conclusion from such a probabilistic process. Then there's the issue of how you would actually grade/compare.
You only need to come up with a catchy "SomethingBench" name, post it on reddit/x and now you're an ai sage. Not to disparage the launch/after comparison though, I'd genuinely enjoy a data point like that
Run a benchmark with a large number of samples, rerun a few days later. Compare results, use statistics to see if there's a statistically significant difference.
You would want to use your own benchmark, certainly, not something publicly known. But outside of that I don't think "this is a benchmark" is such an easy category to determine -- a benchmark should be similar to a typical scenario anyway.
> "this is a benchmark" is such an easy category to determine
I mean, it's not _that_ hard to determine most likely, and/or it's hard to be sure you didn't get found out by llm-assisted analysis on your traffic. It's not going to be a one-shot request and response it's going to be a whole bunch of them in an artificial way, by nature.
And then anything you found is single-use only if you're paranoid because even if they find out later, they have your benchmark now (because you sent it to them to use it even if you don't publish it).
There is a toot from an Open AI person a couple days ago saying they are "pulling all the levers" because of capacity issues. I have no idea what the heck the person is talking about, but I'm guessing there are consequence for those levers.
> "Demand for Astra is really unprecedented. We're pulling all the levers possible to sustain the demand, but I've not seen anything like it until now and we went through very steep growth before. Priority will always be to keep excellent service for existing users, but we might have to pause new Pro subscriptions for a bit if this continues."
I’m not an expert at body language, but I think anyone can see she is trying really, really hard to appear friendly and trustworthy. But when someone is trying that hard to convince you they are benign, it makes you suspicious.
I find the whole Skills.md idea a little silly. We have a technology that struggles to stay within rails due to its inherent makeup. And you think you can fix that by just telling it to?
It makes as much sense as the so-called “humanizer” tools that purport to make LLMs stop using their well-known verbal tells.
All you are doing is saying: “Hey you know that thing you can’t stop doing? Can you stop doing that?” The machine will say “Absolutely!” but eventually start doing it again.
You’re not wrong, but I don’t see a lot of alternatives. So we just don’t try to fix it? At least skills give us a landing pad for “here’s how to attempt to do things consistently in a way I generally approve of”
My suggestion is to get a plain old bicycle (not an electric or stationary bike). One hour outdoors on a bike clears away at least a day’s worth of mental detritus for me.
Take a random essay and add in a bunch of the phrases that LLMs love like “load-bearing,” “crucial,” structural,” and “woven,” and then submit the original and the edited version to an LLM and ask which is better. It will choose the second one virtually every time. They have ingrained biases that associate those words with good writing and arguments.
Sometimes I wonder if there’s just one guy somewhere who loved using the word load-bearing, all his papers got trained on, and now he can’t write anything without being assumed to be Claude.
This is why using other LLMs as scorers for benchmarks and evaluations is such a bad idea, they'll have preferences you can't anticipate and won't understand immediately.
Let me know when AI patches this bug in the wheels on all the vehicles on the highway, AND my box fan when it's next to the light with the bad PWM dimming. Then we can resume our existential doom.
There is no realistic path to human oversight for the vast quantities of data these models are trained on. Imagine the cost of having every Reddit comment ingested human reviewed. Insane.
reply