So his timing was slightly off. I don’t know why people expected LLMs to improve exponentially. Your iPhone now doesn’t look much different than the one 10 years ago. GPT-3 or arguably GPT-4 was the first iPhone moment, everything else will be gradual improvements unless fundamental discoveries are found, but those happen seemingly randomly.
> More over, intelligence has a superexponential return, 90IQ->100IQ < 100IQ->110IQ in terms of returns
That's the second time I've seen the claim that linear increases in intelligence have "superexponential" results, first time was Altman's blog.
But I've not seen any justification for this.
(As you specifically say IQ, note that an IQ is defined as a mapping of standard deviations rather than a mapping of absolute skill, the normal mapping is 15 points being 1σ).
AI is spreading across disciplines like science, math, software development, language, music, and health. You’re looking at it too narrowly. Human-computer symbiosis is accelerating at an unprecedented rate, far beyond the pace of something like the iPhone.
More like computer-human parasitism; it weakens the host.
It also only affects those with a "weak immune system" i.e. those whose bullshit filter doesn't function.
AI is here to stay for some tasks (segment anything, diffusion image generation for accelerating certain kinds of Photoshop), but LLMs are a dead end and AI Winter 2 is coming. They don't work for programming or law or medicine or mechanical engineering or even writing most emails because it's faster to just write the email yourself than to ask the AI to do it.
In what sense are the bleeding edge models incremental improvements over GPT-3 (read his examples of GPT-3 output and imagine any of the top models today producing them!), GPT-3.5, or GPT-4? Look at any benchmark or use it yourself. It's night and day.
Gary Marcus didn't make a lot of specific criticisms or concrete predictions in his essay [0], but some of his criticisms of GPT-3 were:
- "For all its fluency, GPT-3 can neither integrate information from basic web searches nor reason about the most basic everyday phenomena."
- "Researchers at DeepMind and elsewhere have been trying desperately to patch the toxic language and misinformation problems, but have thus far come up dry."
- "Deep learning on its own continues to struggle even in domains as orderly as arithmetic."
Are these not all dramatically improved, no matter how you measure them, in the past three years?
To me, the current LLMs aren't qualitatively different from the char RNNs that Karpathy showcased all the way back in 2015. They've gotten a lot more useful, but that is about it. Current LLMs will have as much to do with GAI as computer games have to do with NNs. Which is to say, games were necessary to develop GPUs which were then used to train NNs, and current LLMs are necessary to incentivize even more powerful hardware to come into existence, but there isn't much gratitude involved in that process.
The strengths and weaknesses of the algorithmic niche that artificial NNs are in hasn't changed a bit since a decade ago. They are still bad at anything I'd want to actually use them for that you'd imagine actual AI would be good at. The only thing that has changed is people's perception. LLMs found a market fit, but if you notice, compared to last decade where we had Deepmind and OpenAI competing at actual AI in games like Go and Starcraft, they've pretty much given up on that in favor on hyping text predictors. For anybody in the field, it should be an obvious bubble.
Underneath it all, there is some hope that an innovation might come about to keep the wave going, and indeed, a new branch of ML being discovered could revolutionize AI and actually be worthy of the hype that LLMs have now, but that has nothing to do with the LLM craze.
It's cool that we have them, and I also appreciate what Stable Diffusion has brought to the world, but in terms of how much LLMs influenced me, they only shorted the time it takes for me to read the documentation.
I don't think that machines cannot be more intelligent than humans. I don't think that the fact that they use linear algebra and mathematical functions makes the computers inferior to humans. I just think that the current algorithms suck. I want better algos so we can have actual AI instead of this trash.
