Was "scale is all you need" actually a real thing said by a real person? Even the most pro scale people like Altman seem to be saying research and algorithms are a thing too. I guess as you say a more important thing is where the money goes. I think Altman's been overdoing it a bit on scaling spend.
> Even the most pro scale people like Altman seem to be saying research and algorithms are a thing too.
I think you missed the nuance in my explanation of both sides. Yes, they believed algorithmic development mattered but small. Tuning, not even considering exporting different architectures than the transformer.
Which Altman said that AGI is a scaling problem, which is why he was asking for $7T. But he was clearly a lier given this from last year. There's no way he really believed this in late 2024.
> Altman claimed that AGI could be achieved in 2025 during an interview for Y Combinator, declaring that it is now simply an engineering problem. He said things were moving faster than expected and that the path to AGI was "basically clear."[0]
I'm with Chollet on this one, our obsession with LLMs have held us back. Not that we didn't learn a lot from them but that our hyper fixation closed our minds to other possibilities. The ML field (and CS in general) gets hyper fixated on certain things and I just don't get that. Look at diffusion models. There was basically a 5 year gap between the first unet based model and DDPM. All because we were obsessed with GANs at the time. We jump on a hypetrain and shun anyone who doesn't want to get on. This is not a healthy ecosystem and it hinders growth.
Just because we end up with success doesn't mean the path to get there was reasonable nor does it mean it was efficient.
Fair enough although that Altman quote doesn't match what he actually said in the interview. He said:
>...first time ever where I felt like we actually know what to do like I think
from here to building an AGI will still take a huge amount of work there are some known unknowns but I think we basically know what to go what to go do and it'll take a while it'll be hard but that's tremendously exciting... https://youtu.be/xXCBz_8hM9w?t=2330
and at the end there was "what are you excited for in 2025?" and Altman says "AGI" but that doesn't specify if that's it arriving or just working on it.
I don't think huge amount of work and known unknowns is the same as we just need to scale.
