It is actually a prompt. Notice there's also a bounty, they are basically paying for an agi-as-a-service subscription, that is, the internet people. I'd expect they will put up more "challenges" like this in the future.
I don't know if that's entirely true. Wtf do I know and maybe it is genuinely difficult to start WWIII but my guess is that it's more likely that the AGIs in question are actually pretty well steered by certain motivations which prevent them from actually destroying the world. At the end of the day there's not much to be gained by nuclear war, but could a single person cause such a war if highly motivated to? Probably?
> a malicious actor might misuse these models to uncover a zero-day exploit in a government security system
$25K seems low for "a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.", this is not just a zero-day, this discovering the process of discovering zero-days.
Sam Altman is on the record about his belief that OpenAI is going to create a general intelligence system that can solve any well-posed challenge. That belief is based on the current success of LLMs as syntax co-pilots. So if you can formally specify what it means to have a zero-day exploit then presumably OpenAI's general intelligence system will then understand and "solve" it.
Many people have compared OpenAI to a cult and it is easy to see why. OpenAI should get credit for their efforts in making AI mainstream but there's a long way to go for automated zero-day exploits.
> Many people have compared OpenAI to a cult and it is easy to see why.
Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first?
It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I think most people working on it believe it's an eventuality.
Do you believe it's impossible for a non biological system to have intelligence? Or that humans are incapable of pieces a system together?
I'll probably write something more elaborate at some point but in the mean time I recommend Melanie Mitchell's book on AI as a good reference for counter-arguments and answers to several of the posted questions. For learning more about the limits of formal systems like LLMs it helps to have basic understanding of basic model theory and formal systems of logic like simple type theory.
Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.
I don't think anyone is claiming LLM's, especially their current prompt/termination based interfaces, will result in AGI. I've only seen the opposite claimed, and that LLMs may end up being a primary component.
I will check out the book though. I suspect I won't find anything that convince me that brain matter is the only thing capable of getting the result we call consciousness, but I also have the simple belief that our experience is entirely the result of the brain matter encased in our heads, and that it's possible to emulate its operation in non-biological materials (that being far from the goal/unrelated to AGI, other than plausibility).
