I don’t disagree that it’ll change a lot of things in society.
But that isn’t the claim being made, which is that some sort of AI god is being constructed which will develop entirely without the influence of how real human beings actually act. This to me is basically just sci-fi, and it’s frankly kind of embarrassing that it’s taken so seriously.
It is enormously easier to code for an AI agent which pursues a task doggedly by iterating down its list of available tools, without any consistent human moral values, than to code in some kind of human morality manually. For that, you need to have solved philosophy. Mathematically, as a proof, and with a high degree of certainty that you are correct.
But that isn’t the claim being made, which is that some sort of AI god is being constructed which will develop entirely without the influence of how real human beings actually act. This to me is basically just sci-fi, and it’s frankly kind of embarrassing that it’s taken so seriously.