But the orthogonality thesis isn't just some idea that may or not be true. It follows directly from Hume's Guillotine, discovered in 1739: no amount of superintelligent reasoning about facts will allow you to derive goals.
No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine.
The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...
No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine.
The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...