Firstly, love the article. The Cognition is a Journey through a Mental Landscape Figure is great, but for the cyborg illustration it would be better if at each node there were many AI branches and the human branch was shown to select one of these paths to follow.
Also, this might be outside the scope of a comment on this article, but this is my first comment so I'll take the risk. The language below reminds me of Fantasia - Sorcerer's Apprentice. If we outsource the actions that fulfil our wishes with a magic hat, then the wishes probably wouldn't be granted in the way we think we intend.
"An antithetical example to this is something like a genie, where the human outsources all of their agency to an external system that is then empowered to go off and optimize the world."
Thought this was a nice analogy and also is a simple way of explaining ai-risk to non-technical folk :)
Firstly, love the article. The Cognition is a Journey through a Mental Landscape Figure is great, but for the cyborg illustration it would be better if at each node there were many AI branches and the human branch was shown to select one of these paths to follow.
Also, this might be outside the scope of a comment on this article, but this is my first comment so I'll take the risk. The language below reminds me of Fantasia - Sorcerer's Apprentice. If we outsource the actions that fulfil our wishes with a magic hat, then the wishes probably wouldn't be granted in the way we think we intend.
"An antithetical example to this is something like a genie, where the human outsources all of their agency to an external system that is then empowered to go off and optimize the world."
Thought this was a nice analogy and also is a simple way of explaining ai-risk to non-technical folk :)