RSI systems may collapse into a small set of goals
We will soon have RSI capable systems, the intelligence explosion seems fairly close, and we haven't yet fully solved alignment. The orthogonality thesis says that there exist minds in the design space that could be very competent at almost all tasks yet have any kind of goals, like sorting pebbles...
Sep 102