Any superintelligence that achieves human values will be adjacent in design space to many superintelligences that cause massive suffering

Why so? Flipping the sign doesn't get you "adjacent", it gets you "diametrically opposed".

If you really want chocolate ice cream, "adjacent" would be getting strawberry ice cream, not having ghost pepper extract poured into your mouth.

They said "adjacent in design space". The Levenshtein distance between return val; and return -val; is 1.

0cousin_it3yThat's a twist on a standard LW argument, see e.g. here [https://wiki.lesswrong.com/wiki/Complexity_of_value]: It seems to me that fragility of value can lead to massive suffering in many ways.

S-risks: Why they are the worst existential risks, and how to prevent them

by Kaj_Sotala 1 min read20th Jun 2017107 comments

21