Music and chocolate are known to be mostly safe. I guess I'm more cautious about new self-modifications that can change my decisions massively, including decisions about more self-modifications. It seems like if I'm not careful, you can devise a sequence that will turn me into a paperclipper. That's why I discount such agents for now, until I understand better what CEV means.

S-risks: Why they are the worst existential risks, and how to prevent them

by Kaj_Sotala 1 min read20th Jun 2017107 comments