I don't think it will be very difficult to impart your intentions into a sufficiently advanced machine

Counterargument: it will be easy to impart an approximate version of your intentions, but hard to control the evolution of those values as you crank up the power. E.g. evolution, humans, make us want sex, we invent condoms.

No-one will really care about this until it's way too late and we're all locked up in nice padded cells and drugged up, or something equally bad but hard for me to imagine right now.

What Are The Chances of Actually Achieving FAI?

by Bound_up 1 min read25th Jul 201714 comments


I get that the shut up and multiply calculation will make it worth trying even if the odds are really low, but my emotions don't respond very well to low odds.

I can override that to some degree, but, at the end of the day, it'd be easier if the odds were actually pretty decent.

Comment or PM me, it's not something I've heard much about.