positiveFuture

Good post, but I am curious about your thinking for :

> when you expect that you’re going to run into the same people over and over again for a long time to come, it’s wise to be consistently trustworthy and helpful even if you could get an advantage in the short run by betraying someone; the long-term costs to your reputation, and the costs of making a habit of doing something that’s usually destructive, vastly outweigh any short-term advantage you could gain this one time.

Isn't this the small world case? Following the pattern, my big world example woul... (read more)

I think the point is that there aren't just two black and white possibilities:

• get extinct by developing superintelligence,
• stop AI R&D.

I would argue that it is much more difficult to understand at which point the AI is too dangerous to be improved than to understand at which point YOUR AI is "better enough" than YOUR ENEMY'S AI (China or the US) to provide you some strategic advantage. Under this premise, if your adversary is rational, once you stop your AI R&D, he may realize that his optimal choice is to do just a bit more R&D and only then st... (read more)