Epistemic Status: I think this argument is robust as long as “the conditions” are true. However, I may have miscalculated and left out a condition. I may also be missing something else entirely. Feedback on the post would be helpful in either case.

Introduction

If you want to survive, pause from behind. This can also be described as “Losing Heroically”.

Imagine a simplified model of a race to ASI between two actors. The two actors can each choose to continue racing ahead, or pause and obtain some X percentage-point reduction of takeover risk. Assume that once the actor in front crosses some finish line, they either achieve complete control over the world, or their ASI goes rogue and kills everyone, with the probability of the second outcome decreasing with pause length.

Given certain (possibly attainable) conditions which I list below, the game-theoretic optimal move is for the laggard in the race to “pause from behind” (Lose Heroically) or to stop racing to ASI and credibly signal the pause to the leader. Under another set of (possibly attainable) conditions, the optimal response of the leader is then to pause as well, until either the laggard resumes racing or takeover risk is reduced as much as is perceived to be possible.

The intuition for why Losing Heroically is the optimal move for the laggard is this: if the laggard cannot win the race to ASI no matter what, they might as well lose with a lower chance of takeover risk. And then the optimal response from the leader is symmetric: if they are able to win no matter what, they might as well win with a lower chance of takeover risk.

The Conditions

Two notes before I talk about the conditions.

I am listing the conditions in their absolute form, as if they are either 100% met or 100% unmet. In reality, some (or all) of these conditions may be only probabilistically met, as in, “there is a W% chance that condition Z is true”. It may still be optimal to Lose Heroically, or to respond with a pause, if the probabilities are high enough.
I am also assuming that both actors have either perfect information or an accurately calibrated estimate of their conditions (the other actor’s conditions do not need to be known). In reality, two other things may happen. The first is a false positive: an actor believes their conditions are met, when in reality their conditions are not

...

Posts

Wikitag Contributions

Comments