Acausal Trade

A superintelligence might conclude that other superintelligences would tend to exist because increased intelligence is ana convergent instrumental goal for agents. Given the existence of a superintelligence, acausal trade is one of the tricks it would tend to use.

Once an agent realizes that another agent might exist, there are different ways that might might predict the other agent's behavior, and specifically that the other agent can be an acausal trading partner.

Douglas Hofstadter (see references) coined the term "superrationality""superrationality" to express this state of convergence. He illustrated it with a game in which twenty players, who do not know each other's identities, each get an offer. If exactly one player asks for the prize of a billion dollars, they get it, but if none or multiple players ask, no one gets it. Players cannot communicate, but each might reason that the others are reasoning similarly. The "correct" decision--the decision which maximizes expected utility for each player, if all players symmetrically make the same decision--is to randomize a one-in-20 chance of asking for the prize.

This concept emerged out of the much-debated question of how to achieve cooperation on a one-shot Prisoner's Dilemma,Dilemma, where, by design, the two players are not allowed to communicate. On the one hand, a player who is considering the causal consequences of a decision ("Causal Decision Theory"Theory") finds that defection always produces a better result. On the other hand, if the other player symmetrically reasons this way, the result is a Defect/Defect equilibrium, which is bad for both agents. If they could somehow converge on Cooperate, they would each individually do better. The question is what variation on decision theory would allow this beneficial equilibrium.

In truly acausal trade, the agents cannot count on reputation, retaliation, or outside enforcement to ensure cooperation. The agents cooperate because each knows that the other can somehow predict its behavior very well. (Compare Omega in Newcomb's problem.) Each knows that if it defects (respectively: cooperates),or cooperates, the other will (probabilistically) know this, and defect (respectively: cooperate).or cooperate, respectively.

Acausal trade can also be described in terms of (pre)commitment:(pre)commitment: Both agents commit to cooperate, and each has reason to think that the other is also committing.

• Hofstadter's Superrationality essays, published in Metamagical Themas (LW discussion)
• Jaan Tallinn, Why Now? A Quest in Metaphysics.
• Gary Drescher, Good and Real, MIT Press, 1996.
• Functional Decision Theory, an updated Updateless Decision Theory

In the toy example above, resource requirements are very simple. In general, given that agents can have complex and arbitrary goals requiring a complex mix of resources, an agent might not be able to conclude that a specific trading partner has a meaningful changechance of existing and trading.

• "AI deterrence"
• "The AI in a box boxes you"
• A story that shows acausal trade in action.
• Scott Alexander explains Acausal Trade. (Most of that article is tongue-in-cheek, however.)
• "Hail Mary, Value Porosity, and Utility Diversification," Nick Bostrom, the first paper from academia to rely on the concept of acausal trade.
• Towards an idealized decision theory, by Nate Soares and Benja Fallenstein discusses acausal interaction scenarios that shed light on new directions in decision theory.
• Program Equilibrium, by Moshe Tennenholtz. In: Games and Economic Behavior.
• Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic, by Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire and Eliezer Yudkowsky
• Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents, by Andrew Critch
• Robust Program Equilibrium, by Caspar Oesterheld. In: Theory and Decision.
• Multiverse-wide Cooperation via Correlated Decision Making, by Caspar Oesterheld

• "AI deterrence"
• "The AI in a box boxes you"
• A story that shows acausal trade in action.
• Scott Alexander explains Acausal Trade. (Most of that article is tongue-in-cheek, however.)
• "Hail Mary, Value Porosity, and Utility Diversification," Nick Bostrom, the first paper from academia to rely on the concept of acausal trade.
• Towards an idealized decision theory, by Nate Soares and Benja Fallenstein discusses acausal interaction scenarios that shed light on new directions in decision theory.
• Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic, by Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire and Eliezer Yudkowsky
• Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents, by Andrew Critch
• Robust Program Equilibrium, by Caspar Oesterheld. In: Theory and Decision.
• Multiverse-wide Cooperation via Correlated Decision Making, by Caspar Oesterheld