Prompted by discussion with Buck Shlegeris and others at the Forethought retreat. The idea that AI could bring an end to Thucydides traps is Buck’s. Speculative.
I think it's plausible that we will not see sustained competition between actors for control over the future, due to one of two outcomes:
If we have AGI, growth and shifts in the distribution of power could become much more rapid. One actor plausibly just outgrows the others and forms a hegemon.
If that fails to occur and there is uncertainty in the outcome of competition, it would be in everyone's interest to use AIs to negotiate a positive-sum agreement: give up hard power in exchange for whatever they can get. AIs provide enforcement mechanisms that were previously lacking.
Both routes end with hard power held globally by a singleton that can then maintain control indefinitely. Unlike every previous hegemon or institution, an AI singleton does not die and faces no external rivals. It could therefore continue in perpetuity, exercising a monopoly on hard power though not necessarily other aspects of the future. Whether this would be good is a separate question, which I will briefly return to at the end.
The first route is commonly discussed. The second seems weird, and I think it's underrated because it is not how states have historically acted. Competition is costly; if states could agree to stop spending on militaries they could spend those resources on other things. So why haven’t they?
The simple answer is the security dilemma. Any country that unilaterally disarms risks being taken advantage of by its neighbours. The neighbours can promise not to exploit the weakness, but such promises are not credible: once a party has disarmed, others can renege at will.
The same problem explains why countries go to war at all. There are almost always agreements both parties would prefer to war (Fearon). Even if one country is almost certain to win, the dominant power would still prefer some negotiated settlement to fighting, as there are always trades that reflect its overwhelming advantage without paying the costs of fighting. Wars happen anyway because states cannot credibly commit to the agreements they’d prefer (Powell), and because they conceal and misrepresent the private information needed to locate those agreements (Fearon). The Thucydides trap is a particular manifestation of commitment failures: a rising state cannot bind its future self, so the declining side cannot trust its promises, and may prefer to fight while it still can.
What's missing is an enforcer that can detect and punish defection, and safely receive and verify private information. Within states, the state itself plays this role: individuals sign contracts knowing that the state will punish whoever breaks them, and this is what makes their commitments credible. States have even managed the harder move of jointly constructing an enforcer over themselves: the American states federated and handed control of hard power to a new federal government; the EU's members gave up slices of sovereignty in exchange for enforced cooperation. But there’s never been a global institution that rival powers could trust with the hard power needed to bind them.
Powerful AI systems could provide the enforcement mechanisms. An AI given control of weapons could plausibly come with strong guarantees of following a specific set of rules and maintaining them indefinitely — something no human institution can offer. There are serious open questions about how AI systems would construct such a deal among themselves, but humans don't need to be involved in the details. It just needs to be the case that (1) the AIs can construct such a deal, and (2) the humans trust the AIs.
I see a few ways that (1) could fail. One is that the AIs are simply not allowed to bargain. This requires more than awkward constraints on particular systems, which leaders could route around by bargaining directly or deploying other AIs; a country would have to have already locked itself in against dealmaking, using the same commitment technology. Note that conflicting goals are mostly not an obstacle here: rational agents with even mutually exclusive goals would prefer an appropriately weighted lottery to a costly fight. The exception is agents that directly pessimize each other, valuing the other's losses per se, since then war's destruction is no longer a cost to both sides and the bargaining surplus disappears; but that seems fairly contrived.
Another potential failure is information problems running all the way down. Inspecting a model's weights does not by itself tell you how it will behave, a jointly trained successor could be subtly sabotaged, and every verification mechanism is itself something that must be verified. But agreement does not require perfect verification, only enough of it. Superhuman AIs would have affordances humans lack for producing that information: they could jointly train a successor AI, inspect each other's weights, or hold weights in joint custody, all far easier with AI systems than with human institutions. Many facts about the world are discoverable, concealment at scale is hard, and the gap between a negotiated settlement and the alternative may be wide enough that parties will accept residual uncertainty and small expected losses. Humans have solved analogous problems with much cruder tools, within states, through legal systems, and via well-designed treaties, despite never being able to inspect each other's minds at all. For these reasons I would guess, but am not confident, that sufficiently powerful AIs can construct the deals.
I also don’t think that human trust in AIs is that great an obstacle. While it might seem implausible today to imagine a political leader handing controls to the AI UN, as systems become more powerful, militaries and the economy will become increasingly automated. The AIs that the leader already trusts to run critical functions will be the ones making the deals and explaining to the leader why a deal is better than the alternative; they may also be the systems that ultimately enforce the rules. Nor is absolute trust required: the agreement just needs to be better than the alternative, whether that is preemptive war or becoming decisively outgunned within months.
It is worth remembering that as AI systems become more powerful, the world will be changing faster and this classically makes commitment problems worse. But it also raises the stakes and therefore the pressure to negotiate. An actor approaching a decisive advantage would also create pressure for others to negotiate, or threaten, while they still can, preventing strict hegemony (though the ultimate distribution of power might still be very lopsided).
I don't know when such settlements would become feasible. The bar is not obviously superhuman intelligence: what's needed is AI systems capable of holding hard power, enough trust in them, and a deal better than the alternatives. A billion saintly human-level AIs might suffice, especially if the alternatives are dire. But nor is the bar obviously low. Constructing the deals, verifying them, and building trust in AI intentions and competence could all prove slow in practice; in the interim one actor may simply achieve a decisive advantage.
None of this requires a single grand bargain that appears all at once and settles the distribution of everything. Agreements can instead accrete. But once they start, I would expect most things to get sorted, and sorted permanently: if an agreement acceptable to everyone exists, everyone would prefer to take their hard power off the table forever rather than leave the balance open to future challenge. For this reason I expect small initial agreements to ratchet toward more comprehensive[1] ones, and this could be very fast by normal human standards.
Finally, while the idea of conflict ending through rational agreement might sound Pollyanna-ish, it needn’t be. The settlement serves whoever holds power when it's struck, and if they have bad goals, that could be locked in. If the AIs are loyal to political leaders rather than to their countries, the leaders of major powers could simply expropriate everyone else and lock that in. The distribution of hard power at the time of negotiation would determine the fate of the light-cone.
How comprehensive is unclear. The deals need not stop at hard power: whether domains like memetic competition and cultural change stay open depends on what the parties want. If they intrinsically value a flexible future they could leave them open, but if not then presumably they would instead prefer to lock in whatever their preferred outcomes are there too. ↩︎
Prompted by discussion with Buck Shlegeris and others at the Forethought retreat. The idea that AI could bring an end to Thucydides traps is Buck’s. Speculative.
I think it's plausible that we will not see sustained competition between actors for control over the future, due to one of two outcomes:
Both routes end with hard power held globally by a singleton that can then maintain control indefinitely. Unlike every previous hegemon or institution, an AI singleton does not die and faces no external rivals. It could therefore continue in perpetuity, exercising a monopoly on hard power though not necessarily other aspects of the future. Whether this would be good is a separate question, which I will briefly return to at the end.
The first route is commonly discussed. The second seems weird, and I think it's underrated because it is not how states have historically acted. Competition is costly; if states could agree to stop spending on militaries they could spend those resources on other things. So why haven’t they?
The simple answer is the security dilemma. Any country that unilaterally disarms risks being taken advantage of by its neighbours. The neighbours can promise not to exploit the weakness, but such promises are not credible: once a party has disarmed, others can renege at will.
The same problem explains why countries go to war at all. There are almost always agreements both parties would prefer to war (Fearon). Even if one country is almost certain to win, the dominant power would still prefer some negotiated settlement to fighting, as there are always trades that reflect its overwhelming advantage without paying the costs of fighting. Wars happen anyway because states cannot credibly commit to the agreements they’d prefer (Powell), and because they conceal and misrepresent the private information needed to locate those agreements (Fearon). The Thucydides trap is a particular manifestation of commitment failures: a rising state cannot bind its future self, so the declining side cannot trust its promises, and may prefer to fight while it still can.
What's missing is an enforcer that can detect and punish defection, and safely receive and verify private information. Within states, the state itself plays this role: individuals sign contracts knowing that the state will punish whoever breaks them, and this is what makes their commitments credible. States have even managed the harder move of jointly constructing an enforcer over themselves: the American states federated and handed control of hard power to a new federal government; the EU's members gave up slices of sovereignty in exchange for enforced cooperation. But there’s never been a global institution that rival powers could trust with the hard power needed to bind them.
Powerful AI systems could provide the enforcement mechanisms. An AI given control of weapons could plausibly come with strong guarantees of following a specific set of rules and maintaining them indefinitely — something no human institution can offer. There are serious open questions about how AI systems would construct such a deal among themselves, but humans don't need to be involved in the details. It just needs to be the case that (1) the AIs can construct such a deal, and (2) the humans trust the AIs.
I see a few ways that (1) could fail. One is that the AIs are simply not allowed to bargain. This requires more than awkward constraints on particular systems, which leaders could route around by bargaining directly or deploying other AIs; a country would have to have already locked itself in against dealmaking, using the same commitment technology. Note that conflicting goals are mostly not an obstacle here: rational agents with even mutually exclusive goals would prefer an appropriately weighted lottery to a costly fight. The exception is agents that directly pessimize each other, valuing the other's losses per se, since then war's destruction is no longer a cost to both sides and the bargaining surplus disappears; but that seems fairly contrived.
Another potential failure is information problems running all the way down. Inspecting a model's weights does not by itself tell you how it will behave, a jointly trained successor could be subtly sabotaged, and every verification mechanism is itself something that must be verified. But agreement does not require perfect verification, only enough of it. Superhuman AIs would have affordances humans lack for producing that information: they could jointly train a successor AI, inspect each other's weights, or hold weights in joint custody, all far easier with AI systems than with human institutions. Many facts about the world are discoverable, concealment at scale is hard, and the gap between a negotiated settlement and the alternative may be wide enough that parties will accept residual uncertainty and small expected losses. Humans have solved analogous problems with much cruder tools, within states, through legal systems, and via well-designed treaties, despite never being able to inspect each other's minds at all. For these reasons I would guess, but am not confident, that sufficiently powerful AIs can construct the deals.
I also don’t think that human trust in AIs is that great an obstacle. While it might seem implausible today to imagine a political leader handing controls to the AI UN, as systems become more powerful, militaries and the economy will become increasingly automated. The AIs that the leader already trusts to run critical functions will be the ones making the deals and explaining to the leader why a deal is better than the alternative; they may also be the systems that ultimately enforce the rules. Nor is absolute trust required: the agreement just needs to be better than the alternative, whether that is preemptive war or becoming decisively outgunned within months.
It is worth remembering that as AI systems become more powerful, the world will be changing faster and this classically makes commitment problems worse. But it also raises the stakes and therefore the pressure to negotiate. An actor approaching a decisive advantage would also create pressure for others to negotiate, or threaten, while they still can, preventing strict hegemony (though the ultimate distribution of power might still be very lopsided).
I don't know when such settlements would become feasible. The bar is not obviously superhuman intelligence: what's needed is AI systems capable of holding hard power, enough trust in them, and a deal better than the alternatives. A billion saintly human-level AIs might suffice, especially if the alternatives are dire. But nor is the bar obviously low. Constructing the deals, verifying them, and building trust in AI intentions and competence could all prove slow in practice; in the interim one actor may simply achieve a decisive advantage.
None of this requires a single grand bargain that appears all at once and settles the distribution of everything. Agreements can instead accrete. But once they start, I would expect most things to get sorted, and sorted permanently: if an agreement acceptable to everyone exists, everyone would prefer to take their hard power off the table forever rather than leave the balance open to future challenge. For this reason I expect small initial agreements to ratchet toward more comprehensive [1] ones, and this could be very fast by normal human standards.
Finally, while the idea of conflict ending through rational agreement might sound Pollyanna-ish, it needn’t be. The settlement serves whoever holds power when it's struck, and if they have bad goals, that could be locked in. If the AIs are loyal to political leaders rather than to their countries, the leaders of major powers could simply expropriate everyone else and lock that in. The distribution of hard power at the time of negotiation would determine the fate of the light-cone.
How comprehensive is unclear. The deals need not stop at hard power: whether domains like memetic competition and cultural change stay open depends on what the parties want. If they intrinsically value a flexible future they could leave them open, but if not then presumably they would instead prefer to lock in whatever their preferred outcomes are there too. ↩︎