One common argument against pacing the frontier is the need to win the Race Against China. I believe that, if you take the possibility of ASI seriously, then the victory conditions in this race are extremely low probability. The proper course of action is to stop racing, signal to your competitors that you’ve stopped, and never start again.
Proponents of the race can be divided roughly into two camps. The first camp views AI as normal technology, akin to industrialization or the internet. To them, the AI race is like any other in human history, where the player who harnesses the technology first gains economic, geopolitical, and military advantages. AI does not pose any existential risk, and the benefits from developing it outweigh any harms. The incentive to race is sound if you accept their premises, and I will not attempt to address this argument here (others havedone so extensively).
The second camp views AI as significantly more consequential than previous technologies. The first player to harness AGI/ASI will gain a Decisive Strategic Advantage (DSA) - achieving complete dominance over all competitors and securing unassailable control of the world. Therefore, they argue, a player with values aligned with liberal democracy must be the winner of the race, lest an authoritarian regime get there first and permanently enslave the world under its yoke. This is the argument I will examine.
Victory Conditions There are various scenarios where DSA can be achieved. The winner of the race:
Builds perfectly aligned Guardian Angel ASI
Gains various capabilities that enable them to permanently disable all competitors
Gains monitoring and gatekeeping capabilities that prevent all competitors from advancing past defined thresholds
Guardian Angel ASI
If the winner is able to fully solve alignment prior to the instantiation of catastrophic risks, this ideal outcome would be achieved. At the present moment, I don’t know of any serious technical people who believe that we are on track to do so. Under race conditions, AI capabilities will continue to grow much faster than safety mechanisms. I assign the probability of this outcome as <1%.
Disable All Competitors
The capability set that enables this outcome is some combination of:
Offensive cyber hacking of critical infrastructure: NatSec agencies, AI labs, power grid, communications, financial systems, transportation systems, etc
Advanced physical weapons systems design and manufacture, disabling off-network threats including nuclear weapons and conventional force
Threat of new weapons of mass destruction: bio, chemical, nano
Control of powerful individuals through coercion or manipulation
Mass social and political manipulation using superhuman persuasion algorithms
Mass surveillance and global monitoring to identify and disable new threats
I have three objections to this outcome.
Similar to the aligned ASI scenario, racing toward these capabilities increases catastrophic risk, and out-paces safety mechanisms. Best guess at successful alignment of AI with these capabilities while avoiding risk instantiation: <10%.
If these capabilities are achieved and deployed without incident, the result is massive concentration of power. The winner of the race, proven to be a power seeking actor, is unlikely to peacefully and voluntarily democratize their AI capabilities, even to those they view as value aligned. All the leading US labs, after all, state that they have to win because everyone else would get it wrong. My best guess at the winner altruistically deciding that they don’t know best and therefore should hand over their power: <10%. Stacking these two probabilities, a good outcome is, again, <1%.
Basic philosophical objection to the morality of this course of action. Even if we achieve the unlikely outcome of disabling competitors followed by controlled democratization, the path we took is drenched in blood. We will have violated the sovereignty of nations and the freedom/life of individuals. One can argue that this is a price worth paying, but that has to be stacked against alternative options, while weighing probabilities of success.
Monitoring and Gatekeeping
This outcome assumes the winner develops the same disabling capability set, and utilizes only monitoring tools until a competitor passes some threatening capability threshold. The same objections from above apply:
At 90% strength - the larger risk space is during development rather than deployment.
At 80% strength - in this case the winner is taking a less power seeking stance, but effectively holds the same amount of power.
Is mitigated as long as the threshold is not passed by competitors. This seems unlikely given the current competitive environment. Once a competitor approaches the threshold, the full suite of disabling tools are deployed and all the moral objections return.
This is a slightly more optimistic outcome than above. If players decide to pursue DSA despite the risks, I hope this is the mindset and operational path they choose.
In summary, the race to DSA is fraught with catastrophic misalignment and power concentration risks. Even victory is bought with blood, the victor ceding any moral high ground. This is not the path we should pursue. Critically, a racing posture signals to your competitors that you intend to harm them if you win. This is very bad for any hopes of cooperation.
What Should We Do Instead?
International cooperation, shared objectives, coordinated technological governance - these are not idealistic pipe dreams. Rather, there is a strong precedent of the world coming together to solve tremendouslychallenging problems. Making AI go well will be harder than anything that came before, but certainly conceivable. Compared to the near certain catastrophe that awaits at the end of the race, cooperation is the obvious choice.
I am not naive to the current political realities, the adversarial stance of the great powers, and the numerous incentive headwinds working against international AI Safety cooperation. I know it’s the hardest problem humanity has ever faced. I know there are significant and powerful failure modes to cooperation. But when the alternatives are far less hopeful, we must choose the only reasonable option remaining.
Have transparent conversations. Start from our shared concerns and common ground. Build trust one step at a time. Create robust verification regimes. Negotiate careful deterrence measures. And stop this crazed race toward the bloody finish line.
A Race With No Winners
One common argument against pacing the frontier is the need to win the Race Against China. I believe that, if you take the possibility of ASI seriously, then the victory conditions in this race are extremely low probability. The proper course of action is to stop racing, signal to your competitors that you’ve stopped, and never start again.
Proponents of the race can be divided roughly into two camps. The first camp views AI as normal technology, akin to industrialization or the internet. To them, the AI race is like any other in human history, where the player who harnesses the technology first gains economic, geopolitical, and military advantages. AI does not pose any existential risk, and the benefits from developing it outweigh any harms. The incentive to race is sound if you accept their premises, and I will not attempt to address this argument here (others have done so extensively).
The second camp views AI as significantly more consequential than previous technologies. The first player to harness AGI/ASI will gain a Decisive Strategic Advantage (DSA) - achieving complete dominance over all competitors and securing unassailable control of the world. Therefore, they argue, a player with values aligned with liberal democracy must be the winner of the race, lest an authoritarian regime get there first and permanently enslave the world under its yoke. This is the argument I will examine.
Victory Conditions
There are various scenarios where DSA can be achieved. The winner of the race:
Guardian Angel ASI
If the winner is able to fully solve alignment prior to the instantiation of catastrophic risks, this ideal outcome would be achieved. At the present moment, I don’t know of any serious technical people who believe that we are on track to do so. Under race conditions, AI capabilities will continue to grow much faster than safety mechanisms. I assign the probability of this outcome as <1%.
Disable All Competitors
The capability set that enables this outcome is some combination of:
I have three objections to this outcome.
Monitoring and Gatekeeping
This outcome assumes the winner develops the same disabling capability set, and utilizes only monitoring tools until a competitor passes some threatening capability threshold. The same objections from above apply:
This is a slightly more optimistic outcome than above. If players decide to pursue DSA despite the risks, I hope this is the mindset and operational path they choose.
In summary, the race to DSA is fraught with catastrophic misalignment and power concentration risks. Even victory is bought with blood, the victor ceding any moral high ground. This is not the path we should pursue. Critically, a racing posture signals to your competitors that you intend to harm them if you win. This is very bad for any hopes of cooperation.
What Should We Do Instead?
International cooperation, shared objectives, coordinated technological governance - these are not idealistic pipe dreams. Rather, there is a strong precedent of the world coming together to solve tremendously challenging problems. Making AI go well will be harder than anything that came before, but certainly conceivable. Compared to the near certain catastrophe that awaits at the end of the race, cooperation is the obvious choice.
I am not naive to the current political realities, the adversarial stance of the great powers, and the numerous incentive headwinds working against international AI Safety cooperation. I know it’s the hardest problem humanity has ever faced. I know there are significant and powerful failure modes to cooperation. But when the alternatives are far less hopeful, we must choose the only reasonable option remaining.
Have transparent conversations. Start from our shared concerns and common ground. Build trust one step at a time. Create robust verification regimes. Negotiate careful deterrence measures. And stop this crazed race toward the bloody finish line.