https://ai-2040.com/ is a compelling look at one path forward in a world where the US decides to cooperate with China to slow down transformative AI, to buy us time to do it safely. (Ok, it has a bunch of potential paths forward, but one key path involving cooperation.)
To me, the key insight in Plan A is that you need a set of layered defenses, a full stack that covers what you govern (e.g. chips, weights, capabilities, outcomes), how you verify (e.g. hardware, facilities, inspectors), and incentives.
The exact proposal for centralizing compute in specific non-party countries with the military staring at them is useful (is is plausible?), but I thought it would be an interesting exercise to think of other places you might govern or verify, and other incentive structures.
So here's a list:
Compute is controlled at the chip level, not at the datacenter level. Chips are manufactured with integrated monitoring that can detect training. Chips themselves are carefully tracked and have remote killswitches, enforced at the fabs (of which there are a very small number, so verification is relatively easy).
This is not a new idea, and in fact I think something like this was described in the Vinge novel Rainbows End. There's already serious policy work looking into this.
There's a lot to like about governance being at the chip level. You don't need massive government run installations, and instead you can rely on the private sector to fund and build out all that capacity (at least in the west). Monitoring is vastly less intrusive than
Where it's more challenging is that it might be hard to prevent someone with physical access to the chip from disabling the control mechanisms. It's a lot easier to blow up a datacenter than it is to detect someone on the other side of the planet fiddling with hardware they own.
Also these chips don't exist today, and would need to be created and rolled out at scale. There isn't a great story about what to do about existing chip stocks -- they may still need to be gathered up somehow a la Plan A.
Joint international project a la CERN. Instead of two competing datacenter complexes each owned by one player, you could have a joint project that both run together. You could even invite other major countries to participate, so it's not just a duopoly. You still need to centralize all the training chips in one location. Monitoring is a lot easier (there's just one thing to monitor and all sides have access), so you don't need the radical transparency publication of all the algorithmic advances, which is potentially infohazardous. There's a lot of legitimacy because everyone has a way to participate.
In return for monitoring being easy, now governance is hard. How to handle disputes seems sticky, and the incentive to try to smuggle algo advances out might be quite high.
Also, physical control of the one location seems very valuable, so siting is probably going to be a difficult issue. It would also be a single point of failure for misalignment, though I'm not sure if it's much worse than Plan A.
Full deterrence. Instead of trying to upend how modern civilization works, you just agree that each side is allowed to take strong action when the other seems to be taking steps towards superintelligence. This is the Hendrycks/Schmidt/Wang Mutually Assured Intelligence Malfunction (MAIM) approach.
We know how MAD works and it held together for many decades. We don't need that many decades, probably, and even a couple would potentially help quite a lot in terms of finding safe outcomes. You dodge the infohazard problem, and don't have really touch governance issues to work through. You don't even really need an agreement -- nobody signed a treaty that allows for nuclear MAD, it just is the equilibrium outcome.
The downside is that it seems potentially unstable, and there would be a lot of ways for things to go wrong. Misunderstandings, temptations to cheat, potential attacks that were falsely justified by fears of ASI all seem possible. Nuclear deterrence works because a nuclear weapon going off is a discrete event, easily detectable and attributable. AI R&D progress is not really like that.
Responsible Scaling Policies enforced by treaty. We have a set of capability red lines that no model is allowed to cross, whether deployed or not. Third party evaluators are embedded within every lab capable of building AGI. If the capability line is hit, training is paused until it can be fixed.
The red lines are defined by safety cases similar to Plan A's, and have a large enough safety buffer that hitting them isn't an instant emergency. The red lines creep up over time as our safety tech gets better. This is nice because you don't actually care about compute, you care about capabilities, so this is more directly measuring the important thing.
Challenges are obvious: you have to be able to identify the labs ahead of time, sandbagging could be a problem (either intentionally from the lab or schemingly from the model). Also, Anthropic found that RSP-style predefined lines were too inflexible given how fast the situation changes.
This could be good in combination with (1).
Slow down with liability, insurance, and punitive damages a la Gabriel Weil. A strict liability regime (which might be good for other reasons) and mandatory insurance makes it very expensive to even consider building advanced AI. Forbid labs from contracting away liability (i.e. downstream customers can't indemnify them). No subsidies plus expensive liability puts a lot of downward pressure on advanced AI development. Punitive damages makes up for the fact that liability cannot cover catastrophic risk (see the link above for more). A treaty with China imposes similar standards in both countries (precedents for this sort of thing show up in trade negotiations, e.g. around environmental standards.)
This is much more incremental and therefore achievable than a grand bargain involving massive government intervention. Also it's nice that better safety tech reduces liability, so there's a structural incentive to work hard on safety. Decentralized incentives are the pattern that has worked best historically and driven huge progress, including with safety progress like fire codes and safety testing (which came from the fire insurance industry).
Liability works less well for labs that don't intend to have a product and just want to build ASI. Also if the governments want ASI for strategic purposes, there will be temptation to cheat in various ways.
This could be useful in combination with others on the list.
You can pick various combinations of these to capture the full stack that Plan A runs down. Chip-based detection, RSP-style redlines as a trigger, and MAIM as enforcement in combination arguably cover the problem, for instance. (5) plausibly has the best incentives overall, and can easily be combined with others on the list.
All would be subject to the same off-ramp as in Plan A. Once we have verifiably safe AI, we should shift to taking advantage of all it offers.
I think it's unlikely that Plan A has exactly the best combination of things-to-govern, inspection regimes, and incentives (though I do think it picked a reasonable set to start with). What elements in combination could end up better than Plan A?
Disclosure: I work at Anthropic. I'm speaking only for myself here and not for the company.
Re 3: it's not obvious that nuclear MAD is stable throughout the AI transition/intelligence explosion, and imo more likely than not that it won't be stable due to technical reasons.
https://ai-2040.com/ is a compelling look at one path forward in a world where the US decides to cooperate with China to slow down transformative AI, to buy us time to do it safely. (Ok, it has a bunch of potential paths forward, but one key path involving cooperation.)
To me, the key insight in Plan A is that you need a set of layered defenses, a full stack that covers what you govern (e.g. chips, weights, capabilities, outcomes), how you verify (e.g. hardware, facilities, inspectors), and incentives.
The exact proposal for centralizing compute in specific non-party countries with the military staring at them is useful (is is plausible?), but I thought it would be an interesting exercise to think of other places you might govern or verify, and other incentive structures.
So here's a list:
This is not a new idea, and in fact I think something like this was described in the Vinge novel Rainbows End. There's already serious policy work looking into this.
There's a lot to like about governance being at the chip level. You don't need massive government run installations, and instead you can rely on the private sector to fund and build out all that capacity (at least in the west). Monitoring is vastly less intrusive than
Where it's more challenging is that it might be hard to prevent someone with physical access to the chip from disabling the control mechanisms. It's a lot easier to blow up a datacenter than it is to detect someone on the other side of the planet fiddling with hardware they own.
Also these chips don't exist today, and would need to be created and rolled out at scale. There isn't a great story about what to do about existing chip stocks -- they may still need to be gathered up somehow a la Plan A.
In return for monitoring being easy, now governance is hard. How to handle disputes seems sticky, and the incentive to try to smuggle algo advances out might be quite high.
Also, physical control of the one location seems very valuable, so siting is probably going to be a difficult issue. It would also be a single point of failure for misalignment, though I'm not sure if it's much worse than Plan A.
We know how MAD works and it held together for many decades. We don't need that many decades, probably, and even a couple would potentially help quite a lot in terms of finding safe outcomes. You dodge the infohazard problem, and don't have really touch governance issues to work through. You don't even really need an agreement -- nobody signed a treaty that allows for nuclear MAD, it just is the equilibrium outcome.
The downside is that it seems potentially unstable, and there would be a lot of ways for things to go wrong. Misunderstandings, temptations to cheat, potential attacks that were falsely justified by fears of ASI all seem possible. Nuclear deterrence works because a nuclear weapon going off is a discrete event, easily detectable and attributable. AI R&D progress is not really like that.
The red lines are defined by safety cases similar to Plan A's, and have a large enough safety buffer that hitting them isn't an instant emergency. The red lines creep up over time as our safety tech gets better. This is nice because you don't actually care about compute, you care about capabilities, so this is more directly measuring the important thing.
Challenges are obvious: you have to be able to identify the labs ahead of time, sandbagging could be a problem (either intentionally from the lab or schemingly from the model). Also, Anthropic found that RSP-style predefined lines were too inflexible given how fast the situation changes.
This could be good in combination with (1).
This is much more incremental and therefore achievable than a grand bargain involving massive government intervention. Also it's nice that better safety tech reduces liability, so there's a structural incentive to work hard on safety. Decentralized incentives are the pattern that has worked best historically and driven huge progress, including with safety progress like fire codes and safety testing (which came from the fire insurance industry).
Liability works less well for labs that don't intend to have a product and just want to build ASI. Also if the governments want ASI for strategic purposes, there will be temptation to cheat in various ways.
This could be useful in combination with others on the list.
You can pick various combinations of these to capture the full stack that Plan A runs down. Chip-based detection, RSP-style redlines as a trigger, and MAIM as enforcement in combination arguably cover the problem, for instance. (5) plausibly has the best incentives overall, and can easily be combined with others on the list.
All would be subject to the same off-ramp as in Plan A. Once we have verifiably safe AI, we should shift to taking advantage of all it offers.
I think it's unlikely that Plan A has exactly the best combination of things-to-govern, inspection regimes, and incentives (though I do think it picked a reasonable set to start with). What elements in combination could end up better than Plan A?
Disclosure: I work at Anthropic. I'm speaking only for myself here and not for the company.