I'm part of a team of 26 researchers that recently released a 212-page paper proposing a new plan for pausing AI, the "hardwired pause."[1] The paper is called "The Feasibility of a Hardwired Pause of Frontier AI Training." Our analysis suggests that, under this hardwired pause plan, we could probably pause for at least a decade if we start stopping within the next year, and this could likely be extended indefinitely. I expect this to be possible without trusting China or tanking the economy.
Note: I am speaking for myself and not the other 25 authors. The working group is led by Will Fithian and Wesley Holliday, and the work continues.
A hardwired pause is probably the best plan out there for how to do this (except maybe it doesn't go far enough).
This paper is likely the most in-depth analysis of pause feasibility in existence.
Discussion and disagreement on any of these is very welcome!
What's a hardwired pause?
A hardwired pause basically means:
Stop making chips that can be used to train new AI models.
Only make chips that can only be used to run inference on existing AI models (or that aren't AI accelerators at all). We call these "inference-only chips"; I'm also calling them "hardwired chips."
This is an instance of an "inference-only" pause, which others have previously proposed.
The main advantage hardwired chips provide is making the pause robust and durable. With other approaches to inference-only pauses, because training-capable chips are still being produced, it would be somewhat easier to build a secret fleet of them, and potentially much easier to rapidly resume a race to superintelligence (if you could seize a large fraction of the governed training-capable chips).
I think any credible pause plan must account for and govern existing chips. Our work also includes an in-depth analysis of how a "chip census" might proceed, and expect all but ~10,000 of the US's ~30,000,000 and China's ~3,000,000 "H100-equivalents" would likely be recoverable.[2]
In a hardwired pause, these chips could initially be transitioned to inference, verified with other techniques, before being decomissioned once hardwired chips are available. Of course, chips could also be governed in other ways, e.g. decomissioned, and the total number of hardwired chips produced could also be controlled... in general, there are many design-choices in the space of hardwired pauses available to policy-makers.
Why is this better than Plan A?
Epistemic Status: I have skimmed much of Plan A, discussed it with authors for >1 hour, and seen a presentation about it prior to its release.
In my mind, Plan A and a Hardwired Pause both start with a chip census and preventing further development until a more durable arrangement is reached. But instead of phasing out training chips for hardwired chips, Plan A relies on continued verification by other means to try to do the following for a duration of roughly 15 years:
The key disadvantage of Plan A I see is the "dry tinder problem": with all of this training-capable compute lying around, if the race resumes for some reason, it could take a very short period of time (e.g. days-to-months) to achieve superintelligence. Plan A tries to mitigate this by making the datacenters easy to destroy (e.g. Chinese datacenters are in Canada). But this still seems likely to be destabilizing, since the datacenters could be destroyed for reasons other than "you defected on the treaty." This makes it riskier to rely on them for things that are critical to national security, but it seems like AI will likely become critical to those things in Plan A.
The main advantage of Plan A I see is that it may in fact not be possible to have an extremely durable pause, especially with algorithmic progress. But I tend to think that for now, we should just be focused on continuously improving our ability to robustly maintain a pause, and that it is premature to assume a pause would need to end before long.
How is this different from MIRI's Plan S?
Epistemic Status: I have read AI Futures Project's short description of Plan S. I have skimmed but not read MIRI's "An International Agreement to Prevent the Premature Creation of Artificial Superintelligence," which I take to be the most detailed version of the plan, but I have discussed them with MIRI's technical governance team for >1 hour. This plan aims to stop AI progress indefinitely using a variety of conventional verification and enforcement methods:
A Hardwired Pause could be viewed as a particular instance of Plan S that differs from MIRI's in the details, specifically:
MIRI's paper doesn't propose a specific verification mechanism, whereas we propose hardwired chips. I currently think the use of hardwired chips is basically a strict upgrade over alternative approaches to verification that don't get rid of training-capable chips,[3] and therefore have a potential dry tinder problem.
MIRI's paper also includes research restrictions. These restrictions and other enforcement mechanisms involved in their plan might be more intrusive and carry greater concerns around civil liberties and concentration of power. Research restrictions are less important with hardwired chips, since they would not be able to run unapproved models. But research is still not without risks!
What are my biggest concerns about a hardwired pause (vs. getting rid of chips entirely)?
I've previously proposed systematically dismantling the compute supply chain, which is basically a hardwired pause without the hardwired inference chips. I think the hardwired pause proposal in our paper may not go far enough, especially if it is implemented too late. Some specific concerns are:
The infrastructure needed to produce hardwired chips is probably quite similar to that needed to produce training-capable chips, so the hardwired pause still provides a much better starting point for a resumed race.
There may be too much algorithmic progress.[4] Specifically, I think of three failure modes, all of which more advanced AI models could help contribute to via AI R&D:
Techniques that dramatically reduce the compute or communication bandwidth needed for training.
Advances in computing hardware.
New AI or computing paradigms that are dramatically more efficient.
A "scaffolding singularity" in which ever more powerful AI is built using existing models as merely one component in a larger system.
Setting aside the above concerns, I'm still very uncertain whether existing models are, or will be, net positive for society.
On the other hand, I see two main advantages of a hardwired pause:
Defection on the pause is less tantalizing. Instead of becoming the only one with powerful AI, you are still up against a bunch of hardwired AI models and so the relative gains you can expect from building a more powerful frontier in violation of the agreement are smaller.
Existing AI systems might help improve the stability of the pause, or otherwise help us make use of a pause (e.g. to do more safety research, or improve collective sense-making and decision-making).
Overall, I'm still inclined to favor the "get rid of the compute plan", while recognizing that the hardwired pause might be much, MUCH more palatable to many actors. Within AI Safety in particular, my sense is that people usually have lower p(doom) than me, usually have more specific and limited concerns about AI than I do, and often value the existence of current AI systems highly. Thus I think, if you are reading this, a hardwired pause should likely be your favorite (pause) plan, as per the title.
Why is the paper so long? What's in it?
The paper includes a reasonably fleshed out example of a hardwired pause, but most of the space is what I believe is the most in-depth analysis of feasibility that's been published. So arguably the main contribution is not the idea of a hardwired pause, but this analysis of feasibility.
In this work, feasibility encompasses durability, palatability, and reliability. And it touches on not just technical, but also political, geopolitical, and economic aspects of feasibility. The main questions regarding feasibility (and my brief answers) include:
Can you actually make chips that are only useful for running existing models?
Strong yes, we have a decent proof-of-concept from Taalas.
Can you find enough of the existing chip supply to prevent secret projects?
Quite likely yes, assuming pause-willingness and no algorithmic breakthroughs. AI chips are expensive capital goods that leave copious paper trails when they're built, purchased, operated, and (if they fail) returned under warranty. In particular, companies in both the U.S. and China get tax writeoffs for telling the government about them when they're purchased, and this applies even to chips smuggled into China.
Can you monitor fabs reliably enough to ensure that appreciable quantities of chips are not being produced in secret?
Quite likely yes. The supply chain is very complicated and would be difficult to replicate secretly; fabs need regular maintenance and produce signature chemical residues; and the location of relevant machines seems to be known.
Would this crash the economy?
There would probably be some economic impact, but much less than the 2008 financial crisis, e.g. it might decrease growth by 25% (vs. leading to negative growth for ~2 years in the case of the 2008 crisis).
Two big points I want to emphasize:
While some AI investments are heavily leveraged, the real economy is fairly insulated from these speculative investments.
A big question is how much diffusion of models-existing-at-the-time-of-a-pause could support these investments; this seems plausible to me.
How much would the U.S. and China need to trust each other for this to work?
Not that much -- this plan is based on verification, not trust, and was developed with the advice of former diplomats with experience in arms control. A central objective of this plan is to give the U.S. and China confidence that nobody is continuing frontier AI development in secret, and that this state of affairs is robust and durable. In implementing a hardwired pause, they would need to cooperate quite a bit, but if they genuinely want an international pause, they would have good reason to do so. The main places I see trust coming in are:
Trusting that purported pause-willingness is real and worth engaging with. Countries don't seriously risk being duped into unilaterally pausing, but policymakers might assume the other side is not engaging in good faith and not want to waste effort on an initiative they expect to fail.
Extending trust may be useful in various small ways during the chip census. For instance, let's say the U.S. and China both have 100k chips unaccounted for, but would be able to find (or "find") an additional 90k chips with more effort. Then they can both benefit from cooperating further on the margin (e.g. "find" another 2k chips) in order to elicit further cooperation from the other side (who might then "find" another 2k chips), etc. But I expect this would be much faster and smoother with more mutual trust. Importantly, if either side is convinced the other side is refusing to cooperate, that could threaten the agreement, so if both sides in fact want the agreement to succeed more than they want to have a bit more dark compute, it seems like they should be able to find a way to cooperate, although there is still a bargaining problem here.
Conclusion
I was inspired to write this after realizing that this work hadn't been properly socialized within the AI Safety community. While admitting a bias towards thinking the things I work on are significant, I think this is among the most important pieces of work for this community. My hope is that it provides a definitive refutation of the claim that pausing is impossible, making it impossible for learned people to credibly make such arguments publicly.
At the same time, this is part of an ongoing effort, and is in some sense a work in progress. There are many questions I am interested in that deserve further investigation, and I am keen to hear your thoughts and know how it is landing with other audiences you present it to.
And Will and Wes are enthusiastic for further discussion and engagement as well! :)
The exact number is hard to predict, of course, and would also depend on the level of international cooperation achieved (more on this at the end of the blog post).
The main advantage I had seen for other verification approaches is that hardwired models might have issues that need patching, and this wouldn't be supported by hardwired chips. But apparently this is not true for all chip designs; I highly recommend checking out https://github.com/Berkeley-AI-Risk/inference-only to learn more about hardwired chip designs.
Notably, the estimated rate of algorithmic progress in this work comes from Dwarkesh's blog and is lower than has been estimated by others, e.g. Aaron Scher from MIRI who pointed me to https://epochai.substack.com/p/the-least-understood-driver-of-ai as currently the best starting point reference on algorithmic progress. This is something I plan to look into more, but to summarize my current understanding: 1) Algorithmic changes may lead to greater returns with more compute, 2) Algorithmic progress behind the frontier can take advantage of models at the frontier (e.g. via distillation). Both of these factors suggest slower algorithmic progress during a hardwired pause: defectors would 1) be working with less compute and 2) be trying to advance the frontier, not playing catch up.
I'm part of a team of 26 researchers that recently released a 212-page paper proposing a new plan for pausing AI, the "hardwired pause."[1] The paper is called "The Feasibility of a Hardwired Pause of Frontier AI Training." Our analysis suggests that, under this hardwired pause plan, we could probably pause for at least a decade if we start stopping within the next year, and this could likely be extended indefinitely. I expect this to be possible without trusting China or tanking the economy.
Note: I am speaking for myself and not the other 25 authors. The working group is led by Will Fithian and Wesley Holliday, and the work continues.
I currently think:
Discussion and disagreement on any of these is very welcome!
What's a hardwired pause?
A hardwired pause basically means:
This is an instance of an "inference-only" pause, which others have previously proposed.
The main advantage hardwired chips provide is making the pause robust and durable. With other approaches to inference-only pauses, because training-capable chips are still being produced, it would be somewhat easier to build a secret fleet of them, and potentially much easier to rapidly resume a race to superintelligence (if you could seize a large fraction of the governed training-capable chips).
I think any credible pause plan must account for and govern existing chips. Our work also includes an in-depth analysis of how a "chip census" might proceed, and expect all but ~10,000 of the US's ~30,000,000 and China's ~3,000,000 "H100-equivalents" would likely be recoverable.[2]
In a hardwired pause, these chips could initially be transitioned to inference, verified with other techniques, before being decomissioned once hardwired chips are available. Of course, chips could also be governed in other ways, e.g. decomissioned, and the total number of hardwired chips produced could also be controlled... in general, there are many design-choices in the space of hardwired pauses available to policy-makers.
Why is this better than Plan A?
Epistemic Status: I have skimmed much of Plan A, discussed it with authors for >1 hour, and seen a presentation about it prior to its release.
In my mind, Plan A and a Hardwired Pause both start with a chip census and preventing further development until a more durable arrangement is reached. But instead of phasing out training chips for hardwired chips, Plan A relies on continued verification by other means to try to do the following for a duration of roughly 15 years:
The key disadvantage of Plan A I see is the "dry tinder problem": with all of this training-capable compute lying around, if the race resumes for some reason, it could take a very short period of time (e.g. days-to-months) to achieve superintelligence. Plan A tries to mitigate this by making the datacenters easy to destroy (e.g. Chinese datacenters are in Canada). But this still seems likely to be destabilizing, since the datacenters could be destroyed for reasons other than "you defected on the treaty." This makes it riskier to rely on them for things that are critical to national security, but it seems like AI will likely become critical to those things in Plan A.
The main advantage of Plan A I see is that it may in fact not be possible to have an extremely durable pause, especially with algorithmic progress. But I tend to think that for now, we should just be focused on continuously improving our ability to robustly maintain a pause, and that it is premature to assume a pause would need to end before long.
How is this different from MIRI's Plan S?
Epistemic Status: I have read AI Futures Project's short description of Plan S. I have skimmed but not read MIRI's "An International Agreement to Prevent the Premature Creation of Artificial Superintelligence," which I take to be the most detailed version of the plan, but I have discussed them with MIRI's technical governance team for >1 hour. This plan aims to stop AI progress indefinitely using a variety of conventional verification and enforcement methods:
Figure 1 from "An International Agreement to Prevent the Premature Creation of Artificial Superintelligence."
A Hardwired Pause could be viewed as a particular instance of Plan S that differs from MIRI's in the details, specifically:
What are my biggest concerns about a hardwired pause (vs. getting rid of chips entirely)?
I've previously proposed systematically dismantling the compute supply chain, which is basically a hardwired pause without the hardwired inference chips. I think the hardwired pause proposal in our paper may not go far enough, especially if it is implemented too late. Some specific concerns are:
On the other hand, I see two main advantages of a hardwired pause:
Overall, I'm still inclined to favor the "get rid of the compute plan", while recognizing that the hardwired pause might be much, MUCH more palatable to many actors. Within AI Safety in particular, my sense is that people usually have lower p(doom) than me, usually have more specific and limited concerns about AI than I do, and often value the existence of current AI systems highly. Thus I think, if you are reading this, a hardwired pause should likely be your favorite (pause) plan, as per the title.
Why is the paper so long? What's in it?
The paper includes a reasonably fleshed out example of a hardwired pause, but most of the space is what I believe is the most in-depth analysis of feasibility that's been published. So arguably the main contribution is not the idea of a hardwired pause, but this analysis of feasibility.
In this work, feasibility encompasses durability, palatability, and reliability. And it touches on not just technical, but also political, geopolitical, and economic aspects of feasibility. The main questions regarding feasibility (and my brief answers) include:
Conclusion
I was inspired to write this after realizing that this work hadn't been properly socialized within the AI Safety community. While admitting a bias towards thinking the things I work on are significant, I think this is among the most important pieces of work for this community. My hope is that it provides a definitive refutation of the claim that pausing is impossible, making it impossible for learned people to credibly make such arguments publicly.
At the same time, this is part of an ongoing effort, and is in some sense a work in progress. There are many questions I am interested in that deserve further investigation, and I am keen to hear your thoughts and know how it is landing with other audiences you present it to.
And Will and Wes are enthusiastic for further discussion and engagement as well! :)
I understand there's a blog post proposing something similar from a few weeks ago: https://www.lesswrong.com/posts/n8u3BfqFoGh4jnzpo/plan-r-ai-safety-by-asics#TEedRFoyYvxJt2KiM
The exact number is hard to predict, of course, and would also depend on the level of international cooperation achieved (more on this at the end of the blog post).
The main advantage I had seen for other verification approaches is that hardwired models might have issues that need patching, and this wouldn't be supported by hardwired chips. But apparently this is not true for all chip designs; I highly recommend checking out https://github.com/Berkeley-AI-Risk/inference-only to learn more about hardwired chip designs.
Notably, the estimated rate of algorithmic progress in this work comes from Dwarkesh's blog and is lower than has been estimated by others, e.g. Aaron Scher from MIRI who pointed me to https://epochai.substack.com/p/the-least-understood-driver-of-ai as currently the best starting point reference on algorithmic progress. This is something I plan to look into more, but to summarize my current understanding: 1) Algorithmic changes may lead to greater returns with more compute, 2) Algorithmic progress behind the frontier can take advantage of models at the frontier (e.g. via distillation). Both of these factors suggest slower algorithmic progress during a hardwired pause: defectors would 1) be working with less compute and 2) be trying to advance the frontier, not playing catch up.