This post will only make sense if you already know about AI 2040. If you don't, consider reading the authors' announcement post (Substack, Less Wrong), or reading the full scenario.
I did not like AI 2027 and expected to feel similarly about AI 2040: Plan A. Having now read Plan A, I can safely say I was wrong: I vastly prefer it to AI 2027 and sincerely hope more people read it. However, Plan A is long and inaccessible to all except the most committed of readers, in large part because the authors decided not to include an executive summary.
As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.
I disagree. I think many people will not read AI 2040, including some who probably should, precisely because it is so long and inaccessible. I also think those people would benefit from reading an executive summary-style post describing it. This post is my attempt at writing that summary.
The Argument
The AI Futures Project authors believe that the world should agree to an international AI-race slowdown treaty—basically just an arms control deal—that still advances research aimed at aligning and controlling AI. They call this Plan A.
Their argument is as follows:
If we progress AI capabilities too fast, we vastly increase the risk of harms stemming from uncontrolled AI.
However, a complete and permanent pause—which they call Plan S—has two key limitations:
The pause will almost certainly fall apart within a few years (or decades) for all the usual reasons.[1] Once it does, the AI research arms race resumes, potentially under even worse geopolitical circumstances.
The pause can’t stop all covert projects. The longer the pause, the greater the risk of some defector secretly developing ASI.[2]
This very helpful image comes from Astral Codex Ten, not AI 2040.
The Plan
Executed perfectly, Plan A looks something like this:
The U.S. and China—and by extension the rest of the world[3]—agree to temporarily pause AI capabilities research. To this end,
Each country publicly declares its compute infrastructure.
The whole world temporarily pauses all new model training runs, probably via something like inference-only verification.
After a long period of negotiations, the world resumes training under the linchpin of Plan A: a joint regulatory regime known as "Total Research Transparency."
Under it, all frontier companies must open-source their algorithms and provide access to their models.[4]
The authors explain this diagram in more detail here. Author (and diagram creator) Daniel Kokotajlo further explains the diagram here.
The world also ensures that AI progress is reversible through methods such as "Mutually Assured Compute Destruction."[5]
America deals with rising unemployment by establishing a "Citizen's Dividend."[6]
As time goes on, AI—which becomes increasingly more aligned as alignment research progresses—helps us improve our alignment research, general governance, and plans for the future.
from AI 2040
Eventually, once we're certain the AIs are fully aligned, we hand off control to them.[7]
Roadblocks and Failure Modes
The authors are acutely aware of Plan A's many downsides. Below are seven potential roadblocks and failure modes.
The initial treaty falls through.
The authors believe that Plan A would be politically feasible for "almost everyone but aren't fully confident in their projection.[8]
The authors acknowledge that this would make Plan A riskier, slower, more frantic, and more expensive. They still think that Plan A is the best response.
Assuming that Plan A's verification steps are carried out competently and work as planned, the authors overwhelmingly expect this project to fail. And, given that "a bunch of covert AGI projects in frontier companies" basically describes our current status quo, this worry is not sufficient to oppose Plan A.
Interestingly, the authors actually cite this as a point in their favor: they believe that Total Research Transparency would make it harder for companies/governments to train in biases and easier for the world to see what their AI models really believe. Either way, the status quo is still worse, so this worry is not sufficient to oppose Plan A.[9]
I plan to write another post focusing on the potential cruxes, points of disagreement, and strongest criticisms of Plan A. For example, I wish that they’d explained their robotics automation forecasts in more detail. I also wish they’d talked a bit more about what happens if AI capabilities progress is slower than they anticipate, which would significantly affect the dynamics at play.
But overall, I think Plan A is excellent. No other plan comes close to being as comprehensive, and I’m confident that the authors’ work will be put to good use even if the plan falls through.
The most pressing [danger] is that treaties can’t last forever - for example, the START treaties fell apart after thirty years. If Plan A met a similar fate, the race to superintelligence would begin again. Better for it to happen sooner under good conditions, than later during a chaotic race.
That being said, the authors are generally sympathetic to Plan S. Excerpted from them:
whether you prefer Plan A or Plan S might come down to how optimistic you are about the future of a US-China deal, and your assessment of technical alignment and control difficulty:
You should prefer Plan S insofar as you think the deal is stable and insofar as you think that scaling to more capable AIs early in the deal doesn’t help much with safety or poses risks outweighing its helpfulness...You should prefer Plan A if you think the deal is unstable and that scaling to more capable AIs early in the deal is helpful and safe.
We are uncertain, but overall think that a well-implemented version of Plan A is probably going to work to avoid loss of control and concentration of power risks, whereas we think that even a well-implemented version of Plan S would plausibly collapse within a decade or so into another dangerous race to ASI.
Do note that their preferred versions of Plan S strongly resembles Plan A:
The best versions of Plan S are those that acknowledge that the deal will eventually end, and simply say “First (step 1) we should stop making frontier AIs more capable, because the AIs and AI companies are getting more powerful every day and there’s so much uncertainty about where it’s headed and how fast. Then (step 2) once the world has had several years to think about things and plan a safe and broadly beneficial path forward, we can resume.”
While Plan A strictly applies to the whole world, the specific scenario primarily revolves around the U.S. and China, the two leading players in AI. Specifically, Plan A assumes that the U.S. is acting optimally in the best interests of itself and the world, while China is acting as it actually would in real life. This is not a safe assumption for a purely probabilistic forecast, but Plan A is more of a positive vision/wish list anyway.
We’ll agree to let each other see all the AI research. Then, if we don’t like something someone is doing, we’ll talk about it and perhaps agree to ban it.
That being said, Total Research Transparency is more restrictive than full open sourcing, as it does not require model weights to be open (and actively advises against it):
The main way that Plan A could be even more open is by allowing or requiring open model weights for frontier models. However, we recommend against publicly releasing frontier model weights. The main reasons for this are:
Open weight models can be used by covert projects to attempt an intelligence explosion; once covert projects have sufficiently capable models, the intelligence explosion probably goes very quickly (for example, under the AI 2040 default capability trajectory, there are only 2 months between TED AI and ASI).
Closed weight models can have refusal safeguards trained into them, while it is trivial to remove safeguards from open weight models. This is important both for limiting countries' AI capabilities and for limiting misuse. For example, terrorists and other actors might be able to use highly capable open weight models to design world-ending bioweapons (e.g., develop mirror life). Bioweapons are very offense dominant. In this scenario, we recommend substantial defense against bioweapons, but prevention is a strongly preferable line of defense to mitigation.
Mutually Assured Compute Destruction is necessary to punish defection and/or roll back capabilities if something goes horribly wrong. From AI 2040:
The US and China agree on the importance of ensuring that the compute is destroyed in the case of deal dissolution. To accomplish this, they agree to build the datacenters in the third-party countries least secure against their rival’s military intervention: China’s new datacenters will be in Canada, and America’s in Mongolia, with both hosts compensated via monetary payments, jobs, and a share in the new AI economy. If the deal dissolves, they reason, America and Canada will immediately move to take control of China’s datacenters in Canada, and China will self-destruct their compute rather than let it fall into American hands (and vice versa). Thus the idea of “Mutually Assured Compute Destruction” is born.
This is by far the most speculative and exotic part of the scenario: the authors confine it to an epilogue section. I'm not terribly interested in it, largely because I think it’s far too early to predict this kind of thing, so I won't get into it here.
The authors expect America to maintain its current AI capabilities lead, leaving the rest of the world scrambling to accept a mutually beneficial deal. Quoting them:
In short, the answer is that anyone concerned about loss of control should think Plan A is an improvement, along with anyone concerned about concentration of power—except for the people in whom the power would concentrate by default.
China...is an example of an actor in whom power would not concentrate by default. In 2029 in this scenario, the US has a significant lead in AI capabilities over China and a significant advantage in compute which will compound the lead. The more powerful AI gets, the scarier it will be to fall behind, as AI 2027 and the later years in this scenario illustrate.
It would thus be harder to establish a treaty if America lost its capabilities lead.
Another concern they mention is the risk of America not agreeing to Plan A:
For these reasons, we expect strong opposition to Plan A from the leading AI companies. We expect them to rationalize arguments for why the deal is bad and why instead what’s best for America and humanity is a different strategy that just so happens to allow them to continue accumulating massive amounts of power.
However, because Plan A implicitly assumes that the U.S. is acting optimally in the best interests of itself and the world, the authors do not discuss this concern further.
I think this is a fair choice to make—it’s hard to write a positive vision for the future if you can’t assume that someone is going to make the right decisions—but for those of us in the real world, it does add an extra barrier to implementation (”even if I think Plan A sounds fine, how would we get and sustain the political will required to make all these drastic changes?”)
I keep repeating this phrase for a reason; if we had viable plans that did better than the status quo on any of these metrics, I'd link to them, and presumably the AIFP authors would too. But I'm unaware of any such plans, and so I have nothing to compare Plan A to.
This is a crazy thing to try releasing. Daniel gave me several justifications for doing it anyway, but the one I remember most is that it’s supposed to be a floor. When some politician proposes a data center ban, or says that we have to gut safety regulation to compete with China, or promises a job retraining program, think to yourself: does this person have a vision for where all of this ends up? If so, is it as good as Plan A? If not, consider demanding that they do better.
In the Plan A Assumptions supplement, author Thomas Larsen argues that the main downsides of building out data centers are:
Increasing the importance of verification and regulation—and thus increasing the magnitude of harm caused by their failure.
Increasing the speed of AI research if the deal breaks down AND the data centers are not destroyed.
He concludes that building out data centers is a good idea if and only if:
Scaling quickly comes with big upsides (i.e. covert projects are a serious concern).
Data verification is reliable.
Mutually Assured Compute Destruction is reliable.
If any of those three premises are false, the argument becomes invalid.
For more on these concerns, particularly regarding the feasibility of Mutually Assured Compute Destruction, consider reading Plan A's problem with dry tinder by Tom Davidson.
This post will only make sense if you already know about AI 2040. If you don't, consider reading the authors' announcement post (Substack, Less Wrong), or reading the full scenario.
Alternatively, if you want to read a longer unofficial summary of AI 2040: Plan A, consider reading Scott Alexander’s introduction and reaction post or Zvi Mowshowitz’s introduction and reaction(s) post.
Intro
I did not like AI 2027 and expected to feel similarly about AI 2040: Plan A. Having now read Plan A, I can safely say I was wrong: I vastly prefer it to AI 2027 and sincerely hope more people read it. However, Plan A is long and inaccessible to all except the most committed of readers, in large part because the authors decided not to include an executive summary.
The authors of AI 2040 explain:
I disagree. I think many people will not read AI 2040, including some who probably should, precisely because it is so long and inaccessible. I also think those people would benefit from reading an executive summary-style post describing it. This post is my attempt at writing that summary.
The Argument
The AI Futures Project authors believe that the world should agree to an international AI-race slowdown treaty—basically just an arms control deal—that still advances research aimed at aligning and controlling AI. They call this Plan A.
Their argument is as follows:
This very helpful image comes from Astral Codex Ten, not AI 2040.
The Plan
Executed perfectly, Plan A looks something like this:
The authors explain this diagram in more detail here. Author (and diagram creator) Daniel Kokotajlo further explains the diagram here.
from AI 2040
Roadblocks and Failure Modes
The authors are acutely aware of Plan A's many downsides. Below are seven potential roadblocks and failure modes.
Coda
I plan to write another post focusing on the potential cruxes, points of disagreement, and strongest criticisms of Plan A. For example, I wish that they’d explained their robotics automation forecasts in more detail. I also wish they’d talked a bit more about what happens if AI capabilities progress is slower than they anticipate, which would significantly affect the dynamics at play.
But overall, I think Plan A is excellent. No other plan comes close to being as comprehensive, and I’m confident that the authors’ work will be put to good use even if the plan falls through.
The START treaties were a series of nuclear weapons disarmament treaties between the U.S. and Russia.
That being said, the authors are generally sympathetic to Plan S. Excerpted from them:
Do note that their preferred versions of Plan S strongly resembles Plan A:
While Plan A strictly applies to the whole world, the specific scenario primarily revolves around the U.S. and China, the two leading players in AI. Specifically, Plan A assumes that the U.S. is acting optimally in the best interests of itself and the world, while China is acting as it actually would in real life. This is not a safe assumption for a purely probabilistic forecast, but Plan A is more of a positive vision/wish list anyway.
In their own words:
That being said, Total Research Transparency is more restrictive than full open sourcing, as it does not require model weights to be open (and actively advises against it):
Mutually Assured Compute Destruction is necessary to punish defection and/or roll back capabilities if something goes horribly wrong. From AI 2040:
For more on how a Citizen's Dividend might work, consider The Windfall Clause (short talk, long paper).
This is by far the most speculative and exotic part of the scenario: the authors confine it to an epilogue section. I'm not terribly interested in it, largely because I think it’s far too early to predict this kind of thing, so I won't get into it here.
The authors expect America to maintain its current AI capabilities lead, leaving the rest of the world scrambling to accept a mutually beneficial deal. Quoting them:
It would thus be harder to establish a treaty if America lost its capabilities lead.
Another concern they mention is the risk of America not agreeing to Plan A:
However, because Plan A implicitly assumes that the U.S. is acting optimally in the best interests of itself and the world, the authors do not discuss this concern further.
I think this is a fair choice to make—it’s hard to write a positive vision for the future if you can’t assume that someone is going to make the right decisions—but for those of us in the real world, it does add an extra barrier to implementation (”even if I think Plan A sounds fine, how would we get and sustain the political will required to make all these drastic changes?”)
I keep repeating this phrase for a reason; if we had viable plans that did better than the status quo on any of these metrics, I'd link to them, and presumably the AIFP authors would too. But I'm unaware of any such plans, and so I have nothing to compare Plan A to.
From Scott Alexander's response post:
In the Plan A Assumptions supplement, author Thomas Larsen argues that the main downsides of building out data centers are:
He concludes that building out data centers is a good idea if and only if:
If any of those three premises are false, the argument becomes invalid.
For more on these concerns, particularly regarding the feasibility of Mutually Assured Compute Destruction, consider reading Plan A's problem with dry tinder by Tom Davidson.