I have lots more to read in AI 2040 and the commentaries that Zvi lists. But I'll state my own reaction, developed in an exchange with @Daniel Kokotajlo: You can't have 14 years (2026-2040) of "rapid progress", in which today's reasoning models have risen to the level of "top human experts", and there are millions or hundreds of millions of them doing stuff, without producing superintelligence long before. If something like Plan A began to unfold in the real world, I would expect this to happen within just a few years.
In my own "final research" sequence, I had intended the next instalment to say something about AI architecture, but at the moment I'm thinking to focus just on correct ontology and values, since that seems to be where we are lagging the most (compared to progress in the ability to make AIs with raw intelligence).
Introducing Plan A
The folks who brought you AI 2027, a so far remarkably accurate set of predictions despite those predictions having seemed freaky to many at the time, now bring you their positive vision that involves more freaky predictions: Plan A.
These guys have rather strong prediction track records. In addition to AI 2027, among other things, Daniel Kokotajlo has What 2026 Looks Like (which is remarkably similar to what 2026 looks like) and Ryan Greenblatt, who is also the chief scientist at Redwood Research, was the #2 most accurate AI forecaster in 2025 out of 413 entries. Past performance is as always no guarantee of future success.
If you’re the type to read at least some of my posts, or if you thought AI 2027 was worth reading, I recommend reading Plan A.
There is also an unofficial visual novel version, for minds very different from my own who would want that.
To be clear up front: I am not endorsing Plan A. I am not suggesting we should go off and try to enact Plan A as written. There is a lot more work to do and a lot of potential problems and downsides to grapple with.
I do think we should do that work, and give it, and its details, serious consideration.
The bulk of this post is engaging with various objections, in progressively more detail. If you only need the highlights, you can safely stop after Thus Selective Optimism.
Table of Contents
You Only Get Five Words
As always, most people who interact with Plan A will not read it.
They will condense it into a few key sentences.
There is clear agreement on which sentences survive and which do not.
The top 5 things people will discuss will largely be, in descending order of focus:
Axios’s Ashley Gold compacts the plan thus:
That’s not technically wrong, but there is an obvious misinterpretation if you allow that much compression.
And includes this very good quote:
The strongest and loudest objection, and in some ways the best one, is some form of:
At the extreme you get people like Joshua Saxe wondering why people didn’t learn their lesson from what didn’t happen to radiologists, and so on. Le sigh, but I appreciate saying it straight, and I appreciated Timothy Lee’s reaction even more:
I would hope that with enough time I could get Timothy Lee to come around, since he has established he takes arguments seriously, but so far I’ve been unable to find a compelling argument to convince such folks that for practical purposes yes sufficiently advanced AIs could and would do all the things.
I think the premise in the Detractors argument, as stated above, is wrong. I think superintelligence is likely to be coming soon, as do the labs. Many do not agree. If you are one of those who do not agree, then you should absolutely not want to implement Plan A, or anything like Plan A, and you should say so plainly.
This is true whether or not you want to further engage with the scenario anyway, to consider the hypothetical where you are wrong. That’s up to you, and ‘no’ is a respectable response.
Top 10 criticisms other are, translated into my language (not intended to pass ITTs):
A lot of these being symmetrical is a sign that the scenario is doing something right.
My basic responses to these objections:
Extended versions of most of these are included later in the post.
Proactive Response To Objections
There is a long tail of other objections as well. Even after listing the top 10 I found that a large percentage of responses to my open thread were not covered.
AI 2040, like AI 2027, tries to proactively answer many of the concerns and objections people have, both common and rare, including via supplementary material.
For a rare example, Tom Davidson is worried about the ‘dry tinder’ problem of creating a compute overhang which they answer here and via the idea of mutually assured compute destruction. Curt Tigges also worries about the overhang.
Nathan Young offers Common Notes on Plan A, an attempt to bring community notes to the entire internet.
Initial Introductions and Endorsements
Scott Alexander, one of those who worked on it, writes an introduction and justification here.
Whereas here are some shorter pitches for why you should read it and it matters:
Jack Galler called it the most important thing he’s ever read. I wouldn’t go that far.
A Positive Vision
They call Plan A a positive vision.
One could also call it an optimistic scenario, as per Richard Ngo.
One strong criticism of AI 2027 was that it laid out two scenarios, but did not tell us what we should do or offer us a practical path forward. In the default scenario, which lays out what the authors expect to happen, we all die.
In AI 2027’s alternative scenario we do a hard reset at a crucial moment, which is less a strategy or plan or positive vision and more like a hail mary pass, where the authors make things turn out well to show that it is in theory possible.
Plan A updates to start from our present situation. Rather than being primarily a prediction, it lays out a positive vision of a possible future where we coordinate (including using various enforcement and verification mechanisms) to slow down the development of superintelligence (ASI), and give it the best chance to go well. Ultimately in 2040 the torch is passed to the AIs, the singularity proceeds in earnest, and hopefully we got it right on the first try and things work out.
The proposed implementation of Plan A is that America and China reach a mutually beneficial deal to slow down AI development and share research information. Joint control is established over existing and new chip supply, with common knowledge of the location of existing concentrations of chips, and universal auditing of data centers. Both sides can verify that the deal is being upheld.
This can sound rather fantastical, but if a few months ago you had described how the US government has responded to the whole Mythos situation, you would have sounded rather fantastical then too. Something has to give. The authors really did invest a lot of work on seeing which paths forward were viable here.
Things go well, including economically along the way, as we make a series of good decisions, and we get to a happy ending, although not without speed bumps as even a slower transition is rather sudden and even if the technical issues are handled there remain lots of real problems. They consider alternative ways this could go, such as China cheating aggressively or flawed safety cases being approved.
During the transition, America experiences explosive economic growth, and uses some of that to pay out a rapidly increasing Citizen’s Dividend, as an extremely generous form of UBI, and also a small fraction is devoted to various forms of defensive acceleration. Various issues are discussed along the way.
If things going vertical in 2040 still sounds super fast, that is because in the authors’ current baseline or default scenario this happens in 2030.
A key focus of Plan A is total research transparency. Daniel Kokotajlo made this diagram to illustrate why he cares so much, with further explanation at the link.
Alternative Plans
They briefly also describe the scenarios where we instead choose Plans B, C, D or S:
Roughly speaking, they predict the presidential election in 2028 is all about AI because by that point it is obvious AI is the Main Thing, with candidates throwing around bold agendas. Then in 2029 one gets implemented, which they are hoping is Plan A.
There are a ton of details and explorations of different topics in Plan A. Those involved put a lot of effort into parts almost no one will see. I could easily write a response post at least as long as the scenario to go into them. This is not that post.
A lot of those details sound like science fiction, because they are serious attempts to predict the future. Best start believing in science fiction stories, because you’re living in one. There are any number of ways that can go, and this might be wrong, but some very science fiction things are happening and a lot more will be happening soon.
Plan S for Shutdown
This perspective, a form of Objection #7 above, is negative on Plan A’s chances of success, but positive on the chances it causes us to gather enough impetus to shift to Plan S, which is a full shutdown.
Or alternatively:
If you are of the camp that the odds are against us and the situation is grim, as I am and Nate Soares is even more so, then every winning scenario involves a lot of things going unexpectedly well.
The reason they give for not choosing Plan S is that they believe that an agreement would be unstable over time, so you have to keep moving forward and also give people a dividend of sorts. The agreement breaking down would potentially be quite bad.
Raymond Arnold thinks out loud here about the importance of ‘how long would such a deal hold?’ in terms of choosing the best plan within the class of As and Ss.
Something (Unexpectedly Good) Ever Happens
Every scenario, including whatever turns out to happen in real life, is going to involve some rather unlikely things happening. For any given thing, ‘nothing ever happens.’ For all things combined, surprising individual things happen all the time and each might be good. If this is not intuitive, look back on the last year, or any other period.
If you have a good plan to change the world, it is going to involve causing a bunch of otherwise low-probability events to happen, either intentionally or via good luck.
Of course, any good plan involves not relying on too many specific good things happening. You need to be as robust as possible, but no more robust than that.
Plan A here definitely has some flexibility and room to recover from failures or to reroute, but not unlimited room. It involves good fortune, but not maximal amounts.
Thus Selective Optimism
Richard Ngo helped work to critique and improve Plan A during its construction, and wrote a critique of the high level framing, called Selective Optimism. He correctly describes Plan A as an ‘optimistic forecast’ but notes this makes it hard to tell which parts of Plan A are things to aim for versus things included for realism.
A lot of people reported struggling with this issue.
There’s no good answer, and no one best answer. If you tell a story, your story is going to have to choose one path out of many. It must balance realism with your hopes, in addition a bunch of individually unlikely things.
The alternative is to not tell a story and not have a scenario at all, to never tell stories or offer particular scenarios at all. I think that’s clearly far worse.
Most centrally, Richard would prefer a slower handoff than described in the scenario. I would as well, as would the authors of Plan A, and this objection is not uncommon.
Thomas Larsen points to this supplemental to illustrate which parts are aspirational versus realistic.
Quickly, There’s No Time
Richard Ngo recommended taking all the dates out to avoid people mainly focusing on the date 2040, since what is valuable are the details.
I think this echoes a lot of similar criticisms of AI 2027, and I see the argument but I rejected it then as well. You can’t tell a story like this properly without dates, you have to pick some point on the curve of potentials to lay out in a potential future.
If you try, guess what the first criticism would be? They didn’t put dates on. What a joke, what do we even do with that? And those people would be right.
In general I don’t like Isolated Demands For Anti-Virality lest someone somewhere accuse you of giving people the wrong idea or focus on the wrong aspect.
Yes, some people will stop at the headline, or latch onto only the date, and you can change the frequency of that somewhat, but mostly that cannot be helped, and those people were never going to engage with the real content anyway, or if anything this makes them more likely to then seriously dig in.
Race Conditions
Richard also challenges the ‘race with China’ framing, expecting other disruptions, but I predict that any such move towards Richard’s framing would cause most to dismiss the scenario as unrealistic, and also risks further politicizing the whole thing. Any failure to acknowledge that America and China really don’t trust each other at this time would make the whole thing land with a thud or worse.
I too of course have always hated the race framing, and tried to fight against it for years, but as Larsen says that’s how people think and talk now, especially in DC, and you have to acknowledge this.
For the opposite view on that, here is Poplicola saying that Plan A fails as a scenario because it doesn’t treat American hostility to China, and its view of ‘authoritarianism’ as the real existential threat (at least when it comes from outside the house) seriously enough.
Also things like this:
I don’t see that proposal as an especially hard part of the problem, as long as the economics work out.
I think in general the way politics works is until the reasons are compelling things look hopelessly naive until suddenly things change, the atmosphere shifts and then they happen. Peace treaties (and other agreements) often involve concessions that looked impossible, right until the deal got signed.
And I think Richard is conflating the ‘race with China’ we are currently in, as in allowing OpenAI and Anthropic to push full speed ahead while we make at least some attempt to deny China chips and otherwise tip the balance, while any attempt to do anything to help is by default dismissed as ‘if we do that we Lose To China’ with a theoretical full-on planned Race With China that I agree we are not doing and are not likely to do any time soon.
This Is A Lot Of Diffusion And Economic Growth
Richard ends with a criticism of the expected fast pace of both AI progress and the diffusion and impact of AI in the broader economy, saying he doesn’t expect AI progress to generalize. You certainly can argue that the forecast here is overly optimistic, even as things spend years going along exactly the path that these styles of prediction would predict. I just don’t think a strong form of that is the way to bet.
In terms of the diffusion and real world impact, which I think is the strongest critique, I agree with Shakeel Hashim here that this does not impact the recommendations much. The economic impacts during this scenario might go a lot slower, but that doesn’t change the central path.
Would the scenario have been memetically more fit if it involved less economic growth in the 2030s? Probably, and I do think the authors are being optimistic here, but the authors are telling us how they think such a scenario would actually play out, and they explain why they think this.
At the limit, with sufficiently advanced AI, diffusion stops being an issue for long, and those who pretend otherwise are denying the premise or being silly.
Thomas Larsen responds here to Richard on all his fronts.
Living In China
Topynate also has some good criticisms. We could use more Chinese perspectives, and more concern about how willing people would be to make a deal. I am 100% fine with trading away half the lightcone, but others may not be. I agree that justifying Plan A requires that alignment be hard enough that we need Plan A, and also doable enough that Plan A can work before it unravels.
Yishan’s model is that China would plausibly be willing to accept AI equality with America, but not a deal that enshrines American superiority in the area. That would mean any deal like Plan A is DOA.
Yishan suggests you could do a Plan A1 that acknowledges this reality, but I think you basically couldn’t. If he is right, then there is probably no ZOPA (Zone of Possible Agreement), as in even a maximally wise and cooperative American government cannot do that without getting a lot of other things in return, especially if this is asked for up front, but desperate times could change things. The Plan A scenario involves rebalancing things like share of robots, so they’re thinking about this aspect.
Prakash makes the same point, that China is not going to accept staying behind and America will insist on staying ahead.
I do agree that finding a ZOPA will at best be difficult, as the two sides have very different views. But there is indeed a long history of similar deals looking impossible for this reason, until suddenly a deal does get made when there is sufficient impetus.
A lot of this is for me that yes, in the world of this level of AI capabilities, there is quite a lot to be gained from a deal. When there is a lot of gains from trade, trade often finds a way.
Planning For Shifting Overton Windows Is Essential
You should have been able to predict the Trump COVID unemployment benefits in January 2020. I didn’t, but that’s on me. If you can get to ‘extended lockdowns that last months or more’ then something like that likely follows. The part that I did not expect in advance was where the lockdowns would reach an equilibrium indefinitely, rather than either solving the problem or being overwhelmed.
Covid also illustrates that the impossible suddenly becomes possible.
Shutting down all ‘non-essential’ activities and services and almost not letting people go outside? Trillions in cash handouts? Vaccine mandates?
It’s all a crazy violation of liberty until the alternative is worse, and that’s with only a low single digit chance of dying. And yes, more and better planning in January and February, even as almost everyone dismissed your premises as crazy and your proposals as authoritarian, would have been extremely valuable.
The Standard Handwave
There are of course those who will pattern match anything that involves any form of ‘slowing down’ or ‘pause’ or ‘buying time’ or any strategy that involves paying a nontrivial price, and queue up their standard responses, without engaging further.
You might think if nothing else that Mythos, and the White House response to this, would make people rethink such dismissals. Well, maybe it sometimes did.
Many simply call those involved an unkind name and move on, feeling superior.
The first example of the polite version of this I saw was here from Maxwell Tabarrok, as retweeted by Tyler Cowen. This is basically ‘oh well you see if we had responded to this exponential too early that would have been bad and looked foolish, so we can only respond too late’ and ‘don’t worry markets solve everything until proven otherwise.’
We also get a side helping of ‘until you know exactly how to solve the problem you’re not allowed to do anything about it’ and also a failure to actually engage with the details of the proposal.
And also ‘the idea we should slow down AI to buy time has no credibility given events, despite the actual slow downs that happened in highly recent events in response to Claude Mythos.’
If you think that’s not a fair characterization, okay then, here’s his full comment, so you can judge for yourself.
Except of course they don’t phrase it like that. But yes, the general view is this, from June 26:
A lack of market incentive is not a good signal of importance. An understanding of what externality or other issue is mechanistically causing a lack of market incentive, however, very much is a good signal that there is an opportunity. Demand of a sort is unable to induce supply. A different kind of profit is plausibly available to be made, or utility is available to be created.
Some Equate Any Controls Over Compute To Authoritarian Dystopia And Those Same People Mostly Think Superintelligence Won’t Happen
We’ve been through many years of this idea, that somehow any controls on chips or compute or models means authoritarian dystopia. This is objection #6.
There is a remarkably high correlation between those who equate such things, and those who deny that superintelligence will be developed any time soon.
Very obviously this should mostly depend on whether you are what I call ASI pilled.
You can be what I call ‘AGI pilled’ but expect returns to rapidly drop off, and thus reject scenarios like AI 2027 or Plan A as unrealistic, and thus you are evaluating any proposal largely on the basis of risk of abuse or concentration of power. If you are what I call ‘ASI pilled,’ where you see the tech side of the scenario as highly possible, then that gets a lot harder, although you can still reject this style of proposal on the grounds that it doesn’t buy you enough to be worth the price.
I am not referring to Krier in particular, but in general it seems like those who want to be concerned about governments therefore choose to not be ASI pilled. Or at minimum there is a common generator causing both responses, that everything must forever be some form of ‘economic normal’ and why aren’t idiots understanding that.
Vitalik Buterin Is Right, The Crux Is Future AI Capability Levels
I respect the hell out of ‘superintelligence won’t happen any time soon, so stop proposing expensive ways to deal with it.’
Except the part where you’re wrong about that, but if that is your True Objection then yes please do state it that way. Vitalik is characteristically very strong on this.
Exactly. If you buy the premise you do not have to buy the particular plan. You do need to take this seriously and not dismiss it out of hand as naive or authoritarian, and if you don’t like the plan you should be trying to think of better plans.
This seems totally fair. The labs think this is happening but that does not mean that you have to agree with them.
It would be a very large mistake to do Plan A, or another similarly disruptive plan, if superintelligence was never coming before 2050, or AI will indefinitely remain a ‘normal technology’ no matter what.
D/acc is a way to ‘play it safe,’ as in work on things that are clearly good. Those things are good things to do, but in the superintelligence scenario that’s not going to get it done. Vitalik realizes this.
Again, yes, exactly. There are a bunch of places where there is no Secret Third Thing you can do, and the tradeoffs are real, and there is no safe play. But you can work to improve your options.
In theory this would be great. If such a deal were credible, I would love to say ‘well, let’s agree that if [X] happened then we would do [costly intervention basket Y].’
The problem, as always, is who is going to make that agreement? How can we expect it to be honored? There is a long history of various things, that used to be considered obvious red lines or alarm bells, being rushed by almost without a second thought.
I, too, see no ‘non-naive’ options, in the sense that people object that Plan A is naive. You have to be naive from at least some point of view.
One can argue that the costs of Plan A and similar interventions are small compared to the benefits, so you don’t need that much confidence you live in a world with future superintelligence before such a path makes sense, if you think Plan A is good in ASI worlds. I don’t think this is obvious. The costs in non-ASI worlds are quite real here. As are the costs in ASI worlds, especially if some things go sideways.
The Authoritarian Objection
I’ve chosen a good version of the authoritarian objection to respond to. Kudos to Ramez Naam for making a real attempt to parse the actual scenario and its proposals and listing them out, rather than only responding to the headline summary.
And also for acknowledging the good intent, and that the plan is designed to minimize the amount of restriction and authoritarianism, given what must be done.
I am less thrilled with his characterization here of this being about ‘a made up threat.’ If that’s how you view all this talk of superintelligence or sufficiently advanced AI, and are reasoning on that basis, then that’s the crux we need to be discussing, since otherwise nothing else matters.
I think he believes what he’s saying here, but I basically don’t believe this can be both good faith and a self-aware response. There are plenty of good models saying this is possible, and Naam is rejecting those models of superintelligence as ‘not believable.’
My experience is that such requests are impossible to meet, and the goalposts will be adjusted as needed. Also, there is no reason to presume that it would need to be software-only in order to require Plan A levels of response. In AI 2027, and in Plan A, the takeoff very much is not software-only.
Anyway, back to the good part of the objection:
If you think that China’s current oppression of people is a greater danger than the dangers posed by highly capable AI, then yes, obviously you should oppose all of this.
Ramez Naam seems to be:
I find it hard to engage with people who state Claim #5 so strongly, especially given people have actual track records. I disagree.
Claim #1 seems very clearly hyperbolic and unreasonable. Can we please stop equating restrictions on compute use to the worst governments in history?
Claim #4 would be enough on its own, if true. Obviously, if you think AI will work out fine without major interventions, or other things are ultimately far more important, then you shouldn’t want to implement Plan A.
Obviously, I disagree here in the strongest possible terms. Claim #2 is not a plausible alternative world, for the same reason, and you can’t use it as the comparison point.
That brings you to point #3. We are, if you accept the technological premise of those who wrote Plan A, which to a large extent I absolutely do, in quite the pickle. By default we all probably die. All known light touch solutions do not help much. Any solution will involve the sacrifice of sacred values, since among other things one of those sacred values is ‘humans remain alive.’
Concepts Of A Plan
Yes, if we can find a way to do it, we all want the various desiderata that Naam lists, which are commonly expressed by many including Altman and Amodei. We want to be able to rely on liberal democracy and free markets while creating minds increasingly more capable than humans at increasingly many tasks, and have that work out for humans. Yay applause lights.
But there is no proposal there. How would this work, even in outline form, under the types of AI capabilities described in Plan A / AI 2040, even if alignment efforts were successful? Is there any proposal that is not, mostly, ‘This Is Fine?’
Indeed, there are some like Seb Krier who reject the very idea of having a plan. I would like to say this entirely misses the point. Even when plans are worthless, planning is essential. Having a concrete idea of what you want to do is a good idea, even if no one has the authority to do it. If you are dismissing the idea of planning to steer the future at all, on principle, as dangerous or wrongheaded or worthless, then you lose.
The other thing you can do in response to a plan is propose a better plan, or a better model of what might happen. Bleys Goodson is trying to figure out what he calls a less authoritarian way to a safe open-research ramp-up, and offers enia.cc to help with modeling what things will look like. I appreciate that Bleys is open about not yet having such a plan to present.
Seb is actually saying, no, we should just muddle through, that this is the best strategy. And I very strongly think no, that’s a really dumb strategy, but even if we do muddle through that is a good reason to figure out our next options.
The Kitchen Sink
Seb Krier also throws the kitchen sink of usual other objections out there, in what I read as the disappointed professor tone, most of which also apply to AI 2027, and to basically any proposal that we collectively try to solve the problem.
One interesting note, that Krier frames as part of an objection (in his #9) is that ‘we already have a lot of Plan A stuff going on, that people thought was never going to happen because normies would not wake up.’ That is not an argument against Plan A, or an argument ‘against alarmism,’ as he puts it, and also I take strong issue with the term ‘alarmism’ when presented like this as a clear negative. Alarms and warnings exist for a reason.
Seb Krier has precommitted to not engaging further, and as I said this feels like reiteration of past arguments and throwing the kitchen sink more than particular new points, so I won’t engage further with him in particular unless he requests it.
Dustin Juliano is another example of throwing the kitchen sink, disagreeing on almost every level, dismissing many distinct things as outright Can’t Happens. He is refreshingly up front that this is what he is doing.
Selective Claims Of Authoritarianism
I also think that the same objections raised here to Plan A can be made, in far stronger form, against past and present government policies across the board even in normal times, and definitely under crisis or wartime conditions.
Consider our response to Covid as per the discussion above, or World War I, or World War II, or the way we collect and report on income taxes, or the war on drugs, or the rules on who can do work or provide healthcare or do research or build which houses or the way we implicitly regulate online speech already. Or look at the way Europe and the UK currently restrict speech and the use of computers and phones, or labor, or air conditioning, or lots of other things.
Do I want to roll a whole lot of that back? Oh yes, absolutely.
Still, none of that means that America, the UK or EU is an authoritarian state, and certainly I hope one would not equate them with 20th century fascism or communism. The same would apply to these kinds of compute and other restrictions.
This is not that high on the list of dangers, but to be as ‘normal’ sounding as possible: Compare the restrictions under Plan A to what happens if a lack of restrictions leads to a Covid-level pandemic. Which restrictions would you prefer?
I welcome a discussion about the particular interventions and restrictions, and whether they would actually be necessary in such scenarios. I am absolutely not endorsing this exact list of responses.
But oh man is ‘the spirit of the 1st and 4th amendments’ being violated a lot worse than this, and for far worse reasons, as it is, all the time, in a currently largely free country. Plan A raises obvious dangers from centralization and expansion of government power, but it is not a centrally authoritarian vision. Indeed, in many ways it is designed to try and head off authoritarianism, including what one might call ‘authoritarianism by the AIs.’
You Either Can Steer The Future Or You Cannot
A lot of these objections have little to do with the particulars of this proposal.
Instead, two more global objections loom largest.
The first is not believing in superintelligence. If you don’t think superintelligence is coming by 2040 either way, then Plan A does not make sense. Fair enough.
The other objection is, essentially, that if humans have the ability to importantly and collectively steer the future, that this is bad, because it would be concentration of power, and would inevitably be abused, and that would be so much worse.
Or the objection is that the humans steering the future cannot be governments, because that in particular is really terrible.
This has little to do with the particular steering that Plan A has in mind.
So, one more time:
The central problem is either humans can collectively steer the future, or they can’t.
(And also, if you don’t give the humans good tools or let them plan and prepare, for fear someone might use those good tools, they will still grasp around for whatever is on hand, and use bad tools, and this will go worse, as we have recently seen.)
The entities steering are either governments, or they are private individuals and organizations that are not government, or no one is steering.
You either can do and actually do the things that actually influence the path of AI development and diffusion, or else you can’t or don’t.
Historically, we have been very fortunate that a mostly hands-off, unsteered version of things has been the winning strategy on almost all levels, yet even that required quite a lot of particular steering. As discussed above, we steer a lot of things quite a lot.
If we get technological developments at the magnitude or speed of those envisioned in AI 2027 or even the slowed vision of Plan A, then the alternative to humans steering those outcomes is that we solve for the equilibrium.
The equilibrium is increasingly many things are turned over to the AIs. The equilibrium is that the AIs rapidly end up in charge because the alternative is being outcompeted. This happens without AIs tricking anyone or firing a shot. The AIs, and their deals or competitions among themselves, and their goals, steer the future.
And that’s the good version.
Once that happens, there is no undoing or fixing it, and by default I do not expect humans to long survive, or the things we value to long endure.
Cooperative Alignment
The approach in Plan A, as presented, treats alignment as a technical problem and a control problem, of needing to create AIs that are trustworthy.
They only consider one form of cooperative alignment, or the idea that the AIs will or won’t cooperate with humans largely based on game theory and decision theory, and based on how we choose to treat them. They focus only on how to make deals with misaligned AIs, rather than how to align with AIs in the first place.
I think within the context of writing the scenario and not getting too distracted basically This Is Fine. If on consideration we would be better off doing alignment this other way, then we would choose to do it this other way, and I don’t think it changes the central strategies so much.
But yes, maintaining this attitude would be a serious mistake on the alignment front, as would plans that seem like they reward misaligned AIs more than aligned AIs at some points. These are details that can be workshopped.
I also don’t view the game theory the same way here. Once you hand off to the kind of AIs that exist in 2040 in Plan A, the humans don’t have leverage, and that’s why this is far more decision theory than game theory: Human cooperation from that point forward is irrelevant. I won’t get into my view of the decision theory involved here, except to say that yes it should be something we spend a lot of attention on, and I hope to see further engagement on this point.