Some context for this post: I’ve been working part-time as a consultant for the AI Futures Project over the last year. Most of the work I’ve done for them has involved critiquing and suggesting improvements for their AI 2040 scenario—some of which were addressed, and some of which weren’t. To their credit, they asked me to write up my remaining critiques into a post that would accompany its launch. In the rest of this post I’ll discuss my three biggest high-level criticisms of AI 2040.

Before doing so, I want to emphasize that there are many interesting and thought-provoking details in the scenario. I’ve focused on the high-level framing of the scenario because that’s where my main disagreements lie; given the scope of these disagreements, it’s hard to evaluate the details.

Since the AI Futures Project paid me to develop and write this criticism, you shouldn’t take this as a fully unbiased perspective. However, they haven’t reviewed this piece, and in general have been open-minded about receiving criticism (as their request for me to post this today demonstrates).

Finally: the preview image for the substack version of this post comes from this video of a dad shouting to his son “don’t crash into the tree!” The relevance will hopefully become apparent.

Forecast vs recommendation

The most important thing about AI 2040 is that it’s neither a forecast nor a set of recommendations, but rather something in between: an optimistic forecast. The key benefit of the optimistic forecast format is that it’s able to convey many details of good futures, showing us how they could all fit together. The key downside, however, is that it’s hard for readers to know which parts of the scenario the authors consider to be actively desirable, versus neutral, versus undesirable but included for the sake of realism.

The most obvious example of this issue is literally in the name. In their main recommendation, humanity hands over control of the world to AIs in 2040. Is this the best scenario that the authors can imagine? Or is it the most realistic out of all the good scenarios? Or is it a good scenario that was chosen to be easy to persuade people to aim for? These distinctions are crucial for inferring the authors’ views, but aren’t clear from the text itself.

Personally, I’d want a far slower handover. Going from “AI Alignment Is Now a Science” in 2038 and “Beginning to Trust AIs” in 2039 to “Passing the Torch to AIs” in 2040 is extremely abrupt. Even if experts become confident that this is a good idea, there’s no way for most normal people to understand and consent to this process so quickly. And indeed, the idea of a coordinated handover seems to require that the process is being run by internationally-coordinated regulators, rather than letting the citizens of each country decide how much influence AI should have in that country. Perhaps the AI 2040 authors would agree with me, and say “unfortunately, nothing more gradual or democratic seems feasible”. But the “optimistic forecast” structure obscures that information by excluding both implausible and undesirable possibilities alike.

Another example of the “forecast vs recommendation” issue arises in the description of the world post-slowdown. The authors are careful to emphasize that the slowdown is consistent with a lot of progress, with quotes like “we’re at previously unimaginable levels of it not feeling like a slowdown”, “the world is going to radically transform despite the pause”, and “five centuries in five years”. But I don’t know if anyone actually wants to see five centuries of scientific advancement in five years. Even staunch accelerationists likely agree that this would be extremely destabilizing—they’re often accelerationists precisely because they don’t expect things to go that fast.

So another way of putting the “forecast vs recommendation” problem is that, when making an optimistic prediction, you face an inherently political choice of how optimistic to be (compared with what you think the default outcome is). The AI 2040 authors might justify their portrayal of the deal by saying that slowing down more is not politically feasible. But in doing so, they’ve advocated for faster progress than almost anyone else endorses, as well as a more rapid handover of power to AIs than almost anyone else endorses. So I’m concerned that they’ve made themselves part of the process by which better outcomes are seen as politically infeasible—now even staunch safety advocates can be portrayed as wanting rapid progress.

My personal opinion is that the scenario is mainly valuable for the details it sketches out, and the actual timeline that it gives should be largely ignored. Indeed, I recommended to the authors that they remove all dates after the deal is implemented, to indicate that the pace of progress from that point on should depend on factors that are very unpredictable to us (like the speed at which alignment research progress, or what citizens vote for). Unfortunately, putting “2040” in the title instead means that the single-sentence summary many people will hear is: their recommended future involves handing over power to AIs by 2040. I worry that (like the original AI 2027 scenario) this title is optimizing too much for catchiness, while failing to convey the core message of the scenario.

International vs domestic interventions

AI 2040 is structured around the idea of defusing the race with China. However, I’m concerned that this is another case where mistaken evaluations of what’s politically feasible make the scenario counterproductive. As Katja Grace has pointed out, the “race” metaphor is a misleading one, because it bakes in the idea that being ahead is “winning”. I’d add a stronger claim: that well before reaching superintelligence, both the US and China will see serious internal political disruption from AI, which will make both of them very cautious about continued progress. So we shouldn’t expect the future of AI to be well-described as a race, except insofar as the “race” metaphor becomes a self-fulfilling prophecy (like it has between domestic AI companies).

In the US, I expect this disruption to mainly play out in the form of conflict between Republicans and Democrats. Both sides are already very wary of how technology can be used against them. Republicans have experienced a decade of social media censorship across many different platforms, and are very concerned about similar dynamics playing out with AI. Meanwhile, Democrats are constantly expressing worry about the power of entrepreneurs like Elon Musk and Alex Karp. Increasingly capable AI will make both of these fears sharper (especially as the affiliations between individual AI companies and political parties become stronger).

Within China, the main axis of conflict is not left vs right, but top vs bottom: specifically, the CCP’s control over the Chinese population. Here it’s more plausible that AI is a stabilizing force, since it can be used for surveillance and censorship. However, there will also be many ways that widespread access to powerful AI could allow the Chinese population to express discontent with their government. In general, Chinese leaders are much more focused on stability than US leaders, and are much more capable of and willing to do long-term planning towards that end. So it seems likely that they will prefer to proceed cautiously.

On the picture I’ve just outlined, leaders within each country will become quite scared of the effects of AI on their domestic balance of power. Will fear of the effects of AI on the global balance of power outweigh that, pushing them to race? I don’t see a strong case for it. Historically speaking, domestic issues are far more salient—even at the height of the Cold War, the US was roiled by conflicts over civil rights, the Vietnam War, etc. There were many ways that an internally unified US could have “raced” much harder against the USSR—but in practice the hardest “racing” was focused on the fairly isolated space race, rather than anything which would have required broader economic and cultural reorientation.

Today, we’re already seeing serious concern by prominent figures both on the left (e.g. Bernie) and on the right (e.g. Bannon) about the domestic consequences of AI. Conversely, concern about the implications of Chinese AI has been primarily publicized by competing factions within the broader Silicon Valley ecosystem. AI companies’ use of China as a bogeyman to justify continuing to scale up AI is fairly straightforward. The focus on China from AI safety advocates is more puzzling. By this I’m thinking of the role of AI safety in pushing for export controls against China, and advocacy like the Situational Awareness and Superintelligence Strategy reports.

My sense is that the field of AI safety overall has been making the same kind of mistake as the AI 2040 scenario does. From a very abstract point of view, if we zoom out enough, it seems like there “should be” incentives for the US and China to race. Therefore people like the AI 2040 authors take eventual racing as a given, and try to figure out ways of making AI safer given that. However, in doing so, they implicitly frame the discussion to make racing seem like the default option, and not racing seem naive. From the perspective of this scenario, the idea that the US and China could trust each other to do the reasonable thing isn’t even worth considering (except as an aside under the label “Domestic-first Plan A”).

This seems like the same kind of mistake that von Neumann made in his extreme hawkishness towards the USSR. In advocating for preemptive use of nuclear weapons, he was abstracting very far away from common-sense intuitions about cooperative strategies, in favor of a kind of (flawed) game-theoretic logic. He was thereby also contributing to a self-fulfilling prophecy that the US and USSR wouldn’t manage to muddle through. Yet in fact there were many concrete frictions preventing the two superpowers from jumping to what von Neumann thought was the equilibrium. This is a trap that intellectuals are much more vulnerable to than typical citizens, whose reasoning is much more concretely grounded. It’s also sometimes driven by strong emotional instincts disguised under a layer of intellectualization (e.g. the Situational Awareness report betrays in many ways a sense of excitement about racing against China, even as Leopold claims to prefer otherwise).

To be clear, I’m not claiming that the US and China definitely won’t race each other into disaster. All of this seems very much up for grabs. However, I think that frames which bake in the assumption that the US and China will behave in very adversarial ways towards each other are misleading and harmful (particularly when they do so under the banner of AI safety, thereby undermining the potential role of AI safety as a focal point for cooperation).

This assumption is conveyed throughout AI 2040—e.g. through the main flow chart of possible options, through statements like “the US and China don’t trust each other” and “neither side trusts the other to honor Plan A”, and through the focus on ensuring that the treaty is fully verifiable. Because of this, Plan A underrates the importance of domestic politics. Even if we ultimately need a highly-verifiable treaty, it would be much more robust if it were grounded in existing domestic regulation. Conversely, a deal that’s top-down from the beginning could easily be used as a way of consolidating domestic power.

Fast takeoff

My sense is that the AI 2040 authors underrate the criticisms I’ve raised above in large part because they expect superintelligence so imminently. My third criticism of AI 2040 is that it buys too uncritically into the idea of a sharp takeoff of AI capabilities. I’m not denying the possibility that this could occur in principle. And it’s true that the last decade (and especially the last 5 years) of AI capabilities progress have been blindingly fast in most measurable ways—far faster than almost anyone (except a few prescient forecasters like Legg, Amodei, Kokotajlo, Leike, and Kurzweil) predicted. However, the real-world impacts of AI (aside from the ballooning revenues of AI companies) have been underwhelming thus far, especially when compared to the progress in measured capabilities.

This great divergence between measured capabilities and real-world impacts is still not very well understood. Below, I’ll give some speculations about what might be causing it—but first, I want to outline the core intuition making me think there’s something important to be explained here. Imagine going 10 or 20 years into the past, telling people a selection of benchmark scores of current AIs, and asking them to predict what the world that contains them looks like. I expect that they would have described a world that was dramatically transformed—perhaps one in which AIs already wielded enormous political power, or had made far-reaching scientific breakthroughs, or at the very least had decimated white-collar jobs. Yet none of these have occurred so far.

So any predictions that we’ll soon hit sharp “vertical” growth in AI capabilities, with a corresponding gain in the power of AIs (or whoever controls them) should explain why the next decade won’t involve a similar kind of divergence as the last decade. My own picture of our median default trajectory involves a gradual accumulation of power by the AI industry, with control over AI becoming about as geopolitically important in 10-20 years as control over the US or Chinese military is today. To be clear, this is still an extremely rapid transition by historical standards. But it falls far short of the rapid jump to world domination that AI 2040 is trying to avoid.

Where’s the specific crux between the two worldviews? I don’t know, but I’ll list some hypotheses. One is on a technical level: large neural networks rely more on memorization than humans do for any given task. So their performance on tasks that are closer to their training distribution creates an illusion that they’ll generalize further than they actually do. Another is on an economic level: there might be “weak links” in the process of deploying AIs that slow down their impacts (as discussed by economists like Chad Jones).

My leading hypothesis is that something more subtle is happening, analogous to the ways that humans fail to make progress. For example, the field of psychology contains many incredibly smart and hard-working psychologists, who have passed many graduate-level exams with flying colors. Yet over the last few decades their contributions have failed to “add up” to robust progress on understanding the mind—and in many ways have taken the field backwards, with the replication crisis making it harder to sift out good research.

We could sharpen this point further by looking at human history, which contained many periods during which a large number of smart people failed to make significant technological progress, or actively regressed. So basic models of fast takeoff may be neglecting the difficulty of cooperating to make cumulative contributions towards a common goal. This fits with Moravec’s paradox, since cooperation is something that evolution has selected humans very hard for.

However, regardless of the specific cause of the great divergence between capabilities and impacts, it’s something which deserves to be grappled with more directly.