This post is based on my personal views, which mostly overlap with the views of my employer ControlAI but does not necessarily fully reflect them. This applies in particular, but not exclusively, to technical opinions about AI development and geopolitical predictions.

You might’ve heard that superintelligent AI (ASI) poses extreme risks like human extinction and other comparably undesirable outcomes.

If you’re like me, you probably looked into possible solutions. And if so, you may have found a range of reassuringly tractable theories of change. To name a few:

• Technical AI safety research agendas
• Racing to ASI so your favorite company or country can get there first and prevent anyone else from building “bad” ASI
• Building a good ASI and handing it control over the whole world (so that we don’t have to be subject to any evil human dictators)

If you think about it, all of these feel quite convenient, especially if you’re a tech-leaning person: you don’t need to change your career at all. Just keep working on your favorite ASI project, and things will work out.

It’s quite easy to come across theories that predict good outcomes without needing to change your strategy at all, even if you’re actively working to bring about ASI as soon as possible. I see these as being mostly semantic stopsigns. Most of them about AI alignment being feasible:

• AI alignment is easy and people are working hard on it, so it’ll probably be ok.
• AI will help us do alignment research.
• Iterative deployment will help us catch problems before AI gets too powerful.

In this post, I want to show you that even if the theories of change mentioned above were applied extremely successfully or if AI alignment actually turned out to be technically easy, all the value in the world is still on track to be destroyed because of AI development. This means, mostly, human extinction. It also includes scenarios that don’t literally qualify as human extinction but are still comparably undesirable. For example, the least-bad scenario I consider in this post is all-out war between nuclear superpowers, and the worst scenarios are suffering risks (s-risks).

There are many ways in which AI development can destroy the world. In this post I'll explain the three most likely pathways. Any plan for survival needs to address all of them and prevent those threats from being realized.

In my opinion, the only solution that addresses all the potential threats is to achieve two things together:

• A level of global coordination sufficient to stop or slow down progress toward ASI, such that all parties can ensure the trajectory of AI development happens according to the consensus and interests of most parties.
• Mass awareness across society of the implications of ASI and of the worst risks posed by AI development, so the various parties can correctly judge whether allowing development to proceed at a certain pace is in their best interest.

This is why I work at ControlAI, which, at the moment, I believe is the best bet for moving the world closer toward this state. However, in this post I won’t try very hard to sell my favorite theory of change (ControlAI's already got a post for that!).

Rather than arguing for international coordination, I will simply describe how common theories of change that don’t take this route don’t prevent the world from being destroyed.

Preamble: pressure to cut corners invalidates most theories of change

Before explaining the multiple ways in which AI development can destroy the world, I need to introduce this concept as it will come up over and over again. ASI would offer its creator an insurmountable competitive advantage, if it didn’t kill them. This means there is an extreme pressure to cut corners to be able to reap its benefits as soon as possible.

This topic has already been explored, so I won’t go into it too deeply. If you want to see an explanation of why ASI is so powerful, look at “Situational Awareness.” If you want to see my own game theoretic analysis of an ASI race, read my paper: “Modeling the geopolitics of AI development.”

Suffice it to say, a large advantage in AI capabilities would allow its creator, or the rogue AI, to perform an extremely low-cost, low-risk takeover of all other countries and actors in the world. From that point on, they’d maintain a singleton: that is, a permanently unassailable total control over the world.

Once you understand this, it follows that you have to ensure no one else builds an AI capable of overpowering you. Assuming you don’t have the means to do this, then you have to be the first to gain this insurmountable advantage, before someone else does it and kills you.

First of all, let’s step into the shoes of a state actor, or any other powerful actor, and see what actions immediately come to mind after realizing the importance of ASI: “If any other actor has an ASI project more advanced than mine, I will try to steal, hijack, or otherwise take control of this ASI project.” Between states, this means espionage and sabotage, including extreme measures up to and including acts of war.

It also means that skilled actors, such as competent psychopaths or propagandists, will try really hard to gain control over the project. In the case of competent psychopaths, they may manipulate their way into the project’s leadership.

This also means that if you are a private company, there is not a chance in hell you will complete your ASI project and get to keep the ASI because:

• Your government will take over the project!
• If your government is sufficiently incompetent, other powerful actors (probably an adversary state) will infiltrate your project, steal your technology, and then sabotage you!^[1]

For whoever develops an ASI, there will be pressure to establish a singleton as soon as possible, so no one else can ever build an ASI or otherwise topple their regime.

Finally, race dynamics interact with AI alignment and control: there is extreme pressure to cut corners to speed up the development and deployment of powerful AI. At any given moment, deciding to cut corners just a little bit more is locally rational to each actor: the sacrifice probably won’t make the difference between catastrophe and success, and it gives a competitive advantage.

Presumably, at some point the perceived risk of catastrophe is so high that the least careful actor is not willing to cut any more corners, and an equilibrium is found. I have no reason to believe this equilibrium settles at a reasonable point! From a state’s perspective, the counterweight for the pressure to care about AI safety is the pressure to avoid total annihilation at the hands of an adversary.

—

If you take only one thing from this post, take this: any theory of change that falls to one of these competitive pressures is completely useless*.^[2]* The only way to avoid these pressures is if we could build common knowledge, at any given time, that no one is trying to develop ASI.

This is why I’m going for international coordination: while it’s very difficult, it would address the problem at the source. After that, if someone wants to build ASI, it should be done under an extremely extensive degree of supervision by all parties, such that the other theories of change on how to safely build ASI become much more feasible.

If you try to address any of the other problems, for example by trying to solve AI alignment and control, before having removed competitive pressures, you are swimming against a strong current and will be swept over the falls.

First filter: all-out war between nuclear superpowers

I think that hawkish writings about China usually fail to take their reasoning to the logical conclusion. For example, Leopold Aschenbrenner’s “Situational Awareness” and Dario Amodei’s essays, including “On DeepSeek and Export Controls” and some of “Machines of Loving Grace.”

People understand that the US and Chinese governments will wake up to the potential of ASI, and that when they do, absent strong international coordination (which Leopold and Dario assume is absent), the governments will be in an all-out race to who can build it first. The mistake Leopold, Dario and others make is to assume this is a restricted game, where most of what is happening is AI R&D and at most countries will engage in mutual espionage and sabotage.^[3]

If you take these views to their logical conclusion, you would see the ending of this story: all-out war between the US and China. When the superpowers try to sabotage each other’s ASI projects, they will not stop at grey-zone or covert sabotage. From a state’s perspective, if your adversary gets ASI, you are done. Your state will stop existing. You might as well have gotten all your major cities vaporized.

I am very confident that a superpower that knows it’s about to lose the race, or even considers a high risk of losing, will engage in unambiguous acts of war. The paper “Superintelligence Strategy” talks about possible kinetic strikes, but I think it will get much worse.

If states start building very hardened ASI projects, then stopping an opponent’s progress can be impossible without taking extreme measures that attempt to make the opponent’s country completely dysfunctional. For example:

• Systematically attacking basic infrastructure (like the electrical grid) throughout the opponent’s territory
• Sabotaging core functions of the opponent’s government, such as attempting or strongly supporting a coup
• Launching an invasion, either to physically stop the ASI projects or to consume all the opponent’s resources through war

If we get to this point, I don’t see any reason to be confident that the situation won’t escalate all the way to a full-blown nuclear war between superpowers.

I think it would be a fool’s errand to try to predict the exact reaction of the national security establishment of the losing superpower. It will depend too much on unpredictable and opaque factors, from the structure of the natsec apparatus to whether the people responsible happen to be in a bad mood at some specific, decisive moment.

But I think it’s important to note that there are strong mechanisms pushing in the direction of arbitrary escalation, and no strong mechanisms preventing it from doing so.

And if all-out war between nuclear powers doesn’t sound bad enough to you, remember this: war would be waged with much more advanced AI than we have today, and the war itself would further shape the incentives around the AI race.

Contra “stable multipolar scenarios”

Stable multipolar scenarios can happen in one of two ways: if AI’s efficacy at war has reached the limits of physics, or if AIs have a way to enforce a consensus (like in the good ending of “AI 2027”).

AI advantages compound, and if the gap is wide enough, one of the competitors (potentially a rogue AI) wins. It seems unlikely that AI’s ability to wage war will climb all the way to the limits of physics while the gap between the various actors never gets wide enough to conclude the conflict.

About AIs enforcing a consensus, roughly, I think this would require AI to already be vastly smarter and more competent than any human or existing human organization. Which makes this proposed “solution” kind of tautological: you still need to pass all the filters and build an ASI that you can trust.

As an example, in “AI 2027,” the two ASIs strike a deal by building a “consensus AI” that will forever enforce, to some degree, the preferences of both AIs. To do this, you’d need to have developed an extremely deep fundamental understanding of how to program AI, the kind of understanding that lets you write an AI as lines of code rather than a neural network.^[4]

Due to the competitive pressures I talk about in this post, the plan would not unfold this way. Much, much earlier than when you’d be able to achieve such a deep understanding of AI, you’d achieve an understanding just barely good enough to build ASI. Then you would build it and thus destroy all value in the world, unless you already figured out a way past all the filters in this post.

An alternative proposal is to have AIs strike deals that are enforced through mutual monitoring. By the time AIs can strike such deals autonomously, they are already fairly superhuman and / or significantly in charge of running the world, and we need to have passed the filters.

To be clear, I don’t necessarily think it’s a bad idea to have weaker AIs help us enforce monitoring-based international agreements. But this needs to be done before AIs get too strong, at which point humanity would have to do it, even if aided by weaker AIs.

(My paper “Modeling the geopolitics of AI development” talks about this filter in more detail, but the thinking is less refined since it was written a while ago.)

Second filter: misaligned AI that kills everyone

It is probably very hard to build an ASI that doesn’t end up killing every human being simply by running it.
The basic argument is that ASI would be so effective that any failure, even partial, would result in an ASI handling extreme amounts of power while not going out of its way to preserve human life and values.

ASI would kill us as a side effect of whatever it ends up doing, just like a human destroys an anthill without a second thought when it’s in the way of a construction project. The field of making sure that ASIs act in a desirable way is called “AI safety.”

The threat model of misaligned AI is the one that has already been explored the most, so I will assume that readers are at least passingly familiar with it and won’t try to convey the basic idea here. If you need an introduction, read the book “If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky and Nate Soares.^[5]

What I want to focus on here is how the pressure to cut corners I mentioned earlier makes it nearly impossible to solve alignment in time for when ASI will be developed. Think of the following competitive pressures:

• Pressure to cut corners on safety methods
• Pressure to deploy as fast as possible
• Pressure to give AI as much autonomy as possible
• Pressure to hand over existing decision loops to AI as quickly and thoroughly as possible

So what happens is, AI projects will develop and deploy AI that is as capable as possible given current capabilities techniques, while only being as safe as absolutely necessary to make them usable. The most important part here is: only as safe as absolutely necessary to make them usable. What does it mean? Well, the first instance of this pattern we’ll discuss is the commercial one.

Software engineers won’t use an AI that cheats to make the tests pass every time, but they’ll use an AI they can usually catch cheating, as long as the violations don’t fall through the cracks often enough for the engineer to get fired.

CEOs will not use AI employees that regularly take costly, irreversible actions to the point that the company loses a lot of money or it gets the CEO in trouble. But they will, for example, use AI that takes illegal actions as long as the company gets fined for less than the money it made, or the crime happens in a third-world country, etc.

So far it doesn’t sound like an extinction risk, but what is the “usability limit” when it comes to integrating AI in the military? What about tail risks, situations that are too rare and so haven’t yet appeared in the feedback loop of fixing AI bugs?

And most importantly, what happens when someone first gets to the capability level where they mostly hand over AI R&D to AIs themselves?

The AI will be just barely safe enough to profitably (not spotlessly!) do jobs that:

• Are roughly as complicated as AI R&D^[6]
• Have short-enough feedback loops that failures have already happened, such that AI companies already have bug tickets for these failures
• Have already addressed these bug tickets

Of course, you will not be able to get this guarantee for novel tasks, such as AI R&D itself. Probably, you won’t even be able to get it for tasks that already exist but are not common enough for you to test the AI thoroughly on them during the (very brief) allotted time. You have to hope that whatever safety you have transfers from this small, nonrepresentative set of tasks to the ones that matter.

Why Technical AI safety agendas do not address this problem

Technical AI safety agendas for addressing extinction risks usually focus on the “misaligned AI that kills everyone” filter, so I have to briefly address why, as a general rule, they don’t work. In fact, they make things worse.

This is because all alignment work is capabilities work.

Take RLHF (reinforcement learning from human feedback), for example.^[7]RLHF improved “alignment,”^[8]but it also improved capabilities a lot more: the AIs that we built after RLHF were more liable to do dangerous things than the ones we built before it existed. This is true even if you do your best to use RLHF to make the model safe.

To the degree that interpretability and scalable oversight work, I confidently predict that they will do exactly the same thing.

The underlying, fundamental reason for this is that capabilities are easier to formalize than safety. By this I mean that capabilities are easier to measure and easier to describe to other people, to AIs, and to code without loss of information.

Imagine that we get an interpretability breakthrough. You would have more readability into the internal algorithms of AIs, but those algorithms are very big and complicated: you wouldn’t automatically know which parts are helpful and which are harmful.

Some will be obviously harmful and removed right away. What then? Maybe you can do some manual searches for patterns you suspect exist? But humans are slow. You can get AIs to help you, but AIs are not (yet) smarter than you, and so they’d miss some stuff. Perhaps AIs already have some misaligned biases and so would sometimes actively hinder your efforts.

On other hand, capabilities, oh how they’d skyrocket. Better interpretability would yield more powerful methods to modify AIs: it would allow engineers and learning algorithms to modify AIs in more targeted, deliberate, and understandable ways than can be done today.

Since capabilities are more formalized, you can quickly train a large team of engineers to make use of the novel techniques. Perhaps you can cut engineers out of this loop entirely, integrating the novel technique as part of automated learning algorithms.

If you want to modify AI to improve a quality that is hard to measure, like safety, you need a human to stand there and opine about each candidate modification. Worse still, the human needs to have good taste about the property you are trying to improve.

To summarize: capabilities can improve at machine speeds, while safety will always be bottlenecked by humans. The only way to solve this dilemma would be to describe our safety desiderata to the same level that we have described our capabilities desiderata. That way, we could potentially automate AI safety, or at least reliably train a team of engineers to do it. Good luck doing that during an all-out race to ASI!

I encourage you to think about this issue yourself, especially if you are a researcher at a major AI company working on a technical AI safety agenda. Your work may end up boosting capabilities more than most of the people over on the capabilities teams.

Third filter: nightmare singletons

Ok, imagine that the alignment problem is on track to get solved, such that a human being (or group of human beings) could operate an ASI without killing themselves and everyone else as a side effect. You and the rest of your team, the “responsible actors” in a world composed mostly of irresponsible ones, have the lead in AI development. You will build ASI first and then establish an eternal utopia, right? No.

Here’s what really happens: the government takes over your project before you get to ASI, by default as a military project, possibly top secret. You are questioned just enough that they know how to make use of the project’s assets (like code, documentation, hardware, etc.), and then you are thrown out the door.

Or maybe a softer version of this happens, where your AI company still technically exists. However, your CEO does not retain effective control of the company, and you have military personnel looking over your shoulder as you work.

If your government is asleep at the wheel, a foreign government will take over your project, or at least steal all the progress you’ve made so far and then pour their resources into going faster than you. Or if all governments are asleep at the wheel, another company will take over your project, or perhaps a charming psychopath CEO will manipulate their way into a top leadership position at the company where you work.

What then?

Whoever controls an ASI can establish a singleton. A singleton is a “world order in which there is a single decision-making agency at the highest level, capable of exerting effective control over its domain, and permanently preventing both internal and external threats to its supremacy.”

—

Let me spell out, for people who haven’t thought about this subject before, how nightmarish this scenario can get.

An individual in control of an ASI could establish a dictatorship that controls the entire earth, possibly the entire universe.

They could monitor every corner of their domain 24/7 and assign a virtually infinite amount of intelligence to analyze all of this information.

They could compel everyone to install brain implants (or forcibly upload them, etc.) and have complete oversight and control over their thoughts, actions, and experiences.

Eventually, they could shape the whole world to their preference until every atom is exactly as they want it, and do it as easily as a child shapes playdough.

—

In AI safety, some people’s strategy is to give power and resources to “good” or “responsible” actors, such as their favorite AI company.^[9]The theory of change for this strategy is that the “responsible” actor is the first to build ASI and establishes a utopian (or at least “good”) singleton.

I think that it is an enormous mistake to trust any one person or company with this. If your strategy is to use ASI to establish a “good” singleton, I will fight to prevent you from succeeding because I don’t trust you. But regardless, I hope this post makes you see that this strategy will break horribly.

If you are part of an ASI project and this is your plan, know this: someone more powerful than you will take your toys away before you get to ASI. Then, they will use them to race to ASI without you.

What happens later is fundamentally unpredictable. The result does not have to be as bad as the nightmare scenario I painted earlier. But from where we’re standing, it could easily get really bad.

I think what happens if any individual or small group of people obtains absolute power over the universe is an extremely dystopian scenario, potentially worse than death depending on your values. I think the same is true for scenarios in which we just barely make enough progress on alignment that ASI doesn’t kill us all as a side effect. ASI may want a future for us, but it could be a future that we find abhorrent, and it would have absolute power over us.^[10]

Even in the best case scenario, where the ASI project is taken over by a government with a very robust democratic process, the situation would most likely be considered a national security emergency. Such emergencies are dealt with by the military (or more generally, the executive branch), which needs to be able to act quickly. As a result, it has weaker democratic oversight compared to other government branches.

What will this government do after having declared an emergency situation, armed with proto-ASI? Would you feel safe if you thought your government was bound to establish a singleton?

How common theories of change fail trivially

Any solution or theory that focuses entirely on technical AI safety fails trivially by not taking into account the two other filters. For example, some people think AI alignment will be easy to solve. I think this view is most likely mistaken on a very deep level. But even if it were correct, it would not address the other two problems at all.

Furthermore, I think that all technical AI safety projects will not be successful, not in a world where actors are able to unilaterally push the frontier of AI development toward ASI. This is due to the pressure to cut corners on safety, and because any technique will accelerate capabilities much more than it accelerates progress in AI safety.

The philosophy of “iterative deployment” will simply not apply in a world where the pace of deployment depends entirely on competitive pressure and is entirely causally disconnected from any consideration of what may be a “responsible” pace for AI development.

There are some who try to acquire personal power or influence so they can exert it “when the time comes.” This can mean attaining influence inside of AI companies or in governments. As I pointed to in the third filter, I think power within AI companies is meaningless.

And I think the people who try to acquire unilateral power within governments are deeply misguided. When push comes to shove, they will fail at gaining enough power to steer the actions of governments.

If the majority of the government does not understand the meaning of ASI, these people will not be able to make massively expensive and complex asks to leadership. For example: “slow down AI development to improve safety” or “pressure other major powers to enter a hefty trust but verify regimes capable of providing mutual assurances on AI development.” If these people try to push these asks without first building a broad support base (probably as broad as a decent voting bloc), then they will simply get purged.

Finally, there are people trying to get a major power to engage in a race to ASI, beat all their adversaries to it, and establish a singleton. I think these theories of change fail on literally all three filters:

• The world will likely be consumed by war before any actor can get to ASI.
• Even if we narrowly avoid all-out war, this theory of change leads to a race to the bottom on AI safety and to uncontrollable ASI that kills everyone.
• Even if the ASI ends up being somewhat controllable, no country on Earth currently has such institutional robustness that it would not produce a dystopia if it acquired ASI.

Conclusion

These were the main three challenges that I think stand between us and surviving ASI. Even if we pass all three, I don’t think things automatically go well.

I have more intuition pumps that I would like to publish in a future post. They are mostly about how, in scenarios with AI that is strong but not as strong as I’ve been implying ASI is, that:

• There is a strong tendency for power to concentrate and for the world to gravitate toward the three outcomes I’ve been describing.
• There is a tendency for human preferences and behavior to mutate beyond recognition, to a degree that we might think of such people as essentially “dead.”

The main way that I envision humanity passing these filters is with deep awareness of what ASI entails and with international coordination.

Deep awareness is necessary so the relevant parties understand what their interests are with respect to ASI. Chiefly, they need to understand that ASI can become powerful enough to destroy the world, and that it is indeed extremely hard to deploy an ASI without destroying the world.

Coordination, backed by mutual monitoring and deterrence, is necessary so the major parties can avoid a race to the bottom over who builds ASI first. Without it, they will end up developing and deploying ASI in the most irresponsible way possible, and thus destroy the world.

Both deep awareness and coordination are necessary so countries can eventually get to work to figure out how to go through this transition while avoiding the horrific failure modes I’ve described, and others yet.

At the moment, my best bet for achieving these goals is to work at ControlAI. If you’re interested in learning more about ControlAI, feel free to read our funding pitch, which also goes in detail about ControlAI’s theory of change. Alternatively, feel free to shoot me a message.

1. This includes things like stealing your weights and then sabotaging your ASI projects, but also trying to insert backdoors into your AI systems. ↩︎

2. And worse than useless if you consider that it absorbs funds and attention. ↩︎

3. Even when they acknowledge the possibility of war, it is treated as something that happens in the very endgame. Countries are not treated as being able to look ahead and strike preemptively. ↩︎

4. Even with such an understanding, code may not be the optimal way to build an AI, and you may choose to use neural networks or a new technique altogether. The point is if you wanted to write it in code, you could. ↩︎

5. Some people criticize Yudkowsky and Soares’ arguments for not engaging properly with the peculiarities of LLMs and claim that LLMs make alignment easier. I have it on my to-do list to write about why the shape of current AI systems doesn’t make me particularly optimistic about alignment. Unfortunately, at the moment I don’t know of a good post to convey this; the best one I can point you to is: “Current AIs seem pretty misaligned to me.” ↩︎

6. In fact, I think this is quite optimistic. AI companies are prioritizing AI R&D over anything else, so it will be one of the first (if not the first) task AIs will be able to perform at its level of complexity. There will not have been trial runs with similarly complex tasks. ↩︎

7. RLHF is the technique that enabled the creation of the first version of ChatGPT. ↩︎

8. Insofar as you could get LLMs to actually do the task you asked them to do, even when the task was not extremely simple and even if you weren’t an expert base-model prompter. ↩︎

9. This includes technical people who decide to work on capabilities at an AI company. ↩︎

10. The bad ending of “AI 2027” falls under this last category, and it was considered the most likely ending by the authors at the time of writing. ↩︎