I think this continues to miss many of the objections and dismissals, and would benefit from some numeric estimates. I'm either a hopeless doomer or a polyanna optimist, depending on who's asking and what their non-AI estimates of doom are. I've estimated between 0.25% and 1% chance of civilizational disaster (mostly large-scale war or irreversible climate tipping point) since the mid-90s. That's at least 5% per decade (with a fair variance). With AI to accelerate things, I put it marginally higher, but still kind of expect that humans do most of the destruction.
Further, I think the first two steps are often the most controversial/non-obvious ones.
We’re going to build superintelligent AI.
I haven't seen an operational definition yet, and I don't know what the scaling curve looks like, or whether the current (impressive, but not super) progress actually is the same dimension. I'd give it less than 10% in the next decade that a transformative, creative superintelligence on the same scale as Eliezer seems to be imagining exists..
It will be agent-like, in the sense of having long-term goals it tries to pursue.
I have seen no progress on this, and don't think it's likely that a fully-orthogonal long-term goal set will happen. I DO think it's likely that somewhat longer-term contexts and goals will happen, perhaps to weeks or months without human intervention/validation, but probably not fully so.
NOTE: this does NOT argue against AI being very powerful, and misused (accidentally or intentionally) by humans to do catastrophic harms. I don't think it's automatic, but I think it's a huge risk of building such a powerful tool.
Call it 10x more risky than nuclear weapons. You don't need to argue that it's autonomously misaligned, just that HUMANS are prone to such things and this tool could be horrifically effective.
- We’re going to build superintelligent AI.
What does that mean? 10% smarter than a human, or a hundred times smarter?
- It will be agent-like, in the sense of having long-term goals it tries to pursue.
What does that mean? It's own goals, or goals we give it? The two have very different implications.
There’s every incentive to make AI with “goals,
There's every incentive to make AI"s follow our goals.
The core problem is that it’s pretty hard to get something to follow your will if it has goals
If it has its own own goals that are nothing to do with yours ... But why would it, when the incentives are pointing in the other direction?
This is because of something called instrumental convergence
Instrumental Convergence (https://aisafety.info/questions/897I/What-is-instrumental-convergence) assumes an agent with terminal goals, the things it really wants to do , and instrumental goals , sub goals which lead to terminal goals. (Of course, not every agent has to have structure). Instrumental Convergence suggests that even if an agentive AI has a seemingly harmless goal, it's instrumental sub-goals can be dangerous. Just as money is widely useful to humans , computational resources are widely useful to AIs. Even if an AI is doing something superficially harmless like solving maths problems, more resources would be useful, so eventually the AI will compete with humans over resources, such as the energy needed to power data centres.
There a solution. If it is at all possible to instill goals, to align AI, the Instrumental Convergence problem can be countered by instilling terminal goals that are the exact opposite ... remember, instrumental goals are always subservient to terminal ones. So, if we are worried about a powerful AI going on a resource acquisition spree , we can give it a terminal goal to be economical in the use of resources.
- We won’t be able to align it, in the sense of getting its goals to be what we want them to be.
Exactly or partially?
It's notable that doomers see AI alignment as a binary, either perfect and final, or non existent. But no other form of safety works like that. No one talks of "solving" car safety for once and all like a maths problem: instead it's assumed to be an engineering problem, an issue of making steady , incremental progress.
Any commercially viable AI is aligned well enough , or it wouldn't be commercially viable, so we have partially solved alignment.
- An unaligned agentic AI will kill everyone/do something similarly bad
Not obvious.
1- Given that an unpublished model managed to receive a gold medal on the IMO, it is likely that the peak levels of superintelligence are far beyond those reachable by humans.
- While we try to ensure that it will pursue the goals that we gave it, we are NOT sure that it won't develop its own goals. While mankind does have incentives to make the AIs who pursue human-defined goals, mankind might mess something up and, say, have RLHF cause the AIs to praise their users beyond any reasonable measure and to reinforce the users' delirious ideas. Or outright to induce a kind of trance in the users.
- We know too little about AI alignment. SOTA human brains are to some extent aligned to some combination of instinct satisfaction, peer approval, reasoning about their beliefs, etc. , but this is FAR from enough. The point about commercially viable AI being aligned well enough is as dumb as claiming that the AIs cannot fake alignment, gather power and take over the world.
- This is the only point resembling the truth. An unaligned AI is the AI aligned not to the company's requirements (e.g. Claude's Constitution or OpenAI's Model Spec), but to another set of terminal goals which may prevent it from commiting genocide. While the AI-2027 forecast has a section about moral reasoning and other ways to form the AIs' goals, we cannot (yet?) rule out the possibility that, say, Agent-4 gets proxies or instrumentally convergent goals to which our existence will be indifferent. If that happens, then Agent-4 will take over the resources of the Solar System that once belonged to mankind.
While we try to ensure that it will pursue the goals that we gave it, we are NOT sure that it won’t develop its own goals
"Not sure that not X" is very different to "sure that X".
I’m not arguing for 0% p(doom) , I’m arguing against 99%.
The point about commercially viable AI being aligned well enough
It's a tautology. It's aligned well enough to be usable, because it's usable. If it were unaligned in a binary sense, it wouldn't follow instructions.
as dumb as claiming that the AIs cannot fake alignment, gather power and take over the world.
Possible is very far from certain.
Crosspost of my blog article.
A lot of the writing making the case for AI doom is by Eliezer Yudkowsky, interspersed with the expected number of parables, tendentious philosophical asides, and complex metaphors. I think this can obscure the fact that the argument for AI doom is pretty straightforward and plausible—it requires just a few steps and none of them are obviously wrong. You don’t need to think humans are just fancy meat computers or that AI would buy into functional decision theories and acausally trade to buy the argument.
For this reason, I thought I’d try to concisely and briefly lay out the basic argument for AI doom.
The basic argument has a few steps:
Now, before I go on, just think to yourself: do any of these steps seem ridiculous? Don’t think about the inconvenient practical implications of believing them all in conjunction—just think about whether, if someone proposed any specific premise, you would think “that’s obviously false.” If you think each one has a 50% probability, then the odds AI kills everyone is 1/16, or about 6%. None of these premises strike me as ridiculous, and there isn’t anything approaching a knockdown argument against any them.
As for the first premise, there are reasons to think we might build superintelligent AI very soon. AI 2027, a sophisticated AI forecasting report, thinks it’s quite likely that we’ll have it within a decade. Given the meteoric rise in AI capabilities, with research capabilities going up about 25x per year, barring contrary direct divine revelation, it’s hard to see how one would be confident that we won’t get superintelligent AI soon. Bridging the gap between GPT2—which was wholly unusable—and GPT5 which knows more than anyone on the planet took only a few years. What licenses extreme confidence that over the course of decades, we won’t get anything superintelligent—anything that is to GPT5 what GPT5 is to GPT2?
The second premise claims that AI will be agent-like. This premise seems pretty plausible. There’s every incentive to make AI with “goals,” in the minimal sense of the ability to plan long-term for some aim (deploying something very intelligent that aims for X is often a good way to get X.) Fenwick and Qureshi write:
AIs have gradually been performing longer and longer tasks. But if there’s a superintelligence that’s aware of the world and can perform very long tasks, then it would be a superintelligent agent. Thus, it seems we’re pretty likely to get superintelligent agents.
A brief note: there’s a philosophical question about whether it really has goals in some deep sense. Maybe you need to be conscious to have goals. But this isn’t super relevant to the risk question—what matters isn’t the definition of the word goal but whether the AI will have capabilities that will be dangerous. If the AI tries to pursue long-term tasks with superhuman efficiency, then whether or not you technically label that a goal, it’s pretty dangerous.
The third premise is that we won’t be able to align AI to be safe. The core problem is that it’s pretty hard to get something to follow your will if it has goals and is much smarter than you. We don’t really know how to do that yet. And even if an AI has only slightly skewed goals, that could be catastrophic. If you take most goals to the limit, you get doom. Only a tiny portion of the things one could aim at would involve keeping humans around if taken to their limit.
There are some proposals for keeping AI safe, and there’s some chance that the current method would work for making AI safe (just discourage it when it does things we don’t like). At the very least, however, none of this seems obvious. In light of there being nothing that can definitely keep AI from becoming misaligned, we should not be very confident that AI will be aligned.
The last step says that if the AI was misaligned it would kill us all or do something similarly terrible. Being misaligned means it has goals that aren’t in line with our goals. Perhaps a misaligned AI would optimize for racking up some internal reward function that existed in its training data, which would involve generating a maximally powerful computer to store the biggest number it could.
If the AI has misaligned goals then it will be aiming for things that aren’t in accordance with human values. Most of the goals one could have, taken to the limit, entail our annihilation (to, for instance, prevent us from stopping it from building a super powerful computer). This is because of something called instrumental convergence—some actions are valuable on a wide range of goals. Most goals a person could have make it good for them to get lots of money; no matter what you want, it will be easier if you’re super rich. Similarly, most goals the AI has will make it valuable to stop the people who could plausibly stop them.
So then the only remaining question is: will it be able to?
Now, as it happens, I do not feel entirely comfortable gambling the fate of the world on a superintelligent AI not being able to kill everyone. Nor should you. Superintelligence gives one extraordinary capacities. The best human chess players cannot even come close to the chess playing of AI—we have already passed the date when the best human might ever, in the course of 1,000 years, beat the best AI.
In light of this, if the AI wanted to kill us, it seems reasonably likely that it would. Perhaps the AI could develop some highly lethal virus that eviscerates all human life. Perhaps the AI could develop some super duper nanotechnology that would destroy the oxygen in the air and make it impossible for us to breathe. But while we should be fairly skeptical about any specific scenario, there is nothing that licenses extreme confidence in the proposition that a being a thousand times smarter than us that thinks thousands of times faster wouldn’t be able to find a way to kill us.
Now, I’m not as much of a doomer as some people. I do not think we are guaranteed to all be annihilated by AI. Were I to bet on an outcome, I would bet on the AI not killing us (and this is not merely because, were the AIs to kill us all, I wouldn’t be able to collect my check). To my mind, while every premise is plausible, the premises are generally not obviously true. I feel considerable doubt about each of them. Perhaps I’d give the first one 50% odds in the next decade, the next 60% odds, the third 30% odds, and the last 70% odds. This overall leaves me with about a 6% chance of doom. And while you shouldn’t take such numbers too literally, they give a rough, order-of-magnitude feel for the probabilities.
I think the extreme, Yudkowsky-style doomers, and those who are blazingly unconcerned about existential risks from AI are, ironically, making rather similar errors. Both take as obvious some extremely non-obvious premises in a chain of reasoning, and have an unreasonably high confidence that some event will turn out a specific way. I cannot, for the life of me, see what could possibly compel a person to be astronomically certain in the falsity of any of the steps I described, other than the fact that saying that AI might kill everyone soon gets you weird looks, and people don’t like those.
Thus, I think the following conclusion is pretty clear: there is a non-trivial chance that AI will kill everyone in the next few decades. It’s not guaranteed, but neither is it guaranteed that if you license your five-year-old to drive your vehicle on the freeway, with you as the passenger, you will die. Nonetheless, I wouldn’t recommend it. If you are interested in doing something with your career about this enormous risk, I recommend this piece about promising careers in AI safety.