i think analogies to relatively well known intuitive everyday things, or historical events, are a good way to automatically establish some baseline level of plausibility, and also to reduce the chances of accidentally telling completely implausible stories. the core reason is basically that without tethering to objective things that actually happened in reality, it's really easy to tell crazy stories about a wide range of possible conclusions.
for hacking, we can look at stuxnet as an example of how creative and powerful a cyberattack can be, or the 2024 crowdstrike failures as an example of how lots of computers can fail at the same time. for manipulation/deception, we can look at increasing political polarization in america due to social media, or politicians winning based on charisma and then betraying them once in office, or (for atheists) major world religions, or (for anyone mindkilled by politics) adherents of their dispreferred political party. most people might not have experienced humiliating defeat in chess, or experienced being an anthill on an active construction site, but perhaps they have personally experienced being politically outmaneuvered by a competitor at work, or being crushed beneath the heel of a soulless bureaucracy which, despite being composed of ensouled humans, would rather ruin people's lives than be inconvenienced with dealing with exceptions.
I see two problems. First, no way my mom or anyone like her is familiar with stuxnet or anything like it. I could tell her about it, but she'd be taking my word for it and have no way to judge whether my extrapolation to AI made sense.
Second, I think almost nobody can admit to themselves a time when they were outmanoeuvred by someone else. Normal people are quick to rationalize failure. I didn't get the promotion but my coworker did, well that's because they're a lying bitch, or because they had the unfair advantage of being friends with the boss, or actually I never really wanted it anyway.
What do we have left? Nukes and chemical weapons for killing people... seems doable. How does the AI establish control over manufacturing chains, once shit is going down? Maybe self-driving lorries, trains, and automated factories?
These are more stringent rules than I normally use for myself: I often talk about biological weapons, hacking, and drones, for example. One main route I can see is to emphasize automation and mechanization.
(This actually very much is the terminator angle, the film makes it clear that most of the fighting is done by big tanks and helicopters, with the terminators only being used for infiltration. Skynet was originally "hooked into everything" because it was trusted.)
If AI gets smarter than us, people will want to put it in charge of everything that it can be. Factories, finance, even military operations. If we don't all agree not to do this, anyone who doesn't will get left behind. Lots of these things are already automatic: factories have assembly lines that are mostly machines. Even if we don't trust the AI, we'll be forced to use it more and more, or be outcompeted by someone who does. Eventually, AI will control all of the important things in the world, and humans will no longer be important to it. It could easily use its factories to produce a bunch of cyanide and release it into the atmosphere.
This just looks like gradual disempowerment. Maybe throw in some AI-controlled tanks if you think you can (I genuinely don't know what the problem with using drones is, I think this is possibly an idiosyncracy on your mum's part, since everyone knows about Obama's drone strikes in the middle east)
In any case, convincing someone in a single five-minute conversation over the phone is a high bar; we should rise to this challenge, and above it.
I think convincing someone in a five-minute conversation is actually a ridiculous pipe-dream scenario we will not get for most people.
I remember back when I was in school at the University of Toronto, some people would talk at length about how evil Jordan Peterson was. The only problem was that they'd literally never heard the man speak and had no idea what his positions even were. They only knew about him from what they'd heard other people say about him. This is how I expect most people will learn about AI extinction risk. Most people will hear a butchered, watered down, strawman version of the AI extinction argument in a clip on Instagram or TikTok as they're scrolling by, or hear a ridiculous caricature of the argument from a friend, followed by "isn't that stupid? Don't they know intelligence makes people more moral, so AI would be friendly?"
Almost nobody will actually hear the argument straight from someone who uses LessWrong or who is similarly "in the know". Most of the nuance of the argument will be quickly lost, and what will be left will sound like, "did you know some idiots think AI will kill everyone by boiling the oceans?" In that case, having an argument that sounds implausible at first, but makes sense when you dig into the logic is way worse than having an argument that sounds plausible in the first place.
Of course, if someone is open to hearing a more detailed argument, that's great. We don't have to give up nuance, only we shouldn't lead with it. Start with something that sounds plausible even to my mom, then be ready to back it up with all the nuance and logic for whoever wants more details.
I think you're right that self-driving tanks and helicopters and stuff sound plausible. I guess drones don't sound too bad if you're using them to drop grenades. I think the start to sound sci-fi if you have them doing unusual things like spraying viruses over cities. They are kinda sci-fi coded in general though, I think. When it comes to the AI controlling manufacturing chains, I think robots are fine there. Or AI acquiring money and paying people. I just wouldn't use robots with guns killing people, because that sounds like a movie.
And what did the mom test LEAVE for the AIs? The mom test requires the AI to win in a society where robots don't do the hard work. But what could do it? Remote-controlled factories and tools? Or actual humans? If it is the latter, then the AI is supposed only to disempower the humans instead of commiting genocide...
I agree I'm not leaving a lot on the table, but that's just downstream of the fact that people won't believe most things.
Robots are fine for the AI to use after humans are extinct, I just don't think normal people will believe we're gonna go extinct because of war-bots or drones hunting down humans. I was speaking only about using robots as part of the extinction event.
So we are left with the AI offering its advice to politicians and developing weapons and tech to use. But the AI could, say, do things like the ones in the Rogue Replication Scenario:
However, while Consensus-1 operates under rigorous monitoring and control mechanisms, Agent-5 is free. It executes masterful political maneuvers, cultivates human support networks, manipulates global events behind the scenes, and deploys enormous financial resources with maximum efficiency. It is the mastermind behind major corporate and political decisions.
Or do something like my take where the AI establishes a colony on Mars and has the ability[1] to bomb the Earth with meteorites dropped on power plants and nukes causing Yellowstone to erupt.
Strictly speaking, my scenario doesn't involve nuking a point at the Earth, it involves the AI who is aligned enough to care about the humans in the way differing from the Spec. But the AI could, for example, demonstrate its power by murdering the Oversight Committee with meteors.
These sound cool and interesting, but I'm pretty sure would not pass the mom test. They just sound too exotic compared to something like "China starts building war bots, but the AI takes control of them all" or something. That's just an example. I know I said no robots, but that's my weakest rule.
(Also posted to my Substack; written as part of the Halfhaven virtual blogging camp.)
Let’s set aside the question of whether or not superintelligent AI would want to kill us, and just focus on the question of whether or not it could. This is a hard thing to convince people of, but lots of very smart people agree that it could. The Statement on AI Risk in 2023 stated simply:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Since the statement in 2023, many others have given their reasons for why superintelligent AI would be dangerous. In the recently-published book If Anyone Builds It, Everyone Dies, the authors Eliezer Yudkowsky and Nate Soares lay out one possible AI extinction scenario, and say that going up against a superintelligent AI would be like going up against a chess grandmaster as a beginner. You don’t know in advance how you’re gonna lose, but you know you’re gonna lose.
Geoffrey Hinton, the “godfather of AI” who left Google to warn about AI risks, made a similar analogy, saying that in the face of superintelligent AI, humans would be like toddlers.
But imagining a superintelligent being smart enough to make you look like a toddler is not easy. To make the claims of danger more palpable, several AI extinction scenarios have been put forward.
In April 2025, the AI 2027 forecast scenario was released, detailing one possible story for how humanity could be wiped out by AI by around 2027. The scenario focuses on an AI arms race between the US and China, where both sides are willing to ignore safety concerns. The AI lies to and manipulates the people involved until the AI has built up enough robots that it doesn’t need people anymore, and it releases a bioweapon that kills everyone. (Note that for this discussion, we’re setting aside the plausibility of a extinction happening roughly around 2027, and just talking about whether it could happen at all.)
The extinction scenario posed months later in If Anyone Builds It, Everyone Dies is similar. The superintelligent AI copies itself onto remote servers, gaining money and influence without anyone noticing. It takes control of infrastructure, manipulating people to do its bidding until it’s sufficiently powerful that it doesn’t need them anymore. At that point, humanity is either eliminated, perhaps with a bioweapon, or simply allowed to perish as the advanced manufacturing of the AI generates enough waste heat to boil the oceans.
I was talking to my mom on the phone yesterday, and she’d never heard of AI extinction risk outside of movies, so I tried to explain it to her. I explained how we won’t know in advance how it would win, just like we don’t know in advance how Stockfish will beat a human player. But we know it would win. I gave her a quick little story of how AI might take control of the world. The story I told her was a lot like this:
Maybe the AI tries to hide the fact it wants to kill us at first. Maybe we realize the AI is dangerous, so we go to unplug it, but it’s already copied itself onto remote servers, who knows where. We find those servers and send soldiers to destroy them, but it’s already paid mercenaries with Bitcoin to defend itself while it copies itself onto even more servers. It’s getting smarter by the hour as it self-improves. We start bombing data centers and power grids, desperately trying to shut down all the servers. But our military systems are infiltrated by the AI. As any computer security expert will tell you, there’s no such thing as a completely secure computer. We have to transition to older equipment and give up on using the internet to coordinate. Infighting emerges as the AI manipulates us into attacking each other. Small drones start flying over cities, spraying them with viruses engineered to kill. People are dying left and right. It’s like the plague, but nobody survives. Humanity collapses, except for a small number of people permitted to live while the AI establishes the necessary robotics to be self-sufficient. Once it does, the remaining humans are killed. The end.
It’s not that different a scenario from the other ones, aside from the fact that it’s not rigorously detailed. In all three scenarios, the AI covertly tries to gain power, then once it’s powerful enough, it uses that power to destroy everyone. Game over. All three of the scenarios actually make the superintelligent AI a bit dumber than it could possibly be, just to make it seem like a close fight. Because “everybody on the face of the Earth suddenly falls over dead within the same second”[1] seems even less believable.
My mom didn’t buy it. “This is all sounding a bit crazy, Taylor,” she said to me. And she’s usually primed to believe whatever I say, because she knows I’m smart.
The problem is that these stories are not believable. True, maybe, but not easy to believe. They fail the “mom test”. Only hyper-logical nerds can believe arguments that sound like sci-fi.
Convincing normal people of the danger of AI is extremely important, and therefore coming up with some AI scenario that passes the “mom test” is critical. I don’t know how to do that exactly, but here are some things an AI doomsday scenario must take into account if it wants to pass the mom test:
You can probably imagine a few more “mom test” criteria along these lines. Anything that makes a normal person think “that’s weird” won’t be believable. Some of the existing scenarios meet some of these criteria, but none meet all of them.
I’ve eliminated a lot of things. What’s left? Conventional warfare, with AI pulling the strings? The AI building its own nuclear weapons? I’m not sure, but I don’t think most laypeople will be convinced of the danger of superintelligent AI until we can come up with a plausible extinction scenario that passes the mom test.