If you pressed them about how AIs are actually created or how that specific creation process could cause AIs to be misaligned, they wouldn’t be able to tell you much.
I don't think people knew much about how nuclear worked and that didn't stop a movement from stopping its development.
Hi exmateriae,
I think that people are only going to support a movement if they feel they understand it enough.
It seems like nuclear is pretty simple, and people seem to think they know enough to have had movements around it.
Additionally, it seems like people often do understand enough about nuclear. If you asked a random person on the street about how nuclear is created and how it could go wrong, they can give you a pretty clear and reasonable answer. People understand that nuclear relies on radioactive materials to produce chain reactions. It can go wrong if these chain reactions go out of control on accident or if they are intentionally used in war. Their explanations are even good enough to convince others to join their movement.
On the other hand, AI is kind of complex, and I think people might feel like they need to know more to feel motivated enough to participate in a movement about it.
If you walked up to a random person on the street who had read IABIED, I think they probably couldn't answer these questions in a way that is satisfactory enough to them to participate in a movement or to convince others to join their cause.
AI is less understood than nuclear, I agree.
If you walked up to a random person on the street who had read IABIED, I think they probably couldn't answer these questions in a way that is satisfactory enough to them to participate in a movement or to convince others to join their cause.
I agree they wouldn't be able to answer.
On the other hand, AI is kind of complex, and I think people might feel like they need to know more to feel motivated enough to participate in a movement about it.
I disagree with the above, there are already citizens up in arms against AI. For instance I know many friends who post stories about the water lost to AI (yeah, I know...) or how everything is hallucinated (their last model used was 4o on the free plan), which is a very recurrent claim in my old field (law) where this is particularly important.
I think you overestimate the level of understanding of the general population of technology. (nuclear in this case but most actually)
if they feel they understand it enough
Exactly, they feel they understand it but have actually no idea.
In France, we're arguably the country with the most citizen exposure to nuclear power. Until not that long ago, most people thought the water vapor coming out of the plants was pollution. If you told most people that basically a nuclear power plant is a huge steam machine, they would be flabbergasted.
People understand that nuclear relies on radioactive materials to produce chain reactions.
In a street poll of 1000 almost representative people, I would bet less than 40% of the french population would be able to say this. I have had to defend nuclear power against arguments you would barely believe the stupidity of.
In 2019(!!) a poll was made and 10% thought that oil and gaz were contributing less co2 and 11% for coal. 70% believed that nuclear power was contributing to the emission of greenhouse gases and although they're technically right, it's painfully obvious in the poll that they believe it's emitting orders of magnitude more than they actually do. This 2017 one is even more shocking.
Most people will talk to you about radioactive waste (we are burying them in a special underground reserve) as if it was a danger much greater than the consequences of oil, gaz and coal.
In Belgium, a poll was made four years ago (they share plants with us and have some of their own) and there was still 13% who thought water vapor was radioactive gaz, 9% thought it was CO2 and 20% didn't know.
Anyway, sorry for going a bit into a rabbit hole but my point is that most people feel they understand enough most technologies when they actually don't and they're ready to have movements against those same technologies. Which is why I don't think that specific point was an issue to raise a movement to pause AI : the fact that people don't actually understand AI is imo even better to have people rally with you, you can tell each groups different stuff. (yeah that's terrible but that's an important part of politics)
Thanks for the thoughtful reply!
I think I'm much less hopeful about resistance to AI than you are. It seems like there's just a general trend of new technologies being developed and deployed so quickly that society doesn't have to ability to develop coherent thoughts about them and to produce meaningful regulation for them before they've already produced great harm. For instance, social media seems to be quite harmful to teens and yet society still hasn't really mobilized to reduce social media use among teenagers, despite social media having been around for more than a decade.
It seems to me like AI x-risk is just a bit too far out of the Overton window for people to really take it seriously right now, and it seems like people aren't organizing quickly enough to respond to other concerns about AI. I'm probably underestimating how seriously people take IABIED since I live in a highly educated area, where people really expect you to defend your beliefs if they're controversial.
Crosspost of my blog article.
Over the past five years, we’ve seen extraordinary advancements in AI capabilities, with LLMs going from producing nonsensical text in 2021 to becoming people’s therapists and automating complex tasks in 2025. Given such advancement, it’s only natural to wonder what further advancement in AI could mean for society. If this technology’s intelligence continues to scale at the rate it has been, it seems more likely than not that we’ll see the creation of the first truly godlike technology, a technology capable of predicting the future like an oracle and of ushering in an industrial revolution like we’ve never seen before. If such a technology were made, it could usher in an everlasting prosperity for mankind or it could enable a small set of the rich and powerful to have absolute control over humanity’s future. Even worse, if we were unable to align such a technology with our values, it could seek out goals different from our own and try to kill us in the process of trying to achieve them.
And, yet, despite the possibility of this technology radically transforming the world, most discourse around AI is surprisingly shallow. Most pundits talk about the risk of job loss from AI or the most recent controversy centering around an AI company’s CEO rather than what this technology would mean for humanity if we were truly able to advance it.
This is why, when I heard that Eliezer Yudkowsky and Nate Soares’ book If Anyone Builds It, Everyone Dies was going to come out, I was really excited. Given that Yudkowsky is the founder of AI safety and has been working in the field for over twenty years, I expected that he’d be able to write a foundation text for the public’s discourse on AI safety. I thought, given the excitement of the moment and the strength of Yudkowsky’s arguments, that this book could create a major shift in the Overton window. I even thought that, given Yudkowsky and Soares’ experience, this book would describe in great detail how modern AI systems work, why advanced versions of these systems could pose a risk to humanity, and why current attempts at AI safety are likely to fail. I was wrong.
Instead of reading a foundational text on AI safety, I read a poorly written and vague book with a completely abstract argument about how smarter than human intelligence could kill us all. If I had to explain every reason I thought this was a bad book, we’d be here all day so instead I’ll just offer three criticisms of it:
1. The Book Doesn’t Argue Its Thesis
In the introduction to the book, the authors clearly bold an entire paragraph so as to demarcate their thesis—“If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything like the present understanding of AI, then everyone, everywhere on Earth, will die.”
Given such a thesis, you would expect that the authors would do the following:
Instead, the authors do the following:
Considering what the authors actually wrote, their thesis should have been, “If an artificial intelligence system is ever made that is orders of magnitude better than humans across all domains, it will have preferences that are seriously misaligned with human values, which will cause it to kill everyone. Also, for vague reasons, the modern field of AI safety won’t be able to handle this problem.”
Notably, this thesis is much weaker and much different than the thesis that they actually chose.
2. The Book Doesn’t Make A Good Foundation For A Movement
Considering that the authors are trying to get 100,000 people to rally in Washington DC to call for “an international treaty to ban the development of Artificial Superintelligence,” it’s shocking how little effort they put into explaining how AI systems actually work, what people are currently doing to make them safe, or even addressing basic counter arguments to their thesis.
If you asked someone what they learned about AI from this book, they would tell you that AIs are made of trillions of parameters, that AIs are black boxes, and that AIs are “grown not crafted.” If you pressed them about how AIs are actually created or how that specific creation process could cause AIs to be misaligned, they wouldn’t be able to tell you much.
And, despite being over 250 pages long, they barely even discuss what others in the field of AI safety are trying to do. For instance, after devoting an entire chapter to examples of CEOs not really taking AI safety seriously, they only share one example of how people are trying to make AI systems safe.
Lastly, the authors are so convinced that their argument is true that they barely attempt to address any counterarguments to it such as:
3. The Crux of Their Argument Is Barely Justified
Lastly, the core crux of their argument, that AI systems will be seriously mis-aligned with human values no matter how they are trained, is barely justified.
In their chapter “You Don’t Get What You Train For,” they make the argument that, similar to how evolution has caused organisms to have bizarre preferences, the training process for AI systems will cause them to have bizarre preferences too. They mention, for instance, that humans developed a taste for sugar in their ancestral environment, but, now, humans like ice cream even though ice cream wasn’t in their ancestral environment. They argue that this pattern will extend to AI systems too, such that no matter what you train them to prefer, they will ultimately prefer something much more alien and bizarre.
To extend analogy about evolution to AI systems, they write,
They justify this argument with a few vague examples of how this misalignment could happen and then re-state their argument, “The preferences that wind up in a mature AI are complicated, practically impossible to predict, and vanishingly unlikely to be aligned with our own, no matter how it was trained.”
For this to be the central crux of their argument, it seems like they should have given it a whole lot more justification, such as, for instance, examples of how this kind of misalignment has already occurred. Beyond the fact that we’re capable of simulating the evolution of lots of preferences, their argument isn’t even intuitively true to me. If we’re training something to do something, it seems far more natural to me to assume that it will have a preference to do that thing rather than to do something vastly different and significantly more harmful.
Conclusion
I was really hoping for this book to usher in positive change for how people talk about the existential risks of AI, but instead I was sorely disappointed. If you want to see a more clear-headed explanation about why we should be concerned about AI, I’d recommend checking out 80,000 Hours’ article “Risks from power-seeking AI systems.”