A few claims from the post (made at varying levels of explicitness) are:
1 . Often people are themselves motivated by concern X (ex: "the race to superintelligence is reckless and highly dangerous") and decide to talk about concern Y instead (ex: "AI-enabled biorisks"), perhaps because they think it is more palatable.
2 . Focusing on the "palatable" concerns is a pretty grave mistake.
2a. The claims Y are often not in fact more palatable; people are often pretty willing to talk about the concerns that actually motivate you.
2b. When people try talking about the concerns that actually motivate them while loudly signalling that they think their ideas are shameful and weird, this is not a terribly good test of claim (2a).
2c. Talking about claims other than the key ones comes at a steep opportunity cost.
2d. Talking about claims other than the key ones risks confusing people who are trying to make sense of the situation.
2e. Talking about claims other than the key ones risks making enemies of allies (when those would-be allies agree about the high-stakes issues and disagree about how to treat the mild stuff).
2f. Talking about claims other than the key ones triggers people's bullshit detectors.
3 . Nate suspects that many people are confusing "I'd be uncomfortable saying something radically different from the social consensus" with "if I said something radically different from the social consensus then it would go over poorly", and that this conflation is hindering their ability to update on the evidence.
3a. Nate is hopeful that evidence of many people's receptiveness to key concerns will help address this failure.
3b. Nate suspects that various tropes and mental stances associated with the word "couarge" are perhaps a remedy to this particular error, and hopes that advice like "speak with the courage of your convictions" is helpful for remembering the evidence in (3a) and overcoming the error of (3).
I think the people I know who worked on SB-1047 are totally happy to say "it's ridiculous that these companies don't have any of the types of constraints that might help mitigate extreme risks from their work" without wavering
I don't think this is in much tension with my model.
For one thing, that whole sentence has a bunch of the property I'd call "cowardice". "Risks" is how one describes tail possibilities; if one believes that a car is hurtling towards a cliff-edge, it's a bit cowardly to say "I think we should perhaps talk about gravity risks" rather than "STOP". And the clause "help mitigate extreme risks from their work" lets the speaker pretend the risks are tiny on Monday and large on Tuesday; it doesn't extend the speaker's own neck.
For another thing, willingness to say that sort of sentence when someone else brings up the risks (or to say that sort of sentence in private) is very different from putting the property I call "courage" into the draft legislation itself.
I observe that SB-1047 itself doesn't say anything about a big looming extinction threat that requires narrowly targeted legislation. It maybe gives the faintest of allusions to it, and treads no closer than that. The bill lacks the courage of the conviction "AI is on track to ruin everything." Perhaps you believe this simply reflects the will of Scott Weiner. (And for the record: I think it's commendable that Senator Weiner put forth a bill that was also trying to get a handle on sub-extinction threats, though it's not what I would have done.) But my guess is that the bill would be written very differently if the authors believed that the whole world knew how insane and reckless the race to superintelligence is. And "write as you would if your ideas were already in the Overton window" is not exactly what I mean by "have the courage of your convictions", but it's close.
(This is also roughly my answer to the protest "a lot of the people in D.C. really do care about AI-enabled biorisk a bunch!". If the whole world was like "this race to superintelligence is insane and suicidal; let's start addressing that", would the same people be saying "well our first priority should be AI-enabled biorisk; we can get to stopping the suicide race later"? Because my bet is that they're implicitly focusing on issues that they think will fly, and I think that this "focus on stuff you think will fly" calculation is gravely erroneous and harmful.)
As for how the DC anecdote relates: it gives an example of people committing error (1), and it provides fairly direct evidence for claims (2a) and (2c). (It also provided evidence for (3a) and (3b), in that the people at the dinner all expressed surprise to me post-facto, and conceptualized this pretty well in terms of 'courage', and have been much more Nate!courageous at future meetings I've attended, to what seem to me like good effects. Though I didn't spell those bits out in the original post.)
I agree that one could see this evidence and say "well it only shows that courage works for that exact argument in that exact time period" (as is mentioned in a footnote, and as is a running theme throughout the post). Various other parts of the post provide evidence for other claims (e.g. the Vance, Cruz, and Sacks references provide evidence for (2d), (2e), and (2f)). I don't expect this post to be wholly persuasive, and indeed a variant of it has been sitting in my drafts folder for years. I'm putting it out now in part (because I am trying to post more in the lead-up to the book and in part) because folks have started saying things like "this is slightly breaking my model of where things are in the overton window", which causes me to think that maybe the evidence has finally piled up high enough that people can start to internalize hypothesis (3), even despite how bad and wrong it might feel for them to (e.g.) draft legislation in accordance with beliefs of theirs that radically disagree with perceived social consensus.
I think more people should say what they actually believe about AI dangers, loudly and often. Even (and perhaps especially) if you work in AI policy.
I’ve been beating this drum for a few years now. I have a whole spiel about how your conversation-partner will react very differently if you share your concerns while feeling ashamed about them versus if you share your concerns while remembering how straightforward and sensible and widely supported the key elements are, because humans are very good at picking up on your social cues. If you act as if it’s shameful to believe AI will kill us all, people are more prone to treat you that way. If you act as if it’s an obvious serious threat, they’re more likely to take it seriously too.
I have another whole spiel about how it’s possible to speak on these issues with a voice of authority. Nobel laureates and lab heads and the most cited researchers in the field are saying there’s a real issue here. If someone is dismissive, you can be like “What do you think you know that the Nobel laureates and the lab heads and the most cited researchers don’t? Where do you get your confidence?” You don’t need to talk like it’s a fringe concern, because it isn’t.
And in the last year or so I’ve started collecting anecdotes such as the time I had dinner with an elected official, and I encouraged other people at the dinner to really speak their minds, and the most courage they could muster was statements like, “Well, perhaps future AIs could help Iranians figure out how to build nuclear material.” To which the elected official replied, “My worries are much bigger than that; I worry about recursive self-improvement and superintelligence wiping us completely off the map, and I think it could start in as little as three years.”
These spiels haven’t always moved people. I’m regularly told that I’m just an idealistic rationalist who’s enamored by the virtue of truth, and who’s trying to worm out of a taboo tradeoff between honesty and effectiveness. I’m sometimes told that the elected officials are probably just trying to say what they know their supporters want to hear. And I am not at liberty to share all the details from those sorts of dinners, which makes it a little harder to share my evidence.
But I am at liberty to share the praise we've received for my forthcoming book. Eliezer and I do not mince words in the book, and the responses we got from readers were much more positive than I was expecting, even given all my spiels.
You might wonder how filtered this evidence is. It’s filtered in some ways and not in others.
We cold-emailed a bunch of famous people (like Obama and Oprah), and got a low response rate. Separately, there’s a whole tier of media personalities that said they don’t really do book endorsements; many of those instead invited us on their shows sometime around book launch. Most people who work in AI declined to comment. Most elected officials declined to comment (even while some gave private praise, for whatever that’s worth).
But among national security professionals, I think we only approached seven of them. Five of them gave strong praise, one of them (Shanahan) gave a qualified statement, and the seventh said they didn’t have time (which might have been a polite expression of disinterest). Which is a much stronger showing than I was expecting, from the national security community.
We also had a high response rate among people like Ben Bernanke and George Church and Stephen Fry. These are people that we had some very tangential connection to — like “one of my friends is childhood friends with the son of a well-connected economist.” Among those sorts of connections, almost everyone had a reaction somewhere between “yep, this sure is an important issue that people should be discussing” and “holy shit”, and most offered us a solid endorsement. (In fact, I don’t recall any such wacky plans that didn’t pan out, but there were probably a few I’m just not thinking of. We tried to keep a table of all our attempts but it quickly devolved into chaos.)
I think this is pretty solid evidence for “people are ready to hear about AI danger, if you speak bluntly and with courage.”[1] Indeed, I’ve been surprised and heartened by the response to the book.
I think loads of people in this community are being far too cowardly in their communication, especially in the policy world.
I’m friends with some of the folk who helped draft SB 1047, so I’ll pick on them a bit.
SB 1047 was an ill-fated California AI bill that more-or-less required AI companies to file annual safety reports. When it was being drafted, I advised that it should be much more explicit about how AI poses an extinction threat, and about the need for regulations with teeth to avert that danger. The very first drafts of SB 1047 had the faintest of teeth, and then it was quickly defanged (by the legislature) before being passed and ultimately vetoed.
One of my friends who helped author the bill took this sequence of events as evidence that my advice was dead wrong. According to them, the bill was just slightly too controversial. Perhaps if it had been just a little more watered down then maybe it would have passed.
I took the evidence differently.
I noted statements like Senator Ted Cruz saying that regulations would “set the U.S. behind China in the race to lead AI innovation. [...] We should be doing everything possible to unleash competition with China, not putting up artificial roadblocks.” I noted J.D. Vance saying that regulations would “entrench the tech incumbents that we actually have, and make it actually harder for new entrants to create the innovation that’s going to power the next generation of American growth.”
On my view, these were evidence that the message wasn’t working. Politicians were not understanding the bill as being about extinction threats; they were understanding the bill as being about regulatory capture of a normal budding technology industry.
...Which makes sense. If people really believed that everyone was gonna die from this stuff, why would they be putting forth a bill that asks for annual reporting requirements? Why, that’d practically be fishy. People can often tell when you’re being fishy.
It’s been a tricky disagreement to resolve. My friend and I each saw a way that the evidenced supported our existing position, which means it didn't actually discriminate between our views.[2]
But, this friend who helped draft SB 1047? I’ve been soliciting their advice about which of the book endorsements would be more or less impressive to folks in D.C. And as they’ve seen the endorsements coming in, they said that all these endorsements were “slightly breaking [their] model of where things are in the overton window.” So perhaps we’re finally starting to see some clearer evidence that the courageous strategy actually works, in real life.[3]
I think many people who understand the dangers of AI are being too cowardly in their communications.[4] Especially among people in D.C. I think that such communication is useless at best (because it doesn’t focus attention on the real problems) and harmful at worst (because it smells fishy). And I think it’s probably been harmful to date.
If you need a dose of courage, maybe go back and look at the advance praise for If Anyone Builds It, Everyone Dies again. Recall that the praise came from many people we had no prior relationship with, including some people who were initially quite skeptical. Those blurbs don’t feel to me like they’re typical, run-of-the-mill book endorsements. Those blurbs feel to me like unusual evidence that the world is ready to talk about this problem openly.
Those blurbs feel to me like early signs that people with a wide variety of backgrounds and world-views are pretty likely to adopt a sensible viewpoint upon hearing real arguments made with conviction.
The job isn’t done yet. Society being ready to have the conversation doesn’t mean that society is having the conversation. We’re still somehow in a world where I occasionally have dinner with elected officials who worry about recursive self-improvement leading to superintelligence inside of three years — and yet none of the congress members who privately praise our book are willing to endorse it publicly yet.
There’s work to be done. And now is perhaps the right time to do it.
This possible window of opportunity is why Eliezer and I wrote the book in the first place. Once I started noticing that the message was really working, e.g. in private meetings with elected officials in D.C., I came back and told Eliezer that I thought the time was ripe. Perhaps a clear, convincing, and successful book (with an associated media tour) will do a lot to thrust this conversation into the mainstream. And then maybe humanity will decide to do something else, and maybe we won't all die.
So, at the risk of pandering: If you see any of the hope here that I see, consider helping us make this book cause a big splash.
The time is ripe. Pre-orders can help our book debut on the best-seller lists, and being on the best-seller list will influence whether our book, and this conversation, catches national attention.
Last month we were hitting Amazon best-seller lists in our categories despite still being in pre-order mode (which I’m told is rare). I’m heartened by this support. If we can keep up the pace of the last month for the next two and a half months, we’ll be in good shape. If we can keep up the pace and then get an extra burst of support right as the book launches, we’ll be in great shape. I think that’s a possibility, with support from this community.
We’re doing everything we can on our end, and it’s going well. If you want to coordinate about breaking open the Overton window, now is quite plausibly the best time for it.
If we do our job well here, there’s a real chance that politicians and journalists and your neighbor are taking extinction-level dangers from AI seriously by the end of the year.
And regardless of whether you think our book is worth throwing your weight behind, I hope you consider that you don’t need to hide your beliefs about AI danger, nor be ashamed of them. People respond well when you speak to them with courage.
The "courage" property I'm talking advocating here is about naming what you think is the real problem plainly, not about being impolite. This notion of courage is completely compatible with saying "you don't have to agree with me, Mr. Senator, but my best understanding of the evidence is [belief]. If ever you're interested in discussing the reasons in detail, I'd be happy to. And until then, we can work together in areas where our interests overlap." The concept is about naming what you think are the key issues, straightforwardly, in plain language. There are loads of ways to do so politely. See also this comment.
That wasn't the only case where the evidence was tricky to read. In January, Senator Cruz had Elon Musk on his podcast and asked “How real is the prospect of killer robots annihilating humanity?” in a context where the senator was overall a bit dismissive. Some of my friends read this as a bad sign, on account of the dismissiveness. I read it it as a good sign, because he was at least starting to think about the key dangers.
And shortly after J.D. Vance read AI 2027, he was asked “Do you think that the U.S. government is capable in a scenario — not like the ultimate Skynet scenario — but just a scenario where A.I. seems to be getting out of control in some way, of taking a pause?” and he answered:
Some of my friends read this as a bad sign, because he seemed intent on a suicide race. I read it as a good sign, because — again — he was engaging with the real issues at hand, and he wasn't treating AI as just another industry at the mercy of over-eager regulators.
My friend caveats that they're still not sure SB 1047 should have been more courageous. All we have actually observed is courage working in 2025, which does not necessarily imply it would have worked in 2023.
By "cowardice" here I mean the content, not the tone or demeanor. I acknowledge that perceived arrogance and overconfidence can annoy people in communication, and can cause backlash. For more on what I mean by courageous vs cowardly content, see this comment. I also spell out the argument more explicitly in this thread.