Thoughts on the shifting x-risk sentiment:

1. AI x-risk is mainstream. That means that worrying about x-risk is decoupling from other things that it has historically correlated with, e.g. EA values, LW epistemics, “high-context”, SF culture, etc.
2. It’s more important than ever to have “arbitragers” like 80K’s videos or Rob Miles — making sense of the situation for the general public, translating ideas from the community. We need the full spectrum from Dwarkesh to Plzdontkillus, and probably even broader audience than that.
3. A while ago, Richard Ngo had a take like “over time, people thinking about AI will focus less and less on superintelligence and x-risk and distant galaxies, and instead focus on immediate benefits and harms of near-term models”. I was very sympathetic to this take, and it matched the history. My best guess is that the current sentiment wave suggests otherwise — people have jumped straight from small-scale harms to RSI->ASI->extinction, without passing through mid-scale harms like terrorism, or mass unemployment. Of course, things could switch once we see AI-enabled terrorism and mass unemployment.
4. People do not like safety lab employees. Not sure how to feel about this. I think “safety lab employees should quit” is very defendable, but people’s sentiment around this is based entirely on association and vibes, rather than “here are all the pros and cons of this particular role, my all-things-considered view is bla”. Their mental model doesn’t distinguish between pretraining vs model organisms, they don’t even know what this is. I think this is bad. It might even spill-over into anti-AI safety, in the same way that after the 2008 financial crash, the public despised both the banks and the financial regulators.
5. The media wave is still on-going. It’s worth taking a few days from your 9-5 to think about how you can ensure that it leads to good, persistent outcomes.
6. We need much better and frequent tracking of public sentiment. There should be an org which just tries to poll and talk to ordinary people constantly, and updates us on where they are at.
7. I think we missed the mark by tweeting all-things-considered p dooms. Eg instead of “I personally think it is >10% within the next decade”, Evan should’ve tweeted “reckless RSI would bla” or “under the current practices bla”.
8. People don’t like probabilities. This isn’t how they think natively. That sucks, but maybe we need to meet them where they are at, and talk in their native epistemic representations. And definitely don’t say “I’d bet on x-risk if only there was a financial instrument for this”. Read Hanson on how people feel about profit.
   1. EDIT: I was wrong. Subjective probabilities are great. See this shortform. They puts x-risk ignorers into a tough position where they must either: (1) Refuse to give probabilities, which is overtly evasive. (2) Give absurd probabilities, which differ from those given by experts in the field, e.g. 0.1%. (3) Give reasonable probabilities (e.g. 10%) which clearly demonstrate that the company is acting recklessly. The things that changed my mind were Habryka's comment to this thread, and watching New York Council asking spokespeople for the AI companies: “As you all four are under oath we want to hear how each of you quantify the risk imposed by AI in the worst-case catastrophic scenario.” Their evasive corporate babble responses sounded ridiculous and the council members did not appreciate it.
9. This is probably the least partisan of any current big issue. But things seem slightly too polarised left. The hold-outs are tech VCs / libertarians and Bluesky libs. I think we should focus on the tech VCs / libertarians.
10. If you’re prominent in EA/AIS, then the Eye of Sauron is upon you. Act with integrity. Be factual. Say what you believe and why you believe it.
11. We need to communicate better stories of risk. The best I’ve seen so far is this by Drew Sparks. I want these for different threat models. We might need to mention nanotech.
12. We need to be ready for mass demonstrations. They’re likely to happen within 12 months, whether we want them or not. I hope this is led by reasonable people who can build bridges.
13. Don’t waste energy responding to convoluted conspiracy theories. The people proposing them don’t care about your response. And it looks to outsiders like these theories are worth responding to. Your caveman brain loves to win silly internet arguments. But stay on message, present your view of the world, respond to the best criticism.
14. Keep pushing the HF/OAI agent swamp story. No theoretical augments about malign priors and instrumental convergence and orthogonality thesis. Reality has generously provided a case study for all of that. You should familiarise yourself with all the details of HF/OAI. Say “Here’s what happened. Unless we act, then soon there will be millions of AI agents, much smarter, and given control over the process of making smarter AIs. They could quickly become so powerful that they could overthrow governments, steal the majority of humanity’s infrastructure (like factories to make robots and more computers), and then hinder our ability to turn them off probably by killing vast numbers of humans”.