Quick Takes

Has there been any significant work on Self-Other Overlap since mid-2024? I recall Yudkowsky saying it was not a totally stupid idea. Seems to me like it's been wildly underinvested in by the safety community?

Harari is kind of disliked in these circles but I think this random interview is often way better AI risk communication than most of what the traditional AI risk communicators do - recommend to watch

I think people adjacent to the field don't appreciate how small AI safety is. For example, sometimes people complain about how AI safety threat models don't seem as well-developed or rigorous as climate change models. I think some of this is a skill issue on the part of AI safety researchers and advocates, which I complained about before. However, the bigger issue is that there is just way more money and people going into addressing climate change than on AI safety. There are tens of millions of people working on reducing climate change, and the core of people working on threat models ("climate scientists") number in the tens of thousands. (Yes, I was surprised by how big the numbers were too). (Most people work on stuff like clean energy). In comparison, there are low single-digit thousands (anything from 1,000 to 4,000 is defensible imo) of people working on AI safety, and collectively maybe low dozens of people working on figuring out threat models. (Most people work on ai safety engineering, some on policy). So we're 4 whole orders of magnitude behind climate change in people working on addressing the issue, and maybe 3 orders of magnitude behind on threat modeling. In terms of money, yearly spending on AI safety is in the high hundreds of millions to maybe 2 billion. In contrast, annual spending on reducing climate change (including clean energy) is in *trillions.* So we're 3-3.5 orders of magnitude behind on spending, and 4 ooms behind on people. The AI safety inputs picture looks a bit less bleak than this immediate comparison might suggest. For example, I think our average talent is somewhat better (higher human capital on both observed and unobserved metrics). There's also more latent interest/willingness-to-spend on AI safety than the "high hundreds of millions to low billions" current spending numbers might suggest. In contrast, climate change spending can't be scaled much higher (since you can't spend that much more than "trillions").

I donated to Pause AI UK today. Why this org in particular? 1) Right now, AI advocacy is clearly the most important thing to do. 2) I've been to some of their protest and one more event, so I know they are doing good work. I think that donating "close to home" in the sense of point 2, is underrated in EA.

Full text of Anthropic's draft proposal for a federal AI bill. Admittedly it is a draft. Perhaps things will change in the future. But it does not strike me as the kind of document that comes from a culture that puts safety above winning the race. Commentary on the bill from Nathan Calvin on Twitter: [...]

We may soon have a need to de-duplicate safety research. I recently spoke with the CEO of an AI safety nonprofit, and he said that a large part of his work is coordinating with others orgs to ensure he isn't accidentally working on the same project. Oftentimes you don't hear about work until it is finished, so it is very easy for work to be partially duplicated when lots of people are working in a similar area. I expect we will see am amplified version of this problem with autonomous researchers. Imagine a Claude in every lab, all looking at the same public research, and all deciding on the same "most promising" research direction. We will need some coordination mechanism once autonomous AI researchers are picking their own research directions.

Writing about AI/AI safety doesn't hurt your popularity. In fact, it probably increases it. Sometimes I talk to people interested in writing/podcasting/video-ing about AI safety, and they think that it'd be good for them to write non-AI things to "attract readers" or "be popular." I think this is silly, especially for people who know a lot about AI and AI safety. I think this is silly for many reasons but the biggest reason is empirical: My best guess is that for most people capable of doing both, your AI safety writing probably outperforms your non-AI writings. I formed this view through observation: * Kurzgesgat's video on HuggingFace reached 20M views in 12 days, which appears far above average for their videos, at least in recent years. * On The Argument, Kelsey Piper's AI posts seem to perform noticeably better than her education posts (~2x likes when I eyeballed). * My own most liked Substack post is about AI (specifically AI usage in the recent papal encyclical), though not AI Safety. * (The likes differential probably understates the difference, since most people engaged with the argument through secondary media. It's still my only article with significant secondary coverage outside the blogosphere since either 2022 or 2024) * My own AI safety posts perform about middle-of-the-pack, maybe slightly above, neither unusually good or bad compared to my general posts. * I suspect they're a bit overcomplicated, however, and if I dumb them down to the level of say my post on stealth or on bees they'd be more popular than either. * Scott Alexander's AI safety posts seem to broadly outperform his non-AI posts. * Daniel K's visit on the Joe Rogan Experience (the #1 podcast in America, I think) is the fourth most viewed episode of the 30 most recent episode on JRE, * Counterexample: Yglesias' top posts seem weakly to be an exception, but it's not like the AI safety posts are overwhelmingly unpopular. There's also a clear theoretical reason to expe