Last year I gave my reasoning on cause prioritization and did shallow reviews of some relevant orgs. I'm doing it again this year.
Cross-posted to my website.
In September, I published a report on the AI safety landscape, specifically focusing on AI x-risk policy/advocacy.
The prioritization section of the report explains why I focused on AI policy. It's similar to what I wrote about prioritization in my 2024 donations post, but more fleshed out. I won't go into detail on cause prioritization in this post because those two previous articles explain my thinking.
My high-level prioritization is mostly unchanged since last year. In short:
In the rest of this section, I will cover:
By donating, I want to increase the chances that we get a global ban on developing superintelligent AI until it is proven safe.
"The Problem" is my favorite article-length explanation of why AI misalignment is a big deal. For a longer take, I also like MIRI's book.
MIRI says:
On our view, the international community’s top immediate priority should be creating an “off switch” for frontier AI development. By “creating an off switch”, we mean putting in place the systems and infrastructure necessary to either shut down frontier AI projects or enact a general ban.
I agree with this. At some point, we will probably need a halt on frontier AI development, or else we will face an unacceptably high risk of extinction. And that time might arrive soon, so we need to start working on it now.
This Google Doc that explains why I believe a moratorium on frontier AI development is better than "softer" safety regulations. In short: no one knows how to write AI safety regulations that prevent us from dying. If we knew how to do that, then I'd want it; but since we don't, the best outcome is to not build superintelligent AI until we know how to prevent it from killing everyone.
That said, I still support efforts to implement AI safety regulations, and I think that sort of work is among the best things one can be doing, because:
Safety regulations can help us move in the right direction, for example:
My ideal regulation is global regulation. A misaligned AI is dangerous no matter where it's built. (You could even say that if anyone builds it, everyone dies.) But I have to idea how to make global regulations happen; it seems that you need to get multiple countries on board with caring about AI risk and you need to overcome coordination problems.
I can think of two categories of intermediate steps that might be useful:
A world in which the USA, China, and the EU all have their own AI regulations is probably a world in which it's easier to get all those regions to agree on an international treaty.
People often criticize the "pause AI" plan by saying it's not feasible.
I agree. I don't think it's going to work. [[1]]
I don't think more "moderate" [[2]] AI safety regulations will work, either.
I don't think AI alignment researchers are going to figure out how to prevent extinction.
I don't see any plan that looks feasible.
"Advocate for and work toward a global ban on the development of unsafe AI" is my preferred plan, but not because I like the plan. It's a bad plan. I just think it's less bad than anything else I've heard.
My P(doom) is not overwhelmingly high (it's in the realm of 50%). But if we live, I expect that it will be due to luck. [[3]] I don't see any way to make a significant dent on decreasing the odds of extinction.
I don't have a strong argument for why I believe this. It just seems true to me.
The short version is something like "the other plans for preventing AI extinction are worse than people think" + "pausing AI is not as intractable as people think" (mostly the first thing).
The folks at MIRI have done a lot of work to articulate their position. I directionally agree with almost everything they say about AI misalignment risk (although I'm not as confident as they are). I think their policy goals still make sense even if you're less confident, but that's not as clear, and I don't think anyone has ever done a great job of articulating the position of "P(doom) is less than 95%, but pausing AI is still the best move because of reasons XYZ".
I'm not sure how to articulate it either; it's something I want to spend more time on in the future. I can't do a good job of it on this post, so I'll leave it as a future topic.
Transformative AI could create many existential-scale problems that aren't about misalignment. Relevant topics include: misuse; animal-inclusive AI; AI welfare; S-risks from conflict; gradual disempowerment; risks from malevolent actors; moral error.
I wrote more about non-alignment problems here. I think pausing AI is the best way to handle them, although this belief is weakly held.
By that I mean the problem of making sure that transformative AI is good for non-humans as well as humans.
This is a reversion to my ~2015–2020 position. If you go back and read My Cause Selection (2015), I was concerned about AI misalignment, but I was also concerned about an aligned-to-humans AI being bad for animals (or other non-human beings), and I was hesitant to donate to any AI safety orgs for that reason.
In my 2024 cause prioritization, I didn't pay attention to AI-for-animals because I reasoned that x-risk seemed more important.
This year, in preparation for writing the AI safety landscape report for Rethink Priorities, they asked me to consider AI-for-animals interventions in my report. At first, I said I didn't want to do that because misalignment risk was a bigger deal—if we solved AI alignment, non-humans would probably end up okay. But I changed my mind after considering a simple argument:
Suppose there's an 80% chance that an aligned(-to-humans) AI will be good for animals. That still leaves a 20% chance of a bad outcome. AI-for-animals receives much less than 20% as much funding as AI safety. Cost-effectiveness maybe scales with the inverse of the amount invested. Therefore, AI-for-animals interventions are more cost-effective on the margin than AI safety.
So, although I believe AI misalignment is a higher-probability risk, it's not clear that it's more important than AI-for-animals.
Last year, I thought a moratorium on frontier AI development was probably the best political outcome. Now I'm a bit more confident about that, largely because—as far as I can see—it's the best way to handle non-alignment problems.
Last year, I donated to PauseAI US and PauseAI Global because I guessed that protests were effective. But I didn't have much reason to believe that, just some vague arguments. In April of this year, I followed up with an investigation of the strongest evidence on protest outcomes, and I found that the quality of evidence was better than I'd expected. I am now pretty confident that peaceful demonstrations (like what PauseAI US and PauseAI Global do) have a positive effect. The high-quality evidence looked at nationwide protests; I couldn't find good evidence on small protests, so I'm less confident about them, but I suspect that they do.
I also wrote about how I was skeptical of Stop AI, a different protest org that uses more disruptive tactics. I've also become more confident in my skepticism: I've been reading some literature on disruptive protests, and the evidence is mixed. That is, I'm still uncertain about whether disruptive protests work, but my uncertainty has shifted from "I haven't looked into it" to "I've looked into it, and the evidence is ambiguous, so I was right to be uncertain." (I've shifted from one kind of "no evidence" to the other.) For more, see my recent post, Do Disruptive or Violent Protests Work? [[4]]
Last year, I looked for grantmakers who I could defer to, but I couldn't find any who I trusted enough, so I did my own investigation. I've become increasingly convinced that that was the correct decision, and I am increasingly wary of people in the AI safety space—I think a large minority of them are predictably making things worse.
I wrote my thoughts about this in a LessWrong quick take. In short, AI safety people/groups have a history of looking like they will prioritize x-risk, and then instead doing things that are unrelated or even predictably increase risk. [[5]] So I have a high bar for which orgs I trust, and I don't want to donate to an org if it looks wishy-washy on x-risk, or if it looks suspiciously power-seeking (a la "superintelligent AI will only be safe if I'm the one who builds it"). I feel much better about giving to orgs that credibly and loudly signal that AI misalignment risk is their priority.
Among grantmakers, I trust the Survival & Flourishing Fund the most, but they don't make recommendations for individual donors. SFF is matching donations on some orgs through the end of 2025 (see the list), which signals which orgs they want more people to donate to.
In the report I published this September, I reviewed a list of interventions related to AI and quickly evaluated their pros and cons. I arrived at four top ideas:
The first two ideas relate to AI x-risk policy/advocacy, and the second two are about making AI go better for animals (or other non-human sentient beings).
For my personal donations, I'm just focusing on x-risk.
At equal funding levels, I expect AI x-risk work to be more cost-effective than work on AI-for-animals. The case for AI-for-animals is that it's highly neglected. But the specific interventions I like best within AI x-risk are also highly neglected, perhaps even more so.
I'm more concerned about the state of funding in AI x-risk advocacy, so that's where I plan on donating.
A second consideration is that I want to support orgs that are trying to pause frontier AI development. If they succeed, that buys more time to work on AI-for-animals. So those orgs help both causes at the same time.
I'm not qualified to evaluate AI policy orgs, but I also don't trust anyone else enough to delegate to them, so I am reviewing them myself.
I have a Google doc with a list of every relevant organization I could find. Unlike in my 2024 donation post, I'm not going to talk about all of the orgs on the list, just my top contenders. For the rest of the orgs I wrote about last year, my beliefs have mostly not changed.
I separated my list into "tax-deductible" and "non-tax-deductible" because most of my charitable money is in my donor-advised fund, and that money can't be used to support political groups. So the two types of donations aren't coming out of the same pool of money.
As I mentioned above, I don't plan on donating to orgs in the AI-for-animals space, and I haven't looked much into them. But I will briefly list some orgs anyway. My first impression is that all of these orgs are doing good work.
Compassion in Machine Learning does research and works with AI companies to make LLMs more animal-friendly.
NYU Center for Mind, Ethics, and Policy conducts and supports foundational research on the nature of nonhuman minds, including biological and artificial minds.
Open Paws creates AI tools to help animal activists and software developers make AI more compassionate toward animals.
Sentience Institute conducts foundational research on long-term moral-circle expansion and digital-mind welfare.
Sentient Futures organizes conferences on how AI impacts non-human welfare (including farm animals, wild animals, and digital minds); built an animal-friendliness LLM benchmark; and is hosting an upcoming war game on how AGI could impact animal advocacy.
Wild Animal Initiative mostly does research on wild animal welfare, but it has done some work on AI-for-animals (see Transformative AI and wild animals: An exploration.
The AI Safety and Governance Fund does message testing) on what sorts of AI safety messaging people found compelling. More recently, they created a chatbot that talks about AI x-risk, which they use to feed into their messaging experiments; they also have plans for new activities they could pursue with additional funding.
I liked AI Safety and Governance Fund's previous project, and I donated $10,000 because I expected they could do a lot of message testing for not much money. I'm more uncertain about its new project, or how well message testing can scale. I'm optimistic, but not optimistic enough for the org to be one of my top donation candidates, so I'm not donating more this year.
Existential Risk Observatory writes media articles on AI x-risk, does policy research, and publishes policy proposals (see pdf with a summary of proposals).
Last year, I wrote:
My primary concern is that it operates in the Netherlands. Dutch policy is unlikely to have much influence on x-risk—the United States is the most important country by far, followed by China. And a Dutch organization likely has little influence on United States policy. Existential Risk Observatory can still influence public opinion in America (for example via its TIME article), but I expect a US-headquartered org to have a greater impact.
I'm less concerned about that now—I believe I gave too little weight to the fact that Existential Risk Observatory has published articles in international media outlets.
I still like media outreach as a form of impact, but it's not my favorite thing, so Existential Risk Observatory is not one of my top candidates.
The biggest news from MIRI in 2025 is that they published a book. The book was widely read and got some endorsements from important people, including people who I wouldn't have expected to give endorsements. It remains to be seen what sort of lasting impact the book will have, but the launch went better than I would've predicted a year ago (perhaps in the 75th percentile).
MIRI's 2026 plans include:
I'm less enthusiastic about policy research than about advocacy, but I like MIRI's approach to policy research better than any other org's. Most AI policy orgs take an academia-style approach of "what are some novel things we can publish about AI policy?" MIRI takes a more motivated approach of "what policies are necessary to prevent extinction, and what needs to happen before those policies can be implemented?" Most policy research orgs spend too much time on streetlight-effect policies; MIRI is strongly oriented toward preventing extinction.
I also like MIRI better than I did a year ago because I realized they deserve a "stable preference bonus".
In My Cause Selection (2015), MIRI was my #2 choice for where to donate. In 2024, MIRI again made my list of finalists. The fact that I've liked MIRI for 10 years is good evidence that I'll continue to like it.
Maybe next year I will change my mind about my other top candidates, but—according to the Lindy effect—I bet I won't change my mind about MIRI.
The Survival and Flourishing Fund is matching 2025 donations to MIRI up to $1.3 million.
Palisade builds demonstrations of the offensive capabilities of AI systems, with the goal of illustrating risks to policy-makers. My opinion on Palisade is mostly unchanged since last year, which is to say it's one of my favorite AI safety nonprofits.
They did not respond to my emails asking about their fundraising situation. Palisade did recently receive funding from the Survival and Flourishing Fund (SFF) and appeared on their Further Opportunities page, which means SFF thinks Palisade can productively use more funding.
The Survival and Flourishing Fund is matching 2025 donations to Palisade up to $900,000.
PauseAI US was the main place I donated last year. Since then, I've become more optimistic that protests are net positive.
Pause protests haven't had any big visible effects in the last year, which is what I expected, [[6]] but it's a weak negative update that the protests haven't yet gotten traction.
I did not list protests as one of my favorite interventions; in the abstract, I like political advocacy better. But political advocacy is more difficult to evaluate, operates in a more adversarial information environment [[7]] , and less neglected. There is some hypothetical political advocacy that I like better than protests, but it's much harder to tell whether the real-life opportunities live up to that hypothetical.
PauseAI US has hired a full-time lobbyist. He's less experienced than the lobbyists at some other AI safety orgs, but I know that his lobbying efforts straightforwardly focus on x-risk instead of doing some kind of complicated political maneuvering that's hard for me to evaluate, like what some other orgs do. PauseAI US has had some early successes but it's hard for me to judge how important they are.
Something that didn't occur to me last year, but that I now believe matters a lot, is that PauseAI US organizes letter-writing campaigns. In May, PauseAI US organized a campaign to ask Congress members not to impose a 10-year moratorium on AI regulation; they have an ongoing campaign in support of the AI Risk Evaluation Act. According to my recent cost-effectiveness analysis, messaging campaigns look valuable, and right now nobody else is doing it. [[8]] It could be that these campaigns are the most important function of PauseAI US.
Recently, more people have been trying to advocate for AI safety by making videos. I like that this is happening, but I don't have a good sense of how to evaluate video projects, so I'm going to punt on it. For some discussion, see How cost-effective are AI safety YouTubers? and Rethinking The Impact Of AI Safety Videos.
I didn't start thinking seriously about non-tax-deductible opportunities until late September. By late October, it was apparent that I had too many unanswered questions to be able to publish this post in time for giving season.
Instead of explaining my position on these non-tax-deductible opportunities (because I don't have one), I'll explain what open questions I want to answer.
There's a good chance I will donate to one of these opportunities before the end of the year. If I do, I'll write a follow-up post about it (which is why this post is titled Part 1).
AI Policy Network advocates for US Congress to pass AI safety regulation. From its description of The Issue, it appears appropriately concerned about misalignment risk, but it also says
AGI would further have large implications for national security and the balance of power. If an adversarial nation beats the U.S. to AGI, they could potentially use the power it would provide – in technological advancement, economic activity, and geopolitical strategy – to reshape the world order against U.S. interests.
I find this sort of language concerning because it appears to be encouraging an arms race, although I don't think that's what the writers of this paragraph want.
I don't have a good understanding of what AI Policy Network does, so I need to learn more.
Americans for Responsible Innovation (ARI) is the sort of respectable-looking org that I don't expect to struggle for funding. But I spoke to someone at ARI who believes that the best donation opportunities depend on small donors because there are legal donation caps. Even if the org as a whole is well-funded, it depends on small donors to fund its PAC.
I want to put more thought into how valuable ARI's activities are, but I haven't had time to do that yet. My outstanding questions:
ControlAI is the most x-risk-focused of the 501(c)(4)s, and the only one that advocates for a pause on AI development. They started operations in the UK, and this year they have expanded to the US.
Some thoughts:
ControlAI is tentatively my favorite non-tax-deductible org because they're the most transparent and the most focused on x-risk.
Two state representatives, Scott Weiner and Alex Bores, are running for US Congress. Both of them have sponsored successful AI safety legislation at the state level (SB 53 and the RAISE Act, respectively). We need AI safety advocates in US Congress, or bills won't get sponsored.
Outstanding questions:
Encode does political advocacy on AI x-risk. They also have local chapters that do something (I'm not clear on what).
They have a good track record of political action:
Encode is relatively transparent and relatively focused on the big problems, although not to the same extent as ControlAI.
All of the orgs on my 501(c)(3) list deserve more funding. (I suspect the same is true of the 501(c)(4)s, but I'm not confident.) My favorite 501(c)(3) donation target is PauseAI US because:
In other words, PauseAI US is serving some important functions that nobody else is on top of, and I really want them to be able to keep doing that.
My plan is to donate $40,000 to PauseAI US.
2025-11-22: Corrected description of AI Safety and Governance Fund.
Although I'm probably more optimistic about it than a lot of people. For example, before the 2023 FLI Open Letter, a lot of people would've predicted that this sort of letter would never be able to get the sort of attention that it ended up getting. (I would've put pretty low odds on it, too; but I changed my mind after seeing how many signatories it got.) ↩︎
I disagree with the way many AI safety people use the term "moderate". I think my position of "this thing might kill everyone and we have no idea how to make it not do that, therefore it should be illegal to build" is pretty damn moderate. Mild, even. There are far less dangerous things that are rightly illegal. The standard-AI-company position of "this has a >10% chance of killing everyone, but let's build it anyway" is, I think, much stranger (to put it politely). And it's strange that people act like that position is the moderate one. ↩︎
Perhaps we get lucky, and prosaic alignment is good enough to fully solve the alignment problem (and then the aligned AI solves all non-alignment problems). Perhaps we get lucky, and superintelligence turns out to be much harder to build than we thought, and it's still decades away. Perhaps we get lucky, and takeoff is slow and gives us a lot of time to iterate on alignment. Perhaps we get lucky, and there's a warning shot that forces world leaders to take AI risk seriously. Perhaps we get lucky, and James Cameron makes Terminator 7: Here's How It Will Happen In Real Life If We Don't Change Course and the movie changes everything. Perhaps we get lucky, and I'm dramatically misunderstanding the alignment problem and it's actually not a problem at all.
Each of those things is unlikely on its own. But when you add up all the probabilities of those things and everything else in the same genre, you end up with decent odds that we survive. ↩︎
I do think Stop AI is morally justified in blockading AI companies' offices. AI companies are trying to build the thing that kills everyone; Stop AI protesters are justified in (non-violently) trying to stop them from doing that. Some of the protesters have been taken to trial, and if the courts are just, they will be found not guilty. But I dislike disruptive protests on pragmatic grounds because they don't appear particularly effective. ↩︎
I want to distinguish "predictably" from "unpredictably". For example, MIRI's work on raising concern for AI risk appears to have played a role in motivating Sam Altman to start OpenAI, which greatly increased x-risk (and was possibly the worst thing to ever happen in history, if OpenAI ends up being the company to build the AI that kills everyone). But I don't think it was predictable in advance that MIRI's work would turn out to be harmful in that way, so I don't hold it against them. ↩︎
On my model, most of the expected value of running protests comes from the small probability that they grow a lot, either due to natural momentum or because some inciting event (like a warning shot) suddenly makes many more people concerned about AI risk. ↩︎
I have a good understanding of the effectiveness of protests because I've done the research. For political interventions, most information about their effectiveness comes from the people doing the work, and I can't trust them to honestly evaluate themselves. And many kinds of political action involve a certain Machiavellian-ness, which brings various conundrums that make it harder to tell whether the work is worth funding. ↩︎
MIRI and ControlAI have open-ended "contact your representative" pages (links: MIRI, ControlAI), but they haven't done messaging campaigns on specific legislation. ↩︎