Anthropic's ownership is like 10% founder (of which 17% will be spent on good AI safety stuff)
On a dollar-weighted basis, Anthropic shareholders overwhelmingly have the sorts of views about superintelligence that lead them to believe that accelerating AI capabilities is a good idea, which makes me think their donations will mostly not be very good. My guess is the vast majority of Anthropic-sourced donations will go to the sort of work that AI companies were already doing internally anyway—basically, inoffensive safety research that's unlikely to actually prevent extinction.
1Anthropic shareholders overwhelmingly have the sorts of views about superintelligence that lead them to believe that accelerating AI capabilities is a good idea
I think Anthropic leadership believe that slightly slowing down AI capabilities progress is net-good for society. I'm focused on Anthropic leadership here because they've pledged much of their money for donations[1] and Claude says that Anthropic leadership owns ~14% of Anthropic (over $100B).
From Anthropic's When AI builds itself
We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.
There's admittedly a bit of ambiguity in their positions of whether slowdowns are good in both the first and last sentences here.[2] Still, unless this ambiguity is deliberate, I read this as them thinking that a slight slowdown is good.
My sense from Dario based on some interviews and writings is that he expects AGI soon and he expects that leadership are not ready for the exponential.
It's also possible that these statements are intentional lies / deceptions, but I don't find that likely. I find Anthropic leadership's overall story of "we think other companies are being irresponsible but we think p(x-risk) is low relative to the average LWer so we'll try to build AGI" to make sense in combination with believing "TAI coming somewhat more slowly is better for society as a whole" especially given how fast Anthropic timelines are. Even if they're optimistic as a whole about the benefits of TAI, it still seems right to prefer it to come out more slowly. Also, they'd prefer for TAI to be developed after Trump's term ends if they could pick.
1I don't know what true feelings live in their hearts, but behaviorally speaking, Anthropic leadership's behavior is not consistent with what I'd predict from someone who wants to slow down AI. It is consistent with what I'd predict from someone who wants to placate AI safety people while continuing to race ahead.
I liked Nate Soares' recent tweets about this (1, 2, 3). Excerpt:
Wikipedia spends more effort convincing you to donate $2 than AI companies spend galvanizing the world to stop the race. There's a real inconsistency between what they say and how they behave. This mealy-mouthed behavior incurs real costs.
Fair. I think I could have phrased it better, but my point is more like Anthropic doesn't seem to care too much about AI development speeds (or their impact on accelerating it) but insofar as they care, they'd think its better for society to have slightly slower societal AI progress. I think this is also consistent with their views.
I do agree that Anthropic's accelerating timelines so much (along with some dishonest behavior) was bad and Dario is very overoptimistic about the benefits of AI.
Ok, my last comment was not that relevant then. What I would say instead is: I believe the most important thing to be doing at current margins is trying to pause AI, and I do not expect the biggest Anthropic donors to spend their money on trying to pause AI. If "Anthropic doesn't seem to care too much about AI development speeds", then they'll probably spend their donation money on other things like technical alignment research, right?
Maybe in my original comment I shouldn't have said they "believe that accelerating AI capabilities is a good idea" because that's not quite right (it's more like "believe that working on AI capabilities is okay") and, more importantly, it's not the relevant bit of info—the relevant thing is that they're much more optimistic than I am about their chances of solving AI alignment or otherwise successfully navigating transformative AI without pausing, which I think will cause them to spend money in ways that aren't very useful according to my beliefs. And not just that we disagree on that particular issue, but that they are not thinking clearly about AI risk, and I expect this will cause them to spend money in highly suboptimal ways even if I can't predict exactly what they will want to spend money on.
1the hedge funds have recently been appreciating at ~5x per year (after fees, before taxes) (for some value of "recently"), which is crazy.
Can you provide more details? This seems more than an order of magnitude off.
I did a google search and here's the first article I found; you can find somewhat more info if you try harder: https://finance.yahoo.com/markets/stocks/articles/aschenbrenner-ai-focused-hedge-fund-155609137.html.
I'm talking about SALP and VARA. (These are not cherry-picked due to their high returns; they're the two hedge funds I talked about 1.5-2 years ago too.) Sorry, I see I didn't specify the hedge funds in the post.
Vara doesn't seem close to 5x. Both of these seem highly unlikely to produce those kinds of returns going forward because of capacity, let alone a 400x compounded return.
I'd be interested in reading a longer defense of this.
Yeah, my 400x view is pretty unstable and I certainly haven't justified it. One basic idea is that if AI companies that are currently 4% of global wealth and they will go to 60% (after dilution), that's 15x growth in your fraction of global wealth just by holding average AI investments, and you can do better by (1) using leverage and (2) making smarter investments.
You do better with leverage at the expense of increasing risk, including total loss in worlds that are only slightly off from what you expect.
You seem to be talking about what people with hundreds of billions collectively can do. Capacity is a real constraint at that point.
It's also likely some of the best investments won't want that much capital. Even if you're right on the 60%, some of that can come from new private companies you can't invest in or can only invest after they've grown a lot.
1David Reinstein's tool might be of interest to readers https://uj-ai-wealth-philanthropy-steelman.netlify.app/
This post summarizes some of my views. It states uncertain things boldly in part to [surface disagreements / help readers point out things that seem wrong or missing]. Note that many things are uncertain and many claims are not justified here.
One tl;dr: money is great even on the margin, and it's crucial to invest extremely well rather than merely well.
In 2026, philanthropists will donate about $1.6B to AI safety nonprofits (being slightly picky about what counts) (~$1.2B CG, ~$400M other). This has recently been growing at about 1.6x/year; this growth rate may increase in the future, even before crunch time.[1]
How much money will ever be available for AI safety philanthropy? Present value >$100B (in the mean and median scenarios, but really the distribution matters).
Anthropic will go public around December 2026. It's not obvious when shareholders will be allowed to sell their equity, but it's very likely by June 2027. Shareholders should probably sell most of their stake as soon as they can: AI safety philanthropists are massively overexposed to Anthropic and you can get similar returns elsewhere, not to mention that money may be more valuable in worlds where Anthropic fails.
If you invest very intelligently, very aggressively, and tax-free from now until superintelligence, I think you grow your expected fraction of global wealth by 400x (unstable) (before potential ~2x hedge fund fees), before considering expropriation. Footnote.[4] That's a huge deal; for people/foundations with lots of money, growing it as much as possible is very important. All three of the italicized conditions provide large multipliers; it's crucial to fulfill them all.[5] It's also nice to avoid direct Anthropic exposure and crucial to ensure that if you have lots of money you'll spend it very well rather than drifting, being captured, donating suboptimally, or waiting too long.
Before the intelligence explosion
AI safety nonprofits present excellent philanthropic opportunities. Fortunately for the world but unfortunately for marginal impact, funding AI safety nonprofits until superintelligence costs just present-value $7B (depending on returns).[6] This is a tiny fraction of available money. Nevertheless, I think the community is currently underspending; I think marginal opportunities are still great (~1.5 micro-AI-takeover-prevention-equivalents per $1M—or 15 basis points per billion—plus a bonus because prosocial actions have various positive side effects, including via other universes and via retrospective compensation for people who made the AI transition go well).
Misc megaprojects.
During the intelligence explosion
Buying compute—for AI labor, experiments/training, and maybe some control or resilience—seems very important. Little work has been done here and it's not certain that you will even be able to turn money into computeor use computewell. But I tentatively think this looks great even on the margin for present-value tens of billions of dollars (~0.4 micro-AI-takeover-prevention-equivalents per $1M given community-average growth; perhaps ~1 given better investing — plus bonus, but very unstable). If I was in charge of all AI safety philanthropy, I tentatively expect I would spend most of my money on this. (And this topic is super neglected; I think it's currently one of the best things to work on; it's currently my main project; I'm happy to pitch my friends on working on it.)
Misc megaprojects. I think it's likely we'll identify some good opportunities to spend a few billion dollars, but I think buying compute is hard to beat.
After the intelligence explosion
Turning capital into control over the cosmic endowment. It's unlikely that this will be possible, but nevertheless I think it looks decent in expectation (~0.5 micro-AI-takeover-prevention-equivalents per $1M, assuming 150x growth in share of global wealth — but unstable).
Some upshots:
I believe technical AI safety work has been growing more quickly, so as it grows to be a larger fraction of the portfolio, the total growth rate will approach its growth rate. And the increasing value of inference compute may increase the amount of money that technical AI safety orgs want.
Rough guess based on public information.
Quick guesses, don't defer: after series H, Anthropic's ownership is like 10% founder (of which 17% will be spent on good AI safety stuff), 24% staff (of which 65% is pledged for donation and 14% will be donated well), 66% investors (of which 5% are AI safety philanthropists, of which 70% will donate well) for a total of 10%*17% + 25%*14% + 65%*5%*70% = 7.5% AI safety, fudge down to 7%.
400x sounds crazy but also the hedge funds have recently been appreciating at ~5x per year (after fees, before taxes) (for some value of "recently"), which is crazy. I think the average AI investment will 15x as a fraction of global wealth, and you can dramatically beat that via (1) smarter investments and (2) leverage.
I think Anthropic stock will merely ~25x as a fraction of global wealth. If Anthropic is currently just under 0.3% of global wealth, and at superintelligence-time it's an expected 13% of global wealth, and there's 50% dilution between now and then, that's 22x growth as a fraction of global wealth. And there are other reasons to disprefer direct Anthropic exposure: money seems lower-EV in worlds where Anthropic succeeds because (1) other philanthropists will have more money and (2) longtermist philanthropy is less important (less low-hanging fruit in expectation).
Many things are unstable.
Actually I'm not sure whether tax-free is crucial. Possibly using standard best practices for tax optimization and deferral mostly suffices.
Assuming $1.5B/year now, growing at 1.6x/year, but 2x/year investment returns, gives $6.7B. This is unstable because 1.6 is close to 2, but whatever. Fudge down slightly because there is less than infinite time until singularity, but fudge up because spending may grow faster than 1.6x/year.