AI Safety Field Growth Analysis 2025
9DanielFilan
4Stephen McAleese
5Benjamin_Todd
5Bogdan Ionut Cirstea
4Stephen McAleese
2Bogdan Ionut Cirstea
3reallyeli
4Jonathan Claybrough
3Stephen McAleese
1Peter Favaloro
2Stephen McAleese
1Peter Favaloro
2Stephen McAleese
New Comment
Based on my recollections of being around in 2015, your number from then seems too high to me (I would have guessed there were at most 30 people doing what I would have thought of as AI x-risk research back then). Can I get a sense of who you're counting?
Thanks for your helpful feedback Daniel. I agree that the estimate for 2015 (~50 FTEs) is too high. The reason why is that the simple model assumes that the number of FTEs is constant over time as soon as the organization is founded.
For example, the FTE value associated with Google DeepMind is 30 today and the company was founded in 2010 so the value back then is probably too high.
Perhaps a more realistic model would assume that the organization has 1 FTE when founded and linearly increases. Though this model would be inaccurate for organizations that grow rapidly and then plateau in size after being founded.
Interesting that the growth rate of technical alignment researchers seem to be a bit slower than capabilities researchers. Compare:
The exponential models describe a 24% annual growth rate in the number of technical AI safety organizations and 21% growth rate in the number of technical AI safety FTEs.
To the following taken from here:
In 2021, OpenAI had about 300 employees; today, it has about 3,000. Anthropic and DeepMind have also grown more than 3x, and new companies have entered. The number of ML papers produced per year has roughly doubled every two years.
This suggests to me that the capabilities field is growing more like 30-40% per year.
In addition, governance seems to be growing even slower than technical safety, even in the last 3 years.
Based on updated data and estimates from 2025, I estimate that there are now approximately 600 FTEs working on technical AI safety and 500 FTEs working on non-technical AI safety (1100 in total).
I think it's suggestive to compare with e.g. the number of FTEs related to addressing climate change, for a hint at how puny the numbers above are:
Using our definition's industry approach, UK employment in green jobs was an estimated 690,900 full-time equivalents (FTEs) in 2023. (https://www.ons.gov.uk/economy/environmentalaccounts/bulletins/experimentalestimatesofgreenjobsuk/july2025)
Jobs in renewable energy reached 16.2 million globally in 2023 (https://www.un.org/en/climatechange/science/key-findings)
I think it's hard to pick a reference class for the field of AI safety because the number of FTEs working on comparable fields or projects can vary widely.
Two extremes examples:
- Apollo Program: ~400,000 FTEs
- Law of Universal Gravitation: 1 FTE (Newton)
Here are some historical challenges which seem comparable to AI safety since they are technical, focused on a specific challenge, and relatively recent [1]:
- Pfizer-BioNTech vaccine (2020): ~2,000 researchers and ~3,000 FTEs for manufacturing and logistics
- Human genome project (1990 - 2003): ~3,000 researchers across ~20 major centers
- ITER fusion experiment (2006 - present): ~2,000 engineers and scientists, ~5000 FTEs in total
- CERN and LHC (1994 - present): ~3000 researchers working onsite, ~15,000 collaborators arouond the world.
I think these projects show that it's possible to make progress on major technical problems with a few thousand talented and focused people.
- ^
These estimates were produced using ChatGPT with web search.
I think these projects show that it's possible to make progress on major technical problems with a few thousand talented and focused people.
I don't think it's impossible that this would be enough, but it seems much worse to risk undershooting than overshooting in terms of the resources allocated and the speed at which this happens; especially when, at least in principle, the field could be deploying even its available resources much faster than it currently is.
While I like the idea of the comparison, I don't think the gov't definition of "green jobs" is the right comparison point. (e.g. those are not research jobs)
2Thanks for this work.
In your technical dataset apart research appears twice, with 10 and 40 FTEs listed respectively, is that intentional? Is it meant to track the core team vs volunteers participating in hackathons etc?
Can you say a bit more about how these numbers are estimated? eg. 1 person looks at the websites and writes down how many they see, estimating from other public info, directly asking the orgs when possible?
I'm pretty sure that's just a mistake. Thanks for spotting it! I'll remove the duplicated row.
For each organization, I estimated the number of FTEs by looking at the team members page, LinkedIn, and what kinds of outputs have been produced by the organization and who is associated with them. Then the final estimate is an intuitive guess based on this information.
I'm surprised that Anthropic and GDM have such small numbers of technical safety researchers in your dataset. What are the criteria for inclusion / how did you land on those numbers?
That's a good question. One approach I took is to look at the research agendas and outputs (e.g. Google DeepMinds AI safety research agenda) and estimate the number of FTEs based on those.
I would say that I'm including teams that are working full-time on advancing technical AI safety or interpretability (e.g. the GDM Mechanistic Interpretability Team).
To the best of my knowledge, there are a few teams like that at Google DeepMind and Anthropic though I could be underestimating given that these organizations have been growing rapidly over the past few years.
A weakness of this approach is that there could be large numbers of staff who sometimes work on AI safety and significantly increase the effective number of AI safety FTEs at the organization.
What changed that made the historical datapoints so much higher? E.g. you now think that there were >100 technical AIS researchers in 2016, whereas in 2022 you thought that there had been <50 technical AIS researchers in 2016.
I notice that the historical data revisions are consistently upward. This looks consistent with a model like: in each year x, you notice some "new" people that should be in your dataset, but you also notice that they've been working on TAIS-related stuff for many years by that point. If we take that model literally, and calibrate it to past revisions, it suggests that you're probably undercounting right now by a factor of 50-100%. Does that sound plausible to you?
Thanks for assembling this dataset!
Good observation, thanks for sharing.
One possible reason is that I've included more organizations in this updated post and this would raise many estimates.
Another reason is that in the old post, I used a linear model that assumed that an organization started with 1 FTE when founded and linearly increased until the current number (example: an organization has 10 FTEs in 2025 and was founded in 2015. Assume 1 FTE in 2015, 2 FTEs in 2016 ... 10 in 2025).
The new model is simpler and just assumes the current number for all years (e.g. 10 in 2015 and 10 in 2025) so it's estimates for earlier years are higher than the previous model. See my response to Daniel above.
Curated and popular this week
