80% of MATS alumni who completed the program before 2025 are still working on AI safety today, based on a survey of all available alumni LinkedIns or personal websites (242/292 ~ 83%). 10% are working on AI capabilities, but only ~6 at a frontier AI company (2 at Anthropic, 2 at Google DeepMind, 1 at Mistral AI, 1 extrapolated). 2% are still studying, but not in a research degree focused on AI safety. The last 8% are doing miscellaneous things, including non-AI safety/capabilities software engineering, teaching, data science, consulting, and quantitative trading.

Of the 193+ MATS alumni working on AI safety (extrapolated: 234):

• 34% are working at a non-profit org (Apollo, Redwood, MATS, EleutherAI, FAR.AI, MIRI, ARC, Timaeus, LawZero, RAND, METR, etc.);
• 27% are working at a for-profit org (Anthropic, Google DeepMind, OpenAI, Goodfire, Meta, etc.);
• 18% are working as independent researchers, probably with grant funding from Open Philanthropy, LTFF, etc.;
• 15% are working as academic researchers, including PhDs/Postdocs at Oxford, Cambridge, MIT, ETH Zurich, UC Berkeley, etc.;
• 6% are working in government agencies, including in the US, UK, EU, and Singapore.

10% of MATS alumni co-founded an active AI safety start-up or team during or after the program, including Apollo Research, Timaeus, Simplex, ARENA, etc.

Errata: I mistakenly included UK AISI in the "non-profit AI safety organization" category instead of "government agency". I also mistakenly said that the ~6 alumni working on AI capabilities at frontier AI companies were all working on pre-training.

Why does the AI safety community need help founding projects?

1. AI safety should scale
   1. Labs need external auditors for the AI control plan to work
   2. We should pursue many research bets in case superalignment/control fails
   3. Talent leaves MATS/ARENA and sometimes struggles to find meaningful work for mundane reasons, not for lack of talent or ideas
   4. Some emerging research agendas don’t have a home
   5. There are diminishing returns at scale for current AI safety teams; sometimes founding new projects is better than joining an existing team
   6. Scaling lab alignment teams are bottlenecked by management capacity, so their talent cut-off is above the level required to do “useful AIS work”
2. Research organizations (inc. nonprofits) are often more effective than independent researchers
   1. “Block funding model” is more efficient, as researchers can spend more time researching, rather than seeking grants, managing, or other traditional PI duties that can be outsourced
   2. Open source/collective projects often need a central rallying point (e.g., EleutherAI, dev interp at Timaeus, selection theorems and cyborgism agendas seem too delocalized, etc.)
3. There is (imminently) a market for for-profit AI safety companies and value-aligned people should capture this free energy or let worse alternatives flourish
   1. If labs or API users are made legally liable for their products, they will seek out external red-teaming/auditing consultants to prove they “made a reasonable attempt” to mitigate harms
   2. If government regulations require labs to seek external auditing, there will be a market for many types of companies
   3. “Ethical AI” companies might seek out interpretability or bias/fairness consultants
4. New AI safety organizations struggle to get funding and co-founders despite having good ideas
   1. AIS researchers are usually not experienced entrepeneurs (e.g., don’t know how to write grant proposals for EA funders, pitch decks for VCs, manage/hire new team members, etc.)
   2. There are not many competent start-up founders in the EA/AIS community and when they join, they don’t know what is most impactful to help
   3. Creating a centralized resource for entrepeneurial education/consulting and co-founder pairing would solve these problems

I am a Manifund Regrantor. In addition to general grantmaking, I have requests for proposals in the following areas:

• Funding for AI safety PhDs (e.g., with these supervisors), particularly in exploratory research connecting AI safety theory with empirical ML research.
• An AI safety PhD advisory service that helps prospective PhD students choose a supervisor and topic (similar to Effective Thesis, but specialized for AI safety).
• Initiatives to critically examine current AI safety macrostrategy (e.g., as articulated by Holden Karnofsky) like the Open Philanthropy AI Worldviews Contest and Future Fund Worldview Prize.
• Initiatives to identify and develop "Connectors" outside of academia (e.g., a reboot of the Refine program, well-scoped contests, long-term mentoring and peer-support programs).
• Physical community spaces for AI safety in AI hubs outside of the SF Bay Area or London (e.g., Japan, France, Bangalore).
• Start-up incubators for projects, including evals/red-teaming/interp companies, that aim to benefit AI safety, like Catalyze Impact, Future of Life Foundation, and YCombinator's request for Explainable AI start-ups.
• Initiatives to develop and publish expert consensus on AI safety macrostrategy cruxes, such as the Existential Persuasion Tournament and 2023 Expert Survey on Progress in AI (e.g., via the Delphi method, interviews, surveys, etc.).
• Ethics/prioritization research into:
  • What values to instill in artificial superintelligence?
  • How should AI-generated wealth be distributed?
  • What should people do in a post-labor society?
  • What level of surveillance/restriction is justified by the Unilateralist's Curse?
  • What moral personhood will digital minds have?
  • How should nations share decision making power regarding transformative AI?
• New nonprofit startups that aim to benefit AI safety.

Yes, I would generally support picking the latter as they have a "faster time to mentorship/research leadership/impact" and the field seems currently bottlenecked on mentorship and research leads, not marginal engineers (though individual research leads might feel bottlenecked on marginal engineers).

We should prioritize people who already have research or engineering experience or a very high iteration speed as we are operating under time constraints; AGI is coming soon. Additionally, I think "research taste" will be more important than engineering ability given AI automation and this takes a long time to build; better to select people with existing research experience they can adapt from another field (also promotes interdisciplinary knowledge transfer).

I talk more about it here.

Yoshua Bengio, Paul Christiano, Geoffrey Irving seem more like technical AI safety experts than AI policy experts, but they arguably have strong influence on governments.

What should be done? I think:

• Office hubs: Expand SASS, which is in close proximity to the new OpenAI and Anthropic offices in Sydney. Start an AI safety hub in Canberra to support the new AISI. Successful AI safety hubs have benefited from prominent founding member organizations like ARC, CG, Redwood, and MIRI (for Constellation) and Apollo, BlueDot, and MATS (for LISA). Similarly, SASS should bring together orgs like the Gradient Institute and Harmony Intelligence in a shared space, and the new Canberra hub should be built around Good Ancestors. Office hubs can benefit member orgs by providing cheaper returns to scale, hosting shared networking events, facilitating collaboration, and providing a pipeline of strong new recruits in the form of office guests and members.
• Training programs: Expand TARA and the Sydney AI Safety Fellowship program, focusing on accelerating top talent and building local mentorship capacity for future programs. Don't focus on maximizing impact on participants; this is less important than reducing the mentorship bottleneck, which is best served by boosting the most advanced participants.
• Academic labs: Build relationships with AI/CS academics at UniMelb, Monash, USyd, ANU, UTS, UNSW, UA, UQ, etc. Help launch AI safety courses like Roy Rinberg and Boaz Barak did at Harvard. Other course inspiration is provided by Stanford and CAIS. Start AI safety academic labs like UC Berkeley CHAI, MIT AAG, NYU ARG, Bau Lab, Stanford HAI, CMU FOCAL, etc.
• Conferences: Run an annual AI safety conference like the Australian AI Safety Forum 2024, bringing together academia, industry, government, and nonprofit field-builders. EAGx is probably not enough, as many people from academia, industry, and government likely won't attend.

I suspect that some LWers would interpret this as a (bad) argument for countries to build datacenters so they can exercise political control over AGI. I don't think this works.

Exactly! Also:

• AI safety researchers tend to have reasonable views on AI safety and can serve as local advisors to governments, which probably trust foreign experts less.
• Securing datacenters for SL5 to prevent proliferation of AGI is really hard and likely requires significant AI security expertise in government and local defence contractors.
• A regulatory market approach to AI safety (possibly only useful pre-superintelligence) requires competent local auditors, standard-setters, and insurers.

AI safety field-building in Australia should accelerate. My rationale:

• OpenAI opened a Sydney office in Dec 2025 and Anthropic is planning to open a Sydney office in 2026. These offices may hire safety staff from local talent, or partner with local auditing, evaluation, and security companies, including Harmony Intelligence, Good Ancestors, and Gradient Institute.
• An Australian AISI was announced for early 2026 and is currently hiring. The UK AISI has benefited from close partnerships with Apollo Research, METR, and the LISA office community. There is a community space in Sydney, the Sydney AI Safety Space, and two field-building organizations, AI Safety ANZ and TARA, but these could expand substantially.
• Australia seems like a prime location for datacenter build-out. OpenAI published an "AI blueprint" for Australia, calling for datacenter build-out, and started building a $4.6B datacenter in Sydney in Dec 2025. Australia is a NATO partner, Five Eyes member, and member of the AUKUS security partnership with the US and UK; it's much more secure and aligned with US/UK interests than Saudi Arabia. Australia is the second-largest exporter of thermal coal, has vast solar and wind resources, and the highest uranium reserves on earth. Australia is currently quite anti-nuclear at the moment, but it has no earthquakes or tsunamis to disrupt power plants. Janet Egan (CNAS) recently called for the development of US military AI projects in Australia, similar to the Pine Gap facility in the Northern Territory. AI safety & security research and political pressure for safety standards should focus on countries with frontier AI companies and datacenters.
• Several prominent AI safety researchers have come from Australia, including Marcus Hutter, Buck Shlegeris, Jan Leike (PhD only), Ramana Kumar, Dan Murfet, and Daniel Filan. A decent number of MATS fellows come from Australia. Australian citizens can easily emigrate to the US (E-3 visa) or UK (YMS visa) for work. Seven of the top-100 computer science universities are in Australia.
• Australia has a history as a middle power between the US and China, two of its largest trading partners. The country may play a significant role in international diplomacy efforts, such as a superintelligence ban or international AGI project.
• Australia has a strong history of environmental activism, which might (or might not) be a useful asset for AI safety political mobilization.

I think that the distribution of mentors we are drawing from is slowing growing to include more highly respected academics and industry professionals by percentage. I think this increases the average quality of our mentor applicant pool, but I understand that this is might be controversial. Note that I still think our most impactful mentors are well-known within the AI safety field and most of the top-50 most impactful researchers in AI safety apply to mentor at MATS.