Epistemic status: a deep dive over the past few months, and focused work over the past few weeks.

TL;DR

• AI safety work is extremely siloed. Of 500+ safety organizations, ~80 build tools of any kind (mostly in security or security-adjacent), and none of the 100 fellowships and courses we mapped are dedicated to tool builders.
• If we take AI progress seriously, we need generalizable, adopted and deployed AI safety tools as public goods.
• We identified clear gaps (multi-agent safety, incident reporting, CBRN monitoring, epistemic risks and more), and some early movement including our pilot AI Safety for Builders fellowship. We also noted some pitfalls (dual use, labs building first, etc).

Introduction

This post argues that we need dedicated efforts to build tools for AI safety. We use "tools" as an umbrella term for AI safety-focused apps, (research) automation, artefacts, cyber ranges, guardrails, harnesses, instruments, libraries, platforms, pipelines, plugins, products, protocols, sandboxes, workflows, etc.

Mapping the current AI safety landscape

As of today, AI safety work is extremely siloed. Based on our exhaustive mapping of the ecosystem^[1]

• Leaving aside a few big labs, there are now more than 500 safety organizations employing roughly 2,000 FTEs (about 4 FTEs per org on average).
• Although there are ~80 organizations building AI safety tools of some kind, most of them are concentrated in security or security-adjacent areas.

The entry path to AI safety is almost entirely non-tool-focused

The paths into AI safety, i.e. courses, fellowships, etc., are almost exclusively focused on research (with a few new ones on fieldbuilding).

Of the 100 fellowships and courses we mapped, not one is dedicated to AI safety tool builders.

Notable AI safety tools

Many AI safety orgs do produce tools, and the outcomes of research undertaken in fellowships are often tools. However, these are mostly built for internal, hyper-specific use cases, without generalizability in mind.

Here's the current landscape of some notable safety-focused tools that are reasonably well adopted:

Why tools, and why now?

If we take AI progress and its implications seriously, ensuring AI goes well means automating a lot of AI safety work. In a world where trustworthy, aligned AI can do massive good (and reduce massive harms), AI safety tools can be seen as important public goods and public infrastructure.

As mentioned before, the argument is not that tools aren't being built, but that they are not sufficiently generalizable, adopted or deployed. There are at least three ways a push to create AI safety-focused tools will help:

1. Acceleration. A focus on horizontal AI safety tools will accelerate AI safety research and work, as well as adoption. A recent blog post by Goodfire stresses why.
2. Affordances. Better, higher-quality tools also mean higher affordances, i.e. they enable people to apply safety use cases in novel, creative ways (this cuts both ways; see pitfalls).
3. Participation. A focus on AI safety tools will also enable non-researchers to contribute to the field, enable builders and engineers to do useful work, and mobilize ecosystems like Bengaluru, Shenzhen, etc.

Tools or products?

Does a focus on tools necessarily imply products (as in for-profit or startup products)? Yes, there is overlap, and some of the tools will (eventually) be products. Others have also argued that AI safety needs products (see articles by Apollo Research and BlueDot).

The main claims are:

• For-profit products attract more funding, especially from venture capital.
• They are also better suited for operations and deployment at large scale.
• They force startups to stay closer to real needs and issues.
• Catering to a market sometimes gives more independence than catering to governments or grantmakers. AE Studio and Faculty are good examples of orgs that operate in a for-profit mould and yet do a lot of impactful safety work.

However, there are good reasons why a focus on AI safety tools doesn't necessarily imply a for-profit, product-style focus. Some considerations:

• For-profit AI safety startups are for-profit first and AI safety second. It's extremely easy to get pulled sideways.
• The market doesn't necessarily price in good outcomes or impact until it's (too) late.

Hence we prefer the more general term "tools" over "products", which implies a specific class of tools with added assumptions and incentives.

Where the gaps are

Some areas, like security, have been reasonably well populated with commercial as well as OSS tools. However, there are some clearly missing areas, where tools are either too few, too siloed or too early. Some of the important ones are:

• Tools for embedded evaluators to monitor frontier LLMs
• Tools for multi-agent safety
• Tools that make interpretability libraries work together
• Incident, flaw and vulnerability reporting tools
• CBRN (chemical, biological, radiological and nuclear) monitoring tools
• Agent standards of various kinds
• Tools to detect and mitigate harms to vulnerable groups
• Tools for epistemic and human-AI codependency risks
• AI risk underwriting and insurance
• Tools around digital minds (and welfare)

What is being done now?

The past few months have seen some focus, albeit indirect, on AI safety tools and AI safety for builders:

• A dedicated org called Neolithic launched - focussed on building tools to scale AI safety research.
• Generalist- and fieldbuilder-focused streams and programmes, such as Generator Residency, the MATS and ERA fieldbuilding streams, BlueDot's incubator week, and Gen Stream for generalists.
• VCs like Halcyon, Juniper, SAIF.vc, Babuschkin Ventures, Fifty Years, Mythos and Lionheart.
• Incubators like Seldon Lab, Surplus and Catalyze Impact.
• Coefficient Giving (via the newly launched Tailwind initiative) is interested to fund ideas around AI uplift for AI safety. They say “investing in tools and best practices for using AI to accelerate AI safety and governance is a public good that individual contributors often lack capacity for.”
• We're also piloting a small AI Safety for Builders fellowship; see more below.

Some considerations and likely pitfalls

• Dual use is clearly the most important concern here, as with anything impactful in AI safety. Tools that can accelerate AI safety can often accelerate AI capabilities or misuse too.
• Too generic. In the quest to be generalizable, tools can end up becoming too generic (although this can be partially addressed by expanding the canon of tools to platforms, pipelines, protocols, workflows, etc.).
• Built by the labs first. There is a concern that most platform- or tooling-shaped safety work gets built by default by the frontier labs and the big agent frameworks faster than an outside org can ship equivalents. Ronak Mehta who was building Coordinal (an automated safety research platform), recently wrote about lessons from its postmortem.
• Or it is simply too hard, like the recent experience with automated AI alignment research.
• Product pitfalls. Then there are other pitfalls for AI safety tools that are products, briefly touched on above:
  • Poor identification of who will use it, and how?
  • Getting pulled sideways, away from AI safety as the main concern.

AI Safety for Builders

We recently won a Manifund grant to pilot an AI Safety for Builders micro-fellowship. It's free and online: 4 weeks at 8–12 hrs/week, with teams of 2–5 building open-source AI safety tools on projects scoped by mentors. We expect to start within the next 3 months.

We're currently open to expressions of interest from mentors. Please get in touch by filling in the form here, or write to team@aisafety.builders.

1. ^{^}

   Data compiled using an auto-update loop which uses Exa as the primary search engine. Updated till mid-October 2026 to the best of our knowledge, but there may be errors.

2. ^{^}

   Partial means the programme supports building in some form, but its main output isn't AI safety tools.