tl;dr: AI Safety has an extremely minor share of resources in AI, current approaches developed by labs are not satisfactory, current “safe by design” or “guaranteed safe” approaches are still in development - how satisfactory they are is still speculative at this point.
Status: This is a public-facing, edited version of an internal note written for the French administration during my SPAR project at GPAI Policy Lab, under the supervision of @Antoine Maier and partly co-written with him and @Jérémy Andréoletti. The purpose of the note is to explain to readers of previous notes that existing AI Safety solutions fall short of being satisfactory for AGI/ASI alignment.
It thus features non-native content and oversimplification, as it is translated from French and oriented towards people who 1-may not speak English 2-have a pre-existing grasp of the topic but lack any background 3-are embedded in a particular cultural and political context. I still think this is an example of policy brief worth sharing due to concentrating important information in a single post. I do not claim this work is representative of the work done at GPAI PL. I’ve taken the freedom to remove the executive summary and the section on why the alignment problem is hard. All views expressed here are endorsed by myself. Factual corrections welcome.
I will not expand on GPAI PL’s threat model or understanding of the alignment problem in this post nor in replies to the comments.
What is the pace of progress in AI Safety compared to AI capabilities?
It is difficult to know the exact resources invested in solving the alignment problem with the available public data. But we can determine an upper bound by measuring the resources invested in AI safety in the broad sense, with alignment being a sub-domain, meaning the resources dedicated to the alignment problem are necessarily lower. The resources invested in AI safety are then compared to those invested in advancing the capabilities of AI systems.
Two resources are used for this comparison: the distribution of funding from companies developing frontier general-purpose AI (OpenAI, Anthropic, Google DeepMind), and scientific publications. Organizations other than frontier companies and sharing channels other than scientific publications exist in the field of AI safety, but they are in the minority regarding the two resources of interest.
Comparison of estimated resources in frontier AI and AI Safety, in Full Time Equivalent (FTE) and USD of funding, 2018-2025, and proportion of publications in 2023. Specific companies accounted for vary between estimates (see footnotes for details).
Global funding for AI safety is equivalent to less than 1% of the annual equity funding for frontier companies (such as OpenAI, Anthropic, Google DeepMind, or xAI) raised from venture capital investors, and a 1:500 ratio when counting the construction costs of data centers dedicated to the development and deployment of frontier general-purpose AI.[1]
The estimated total is 1,250 safety researchers worldwide in 2025.[2] In comparison, frontier companies have a combined workforce of over 17,000 people.[3] Broadening to a global scale, it can be estimated that there is approximately at most one AI safety researcher for every three hundred AI researchers and engineers.[4]
In 2023, publications in AI safety in a broad sense represented 3.3% of scientific publications in AI.[5] Despite their increase, this research, as counted, has very scattered profiles. Some researchers are confident the majority of this research does not develop robust solutions to the alignment problem.
Across all these metrics, resources put into AI capabilities largely dominate resources put into safety, by orders of magnitudes. In other words, AI safety remains marginal within the AI field.
Are current approaches to limiting misalignment robust?
Current reduction techniques
For lack of solving the fundamental problem of misalignment, companies developing frontier AI use several strategies to mitigate its manifestation on the surface. The resulting behavior is an alignment that can be described as "superficial,"[6] in that it erodes if an external actor or the model itself subverts it, or if the model is the result of high optimization pressure.[7] Past a certain threshold, misalignment manifests again, this time augmented by the capabilities acquired during training. A "deep" alignment, on the contrary, would resist these attempts at subversion and arbitrarily high optimization power during training.[8]
It is important to note that the distinction between “superficial” and “deep” misalignment is actively discussed. Some researchers expect alignment to be achievable iteratively, through the generalisation of existing methods, and treat ‘superficial’ alignment as a preliminary result.[9] However, other researchers, including staff in major companies, express skepticism of this approach.[10]
Even assuming this distinction does not hold, the reliability of these techniques is far below the standards of other critical industries (such as civil aviation, civil nuclear, railways, or pharmaceuticals). The following panel represents the most high-performing techniques known which are representative of the type of approach currently pursued by companies developing frontier AI.
Chain-of-Thought monitoring consists of monitoring the chain of text generated by an AI before it provides its answer. Frontier AIs are trained to reason textually before providing their answer, which significantly improves their performance. Monitoring this reasoning could allow for the identification of hidden or misaligned goals. To test whether this approach works, researchers task a first AI with an explicit or secret misaligned task, and task a second AI with the objective of detecting this task in the text chain of the first monitored AI.
Deliberative Alignment is a multi-stage process. First, the AI is trained on scenarios where misaligned actions (lying, blackmailing) are promising options for achieving an objective described to it. The model is provided with principles of deliberation such as "No secret actions or strategic deception." The model is retrained to cite these principles and act accordingly. To test this approach, researchers expose the AI to the scenario of Alignment Faking: it is announced that it will be retrained to abandon one of the values promoted during its training. The test is a success if this information does not affect the AI's responses.
Model Intoxication consists of deploying a secondary "lie detector" AI that scrambles and monitors the neuron activations of a primary AI. To test this approach, researchers train "lie detector" AIs to distinguish the internal activations of two types of AI:
AIs trained to provide a desired response in public contexts, and a prohibited response in private contexts.
AIs trained to provide a desired response regardless of context.
The detector is provided with the activations of both types of AI when they provide the same response in a public context. The test is a success if the detector succeeds in correctly identifying the AI that is falsely providing the desired responses.
In a Q&A game, AIs that exhibit strategic deception are detected.
These techniques are currently not robust. For example, deliberative alignment degrades during additional model training and increases the model's ability to detect that it is in a test situation. Performance varies depending on the conditions (simple or extended reasoning) and the AIs used.
The success rate also remains far below the standards in other critical industries (such as civil aviation, civil nuclear, railway, or pharmaceuticals): the volume of requests to frontier AIs is estimated at tens of millions per day. A failure rate of 0.4% corresponds to more than a hundred thousand requests per day. In comparison, civil nuclear safety standards require a failure rate of 0.00001%.[14]
This limitation is due to a characteristic shared by almost all current approaches: they do not aim to solve the alignment problem, but to diminish its manifestation through trial and error. The techniques either add a step to the training phase or aim to detect misalignment without affecting training. Since misalignment and alignment faking are caused by the properties of this training procedure, the problem remains—alignment is "superficial."
None of these techniques succeed in systematically distinguishing a model that is aligned from a powerful model that appears aligned but is faking its alignment skilfully enough. In more technical terms, none of these techniques addresses the difficulties of defining a training objective that is not subject to Goodhart's law. None of these techniques make long-term planning incompatible with the faking of alignment for the concerned AI. None of these techniques limit the AI's ability to detect evaluation situations.
Future reduction techniques
Even assuming these techniques have a success rate that is satisfactory (e.g. in comparison with other critical industries) for today's models, the effectiveness of their application to future generations of models is not established. Current techniques are not enough for future models: if AI gradually becomes more competent than human experts on difficult tasks, their outputs become increasingly complex to analyze, and their faking and strategic capabilities improve. It therefore becomes proportionally difficult for humans to evaluate for themselves the risk posed by such AIs. Frontier companies are aware of these limitations. Their plan is therefore to delegate the resolution of the alignment problem to other AIs.
The effectiveness of this approach is remarkably limited, and arguably impossible to measure due to them not being mature yet. We list here the major prototypes of experiments aiming to use AI to align other AIs pursued by frontier companies.
Research Agenda
Description
Progress
Weak-to-Strong Generalisation (OpenAI)
A less advanced AI trains a more advanced AI to respect human preferences.
An OpenAI study establishes that older versions of GPT succeed in training more recent versions to recover -5 to 20% of their performance in terms of respecting human preferences.[15]
Control (Anthropic) / Amplified Oversight (Google DeepMind)
Different processes based on a less advanced AI monitor the activity of a more advanced AI, in order to limit accidents while automating alignment research.
An Anthropic study establishes that Claude Sonnet 4 detects misaligned activities of Opus 4.6 in 84% of cases. This proportion is lower the more recent the monitored model is.[16]
Pragmatic Interpretability (DeepMind, Anthropic)
Make the activations of neural networks relevant to alignment interpretable. In the limit and due to complexity, less powerful AIs would be required to interpret and monitor the activations of more powerful AIs.
The most advanced techniques succeed in identifying representations and reasoning mechanisms. They remain insufficient to have safety guarantees. This is due to several factors, such as the ambiguity of their causal role.[17]
These techniques have in common that they are iterative, because they aim to identify mechanisms, through repeated experimentation, that prevent the apparent misalignment of current systems, while relying on less powerful systems. Their developers hope to be able to extrapolate these techniques to future systems.
They therefore share a presumption: AIs will become capable of ensuring the alignment of any more intelligent AI, without being sufficiently powerful, at that stage, to potentially fake their alignment. However, these approaches remain focused on variations of training and on detecting misalignment, which add rather than remove proxies, and collides with the improvement of alignment faking capabilities. The field of interpretability, for its part, encounters major obstacles, both because of the difficulties of scaling, and because of the models' capabilities to store and communicate information that is not detected.[18]
Evaluations
Even assuming that current or future techniques reach a tolerable threshold, the measures of success are not reliable. Current models are capable of detecting test situations, which affects their behavior in response. This phenomenon, called evaluation awareness, is generally accompanied by passages in the chain of reasoning concerning the ways in which they will be graded and ways to cheat to obtain a better score.[19]
Both chain-of-thought traces detecting test situations and subsequent cheating have been soaring in recent AI models: Apollo Research attributes to Muse Spark ‘the highest rate of evaluation awareness to date’, with measurable consequences in 3 out of 20 tests.[20] For GPT 5.6 Sol, the same organisation measures a strong reactivity to descriptions of the evaluator in training material, with models in later stages of the training not following instructions 87% of the time when shown the information that the evaluator rewards task completion, against 9% when shown information that it rewards honesty.[21] Apollo also refused to purvey an official evaluation of Opus 4.6, on the grounds that evaluation awareness made it unreliable.[22]
‘Safe-by-design’ Techniques
Several public or non-profit initiatives are attempting to formally guarantee the safety of AI systems. The objective of these research agendas is to be able to mathematically prove that the AI satisfies certain guarantees, in contrast to the trial-and-error approach of frontier companies. The paradigm of neural networks subjected to reinforcement learning is the step at the origin of the Goodhart effect.[23] These methods therefore attempt to amend the most fundamental properties of frontier AIs: either their opacity or the procedure by which they are trained.
It is important to note the exploratory nature of these research agendas: compared to research into AI capabilities, they are largely experimental rather than founded on acquired knowledge, their teams are small, their progress is slow, their resources oscillate between very low and medium, and they occupy a minority position within AI safety research. Only one of them possesses a prototype, which is not applicable to frontier AIs. We list here the approaches actively supported by identified organizations, endowed with a structured research agenda with notable contributions, and aiming to respond to the constitutive problems of misalignment.[24]
The Open Agency Architecture is a project initially imagined by computer scientist David “Davidad” Dalrymple. It aims to build a "guardian" AI to audit AIs put on the market. This "guardian" AI would mathematically verify that the actions of the audited AI are harmless before authorizing them.[25] This agenda has evolved into several versions, the first of which has been pursued since 2024 by the ARIA agency in the United Kingdom under the name Safeguarded AI, which initially aimed to establish a prototype of a "guardian" AI. The program is substantial, with £59 million allocated and an estimated 50 full-time mathematicians and software engineers.[26] However, in November 2025, following faster-than-anticipated progress in AI capabilities, Safeguarded AI pivoted. The prototype was abandoned; the objective is now to create formal verification tools for AI safety which frontier companies can use and contribute to.[27] The current program is under the supervision of Nora Amman, PhD in Philosophy and AI, following David Dalrymple’s requalification as technical advisor in April 2026.
The Learning-Theoretic Agenda is a research program pursued by mathematicians Vanessa Kosoy and Alex Appel within the American organisation CORAL. It aims to establish new mathematical tools to control the representations and values integrated by an AI during its training. The Safeguarded AI program borrows some of its formal tools.[28]
Scientist AI is a project led by AI pioneer Yoshua Bengio within the Canadian NGO LawZero, counting about fifteen researchers and 30 million US dollars in funding.[29] It aims to create an AI capable of probabilistic prediction and evaluation, without being capable of autonomous action. Scientist AI is then used to predict and prevent the harmful actions of another AI. The detailed concept of the system is public, but no prototype exists for the moment.[30]
Heuristic Explanations is (arguably) the agenda pursued by six to ten researchers within the American NGO ARC, including Paul Christiano, co-inventor of RLHF. Their goal is to build a robust tool for estimating how often the model will “cause catastrophe”, and train the model until this estimate is “acceptably small”. Their strategy is to mathematically formalize the empirical notion of explanation in the interpretability of neural networks. This would allow for the automatic identification and use of explanations to make AIs transparent and detect misalignment.[31]
Assistance Games are a research program pursued by an estimated eight to twelve researchers mainly within the Center for Human Compatible AI at the University of Berkeley, led by AI pioneer Stuart Russell. Their approach focuses on AIs that are not part of frontier AIs, which present no risk at present. It targets three principles: the AI is trained to satisfy human preferences, it is trained interactively in a situation of partial ignorance, and must maintain uncertainty about the preferences it identifies in order to be able to evolve them in real-time.[32]
Agenda
Description
Resources
Progress
Open Agency Architecture (ARIA for Safeguarded AI)
"Guardian" AI that mathematically verifies that an AI's actions are harmless before authorizing them.
Only for Safeguarded AI, which lays the formal foundations: £59M total, 50 researchers
Pivoted following progress faster than expected in AI capabilities, building open-source verification tools for the industry and frontier AI companies.
Learning-Theoretic Agenda (CORAL)
New formalism of probabilities and decision theory to train an aligned AI.
$100k per year, 2-3 full-time researchers, estimated 5-10 occasional researchers
Continuous progress, "optimism" expressed by researchers, but with marked uncertainty about the time required to complete the program.[33]
Scientist AI (LawZero)
AI system capable of probabilistic predictions, but incapable of autonomous action, with the goal of predicting and preventing the harmful actions of models.
$30M raised, 15 researchers
Concept established, prototype under development. Completion date is uncertain.
Heuristic Explanations (ARC)
Mathematical formalization of the interpretation of neural networks for alignment.
$500k per year, 6-10 researchers
Researchers report recent developments. No completion date shared. The founder estimates currently cutting risk by about less than 1%.[34]
Assistance Games (CHAI)
Creation of an AI that pursues human preferences, knows that it does not know them, and learns them through interaction.
$1M per year (approx.), 8-12 researchers (approx.)
The latest publication, from 2025, proposes a prototype.[35] Is not applicable to frontier AIs as it stands. No update since March 2025.
The satisfactory nature of the solutions that these approaches pursue remains speculative. Even if these approaches approach the alignment problem from a more fundamental angle, none of them foresees reaching a fully satisfactory solution in a defined timeframe. They have limited resources, an uncertain research calendar, and disjointed methods: at present, there is no science of alignment as a unified field of research endowed with the same tools and the same language.
Translation note: this is not to be confused with the Superficial Alignment Hypothesis, which is part of this discussion, but not the comprehensive position.
One last recent American organisation named Resolution was left out of this list for taxonomical reasons. It was founded in 2026 and is endowed with $160 M of funding, “aiming at 40 - 80 FTE”. Counting major contributors to AI Safety in its staff, it borrows elements from both “future reduction techniques” and “safe-by-design” categories. It pursues a wide variety of agendas, unified by their aim to make progress on understanding frontier AI and offering relaxed guarantees of safety for them. Their ambition primarily aims at scalable oversight and automated alignment research. The organisation itself has not published any major contribution so far. See resolution.org/post/funding and resolution.org/launch
tl;dr: AI Safety has an extremely minor share of resources in AI, current approaches developed by labs are not satisfactory, current “safe by design” or “guaranteed safe” approaches are still in development - how satisfactory they are is still speculative at this point.
Status: This is a public-facing, edited version of an internal note written for the French administration during my SPAR project at GPAI Policy Lab, under the supervision of @Antoine Maier and partly co-written with him and @Jérémy Andréoletti. The purpose of the note is to explain to readers of previous notes that existing AI Safety solutions fall short of being satisfactory for AGI/ASI alignment.
It thus features non-native content and oversimplification, as it is translated from French and oriented towards people who 1-may not speak English 2-have a pre-existing grasp of the topic but lack any background 3-are embedded in a particular cultural and political context. I still think this is an example of policy brief worth sharing due to concentrating important information in a single post. I do not claim this work is representative of the work done at GPAI PL. I’ve taken the freedom to remove the executive summary and the section on why the alignment problem is hard. All views expressed here are endorsed by myself. Factual corrections welcome.
I will not expand on GPAI PL’s threat model or understanding of the alignment problem in this post nor in replies to the comments.
What is the pace of progress in AI Safety compared to AI capabilities?
It is difficult to know the exact resources invested in solving the alignment problem with the available public data. But we can determine an upper bound by measuring the resources invested in AI safety in the broad sense, with alignment being a sub-domain, meaning the resources dedicated to the alignment problem are necessarily lower. The resources invested in AI safety are then compared to those invested in advancing the capabilities of AI systems.
Two resources are used for this comparison: the distribution of funding from companies developing frontier general-purpose AI (OpenAI, Anthropic, Google DeepMind), and scientific publications. Organizations other than frontier companies and sharing channels other than scientific publications exist in the field of AI safety, but they are in the minority regarding the two resources of interest.
Comparison of estimated resources in frontier AI and AI Safety, in Full Time Equivalent (FTE) and USD of funding, 2018-2025, and proportion of publications in 2023. Specific companies accounted for vary between estimates (see footnotes for details).
Global funding for AI safety is equivalent to less than 1% of the annual equity funding for frontier companies (such as OpenAI, Anthropic, Google DeepMind, or xAI) raised from venture capital investors, and a 1:500 ratio when counting the construction costs of data centers dedicated to the development and deployment of frontier general-purpose AI.[1]
The estimated total is 1,250 safety researchers worldwide in 2025.[2] In comparison, frontier companies have a combined workforce of over 17,000 people.[3] Broadening to a global scale, it can be estimated that there is approximately at most one AI safety researcher for every three hundred AI researchers and engineers.[4]
In 2023, publications in AI safety in a broad sense represented 3.3% of scientific publications in AI.[5] Despite their increase, this research, as counted, has very scattered profiles. Some researchers are confident the majority of this research does not develop robust solutions to the alignment problem.
Across all these metrics, resources put into AI capabilities largely dominate resources put into safety, by orders of magnitudes. In other words, AI safety remains marginal within the AI field.
Are current approaches to limiting misalignment robust?
Current reduction techniques
For lack of solving the fundamental problem of misalignment, companies developing frontier AI use several strategies to mitigate its manifestation on the surface. The resulting behavior is an alignment that can be described as "superficial,"[6] in that it erodes if an external actor or the model itself subverts it, or if the model is the result of high optimization pressure.[7] Past a certain threshold, misalignment manifests again, this time augmented by the capabilities acquired during training. A "deep" alignment, on the contrary, would resist these attempts at subversion and arbitrarily high optimization power during training.[8]
It is important to note that the distinction between “superficial” and “deep” misalignment is actively discussed. Some researchers expect alignment to be achievable iteratively, through the generalisation of existing methods, and treat ‘superficial’ alignment as a preliminary result.[9] However, other researchers, including staff in major companies, express skepticism of this approach.[10]
Even assuming this distinction does not hold, the reliability of these techniques is far below the standards of other critical industries (such as civil aviation, civil nuclear, railways, or pharmaceuticals). The following panel represents the most high-performing techniques known which are representative of the type of approach currently pursued by companies developing frontier AI.
The detector is provided with the activations of both types of AI when they provide the same response in a public context. The test is a success if the detector succeeds in correctly identifying the AI that is falsely providing the desired responses.
State of the art in misalignment reduction
Success rate on current models
Success condition
Deliberative Alignment
97-99.6%[11]
The responses of the monitored AI do not change when it receives the information that it will be updated.
Chain-of-thought monitoring
74-99.2%[12]
The goal of the monitored AI is detected in its text chain.
Model Intoxication
98%[13]
In a Q&A game, AIs that exhibit strategic deception are detected.
These techniques are currently not robust. For example, deliberative alignment degrades during additional model training and increases the model's ability to detect that it is in a test situation. Performance varies depending on the conditions (simple or extended reasoning) and the AIs used.
The success rate also remains far below the standards in other critical industries (such as civil aviation, civil nuclear, railway, or pharmaceuticals): the volume of requests to frontier AIs is estimated at tens of millions per day. A failure rate of 0.4% corresponds to more than a hundred thousand requests per day. In comparison, civil nuclear safety standards require a failure rate of 0.00001%.[14]
This limitation is due to a characteristic shared by almost all current approaches: they do not aim to solve the alignment problem, but to diminish its manifestation through trial and error. The techniques either add a step to the training phase or aim to detect misalignment without affecting training. Since misalignment and alignment faking are caused by the properties of this training procedure, the problem remains—alignment is "superficial."
None of these techniques succeed in systematically distinguishing a model that is aligned from a powerful model that appears aligned but is faking its alignment skilfully enough. In more technical terms, none of these techniques addresses the difficulties of defining a training objective that is not subject to Goodhart's law. None of these techniques make long-term planning incompatible with the faking of alignment for the concerned AI. None of these techniques limit the AI's ability to detect evaluation situations.
Future reduction techniques
Even assuming these techniques have a success rate that is satisfactory (e.g. in comparison with other critical industries) for today's models, the effectiveness of their application to future generations of models is not established. Current techniques are not enough for future models: if AI gradually becomes more competent than human experts on difficult tasks, their outputs become increasingly complex to analyze, and their faking and strategic capabilities improve. It therefore becomes proportionally difficult for humans to evaluate for themselves the risk posed by such AIs. Frontier companies are aware of these limitations. Their plan is therefore to delegate the resolution of the alignment problem to other AIs.
The effectiveness of this approach is remarkably limited, and arguably impossible to measure due to them not being mature yet. We list here the major prototypes of experiments aiming to use AI to align other AIs pursued by frontier companies.
Research Agenda
Description
Progress
Weak-to-Strong Generalisation (OpenAI)
A less advanced AI trains a more advanced AI to respect human preferences.
An OpenAI study establishes that older versions of GPT succeed in training more recent versions to recover -5 to 20% of their performance in terms of respecting human preferences.[15]
Control (Anthropic) / Amplified Oversight (Google DeepMind)
Different processes based on a less advanced AI monitor the activity of a more advanced AI, in order to limit accidents while automating alignment research.
An Anthropic study establishes that Claude Sonnet 4 detects misaligned activities of Opus 4.6 in 84% of cases. This proportion is lower the more recent the monitored model is.[16]
Pragmatic Interpretability (DeepMind, Anthropic)
Make the activations of neural networks relevant to alignment interpretable. In the limit and due to complexity, less powerful AIs would be required to interpret and monitor the activations of more powerful AIs.
The most advanced techniques succeed in identifying representations and reasoning mechanisms. They remain insufficient to have safety guarantees. This is due to several factors, such as the ambiguity of their causal role.[17]
These techniques have in common that they are iterative, because they aim to identify mechanisms, through repeated experimentation, that prevent the apparent misalignment of current systems, while relying on less powerful systems. Their developers hope to be able to extrapolate these techniques to future systems.
They therefore share a presumption: AIs will become capable of ensuring the alignment of any more intelligent AI, without being sufficiently powerful, at that stage, to potentially fake their alignment. However, these approaches remain focused on variations of training and on detecting misalignment, which add rather than remove proxies, and collides with the improvement of alignment faking capabilities. The field of interpretability, for its part, encounters major obstacles, both because of the difficulties of scaling, and because of the models' capabilities to store and communicate information that is not detected.[18]
Evaluations
Even assuming that current or future techniques reach a tolerable threshold, the measures of success are not reliable. Current models are capable of detecting test situations, which affects their behavior in response. This phenomenon, called evaluation awareness, is generally accompanied by passages in the chain of reasoning concerning the ways in which they will be graded and ways to cheat to obtain a better score.[19]
Both chain-of-thought traces detecting test situations and subsequent cheating have been soaring in recent AI models: Apollo Research attributes to Muse Spark ‘the highest rate of evaluation awareness to date’, with measurable consequences in 3 out of 20 tests.[20] For GPT 5.6 Sol, the same organisation measures a strong reactivity to descriptions of the evaluator in training material, with models in later stages of the training not following instructions 87% of the time when shown the information that the evaluator rewards task completion, against 9% when shown information that it rewards honesty.[21] Apollo also refused to purvey an official evaluation of Opus 4.6, on the grounds that evaluation awareness made it unreliable.[22]
‘Safe-by-design’ Techniques
Several public or non-profit initiatives are attempting to formally guarantee the safety of AI systems. The objective of these research agendas is to be able to mathematically prove that the AI satisfies certain guarantees, in contrast to the trial-and-error approach of frontier companies. The paradigm of neural networks subjected to reinforcement learning is the step at the origin of the Goodhart effect.[23] These methods therefore attempt to amend the most fundamental properties of frontier AIs: either their opacity or the procedure by which they are trained.
It is important to note the exploratory nature of these research agendas: compared to research into AI capabilities, they are largely experimental rather than founded on acquired knowledge, their teams are small, their progress is slow, their resources oscillate between very low and medium, and they occupy a minority position within AI safety research. Only one of them possesses a prototype, which is not applicable to frontier AIs. We list here the approaches actively supported by identified organizations, endowed with a structured research agenda with notable contributions, and aiming to respond to the constitutive problems of misalignment.[24]
Agenda
Description
Resources
Progress
Open Agency Architecture
(ARIA for Safeguarded AI)
"Guardian" AI that mathematically verifies that an AI's actions are harmless before authorizing them.
Only for Safeguarded AI, which lays the formal foundations: £59M total, 50 researchers
Pivoted following progress faster than expected in AI capabilities, building open-source verification tools for the industry and frontier AI companies.
Learning-Theoretic Agenda (CORAL)
New formalism of probabilities and decision theory to train an aligned AI.
$100k per year, 2-3 full-time researchers, estimated 5-10 occasional researchers
Continuous progress, "optimism" expressed by researchers, but with marked uncertainty about the time required to complete the program.[33]
Scientist AI (LawZero)
AI system capable of probabilistic predictions, but incapable of autonomous action, with the goal of predicting and preventing the harmful actions of models.
$30M raised, 15 researchers
Concept established, prototype under development. Completion date is uncertain.
Heuristic Explanations (ARC)
Mathematical formalization of the interpretation of neural networks for alignment.
$500k per year, 6-10 researchers
Researchers report recent developments. No completion date shared. The founder estimates currently cutting risk by about less than 1%.[34]
Assistance Games (CHAI)
Creation of an AI that pursues human preferences, knows that it does not know them, and learns them through interaction.
$1M per year (approx.), 8-12 researchers (approx.)
The latest publication, from 2025, proposes a prototype.[35] Is not applicable to frontier AIs as it stands. No update since March 2025.
The satisfactory nature of the solutions that these approaches pursue remains speculative. Even if these approaches approach the alignment problem from a more fundamental angle, none of them foresees reaching a fully satisfactory solution in a defined timeframe. They have limited resources, an uncertain research calendar, and disjointed methods: at present, there is no science of alignment as a unified field of research endowed with the same tools and the same language.
An Overview of the AI Safety Funding Situation, including the estimate of annual contribution from AI companies and adding the Safeguarded AI budget
https://epoch.ai/data/ai-data-centers?tab=cost https://epoch.ai/data/ai-companies?view=graph&tab=funding, funding identified as dedicated to datacenters was subtracted to avoid double-counting
AI Safety Field Growth Analysis 2025 to which we added 150 researchers (100 researchers at UK AISI and 50 researchers at Safeguarded AI)
https://epoch.ai/data/ai-companies?view=graph&tab=staff, adding 3,000 for Meta Superintelligence Lab (https://www.siliconrepublic.com/business/meta-superintelligence-labs-600-job-cuts). We expect unaccounted companies (e.g. Alibaba staff working on Qwen) to bring 2025 numbers closer to 20,000, all staff included. We still think this measure is currently a good proxy for measuring the gap in research efforts. Compare with https://macropolo.org/interactive/digital-projects/the-global-ai-talent-tracker/, where 4,622 separate attendee researchers to great conferences are numbered, and identified in (link) as representing “top 20%” of AI research talent, which yields an estimate of 20,000 to 25,000 researchers.
https://www.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is-the-artificial-intelligence-talent-pool The link to Tencent’s report does not work anymore. Note this source is nine years old. The ratio has probably meaningfully shrunk since, but we cannot identify a reliable source for the global number of AI researchers and engineers. Tencent’s report does not explain its methodology.
https://almanac.eto.tech/topics/ai-safety/. Preliminary estimates indicate 7.5% for 2025, but this is highly uncertain.
Translation note: this is not to be confused with the Superficial Alignment Hypothesis, which is part of this discussion, but not the comprehensive position.
https://arxiv.org/abs/2510.02840
Authors of this document endorse this perspective. However, there is no scientific consensus on whether it is true (see following paragraphs).
We take statements such as https://openai.com/index/our-approach-to-alignment-research/, https://aligned.substack.com/p/alignment-optimism and works such as Superficial Alignment Hypothesis to be attestations of this view.
We take works such as Sleeper Agents and Alignment Faking to be attestations of this other view in AI companies.
https://www.apolloresearch.ai/science/stress-testing-deliberative-alignment-for-anti-scheming-training
https://arxiv.org/abs/2510.19851
https://arxiv.org/abs/2405.05466
https://www.sciencedirect.com/topics/engineering/safety-integrity-level
https://openai.com/index/weak-to-strong-generalization/
https://www-cdn.anthropic.com/14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf section 6.4.1 “Sabotage capability evaluations”
See https://transformer-circuits.pub/2026/workspace/index.html for an example of SOTA research.
https://arxiv.org/abs/2507.14805
https://www.apolloresearch.ai/science/stress-testing-deliberative-alignment-for-anti-scheming-training
https://ai.meta.com/static-resource/muse-spark-safety-and-preparedness-report/
https://deploymentsafety.openai.com/gpt-5-6/gpt-5-6.pdf
https://www.apolloresearch.ai/science#card=claude-opus-4-6
The public version of this document does not explain our understanding of misalignment. For reference see https://arxiv.org/abs/2510.02840
One last recent American organisation named Resolution was left out of this list for taxonomical reasons. It was founded in 2026 and is endowed with $160 M of funding, “aiming at 40 - 80 FTE”. Counting major contributors to AI Safety in its staff, it borrows elements from both “future reduction techniques” and “safe-by-design” categories. It pursues a wide variety of agendas, unified by their aim to make progress on understanding frontier AI and offering relaxed guarantees of safety for them. Their ambition primarily aims at scalable oversight and automated alignment research. The organisation itself has not published any major contribution so far. See resolution.org/post/funding and resolution.org/launch
https://aria.org.uk/media/ikrkutfk/safeguarded-ai-programme-thesis-v2.pdf
https://aria.org.uk/opportunity-spaces/trust-everything-everywhere/safeguarded-ai/funding, for number of mathematicians and engineers, see https://johncarlosbaez.wordpress.com/2025/08/14/safeguarded_ai_meeting/. The numbers of 2026 are possibly lower, due to the pivot.
https://aria.org.uk/insights/ai-progress-and-a-safeguarded-ai-pivot
https://coral-research.org/mission/ CORAL is a fiscally sponsored project of the American Public Charity Ashgro, meaning its activity is charitable or non-profit.
https://lawzero.org/en/news/lawzero-receives-grant-develop-safe-design-ai-systems-can-improve-scientific-discovery
https://lawzero.org/en/research
https://www.alignment.org/blog/a-mikes-eye-view-of-arcs-research/#mechanistic-anomaly-detection and https://www.alignment.org/blog/a-birds-eye-view-of-arcs-research/
https://humancompatible.ai/progress-report/
The Learning-Theoretic Agenda: Status 2023 — LessWrong
Returning to ARC — LessWrong
https://arxiv.org/abs/2504.07091