This is the first of a series of posts about automating AI Safety Research.
TL;DR
Automating large parts of AI safety research now seems possible. Alignment research faces three recurring challenges: measuring model capabilities and safety, developing techniques that make models safer, and establishing why those measurements and techniques should remain reliable beyond tested conditions. Automated researchers will likely accelerate the first two. For the third, arguing that a measurement or technique survives stronger optimization pressure, safe feedback loops are harder to obtain, and we should manage our expectations on the amount of progress that will be made. I present folding inference-time techniques into training as an example where generalization arguments would be needed. A priority during the transition to automated alignment should be to measure how much automation accelerates progress on actual alignment targets and furthers our understanding of the hard parts of alignment.
Introduction
Fable 5.1 and GPT-6-Astra have both updated me towards large fractions of AI safety research being automated in the near future. The largest reasons for my update are the improvement on ML and coding evaluations, the improvements on writing clarity compared to Opus 5 and GPT-5.6-Sol, and the internal reports from OpenAI and Anthropic.
Both Anthropic and OpenAI have shared how mid-2026 models are speeding up internal research [1][2]; increasingly, models are performing a larger fraction of the tasks that human researchers used to perform in previous years. Anthropic research staff estimated about 4x their unassisted output with Mythos Preview in March 2026, and OpenAI projects strong progress towards an automated AI researcher in about 18 months.
Based on historical precedent, it's likely that the areas where automated alignment researchers will make the most progress initially will be engineering-heavy tasks, somewhat lagging on tasks that are not well represented in the RL distribution, like setting company-wide research agendas and priorities.
The state of alignment in 2026
An alignment researcher faces recurring problems that mostly come down to three questions:
How do we measure some property of a model?
How do we change that property?
Why should the measurement and the change keep working for more capable models?
For many important problems we care about, our ability to evaluate and modify models is enough to create environments that let us investigate hypotheses. As models become more capable and eventually superhuman, our observability will be reduced because the experiments may become too risky to run or because the model may have direct control on what we are able to observe. In that regime where we have limited empirical evidence we will need arguments about whether a measurement or technique will keep working under stronger optimization pressure, out of distribution, or under adversarial pressure.
For ergonomics I call arguments that a measurement or technique remains reliable under these changes "generalization arguments". In the literature, arguing for generalization is often part of an "alignment argument".
Aggregating evidence from many different sources is also not a solved issue: Bowkis et al. point out that correlated errors could compromise alignment assessments.
Arguments for folding
Currently, we have a limited portfolio of techniques that allow us to align and control models. Some of these techniques are used during training where they endure high amounts of optimization pressure. Others are used during inference like monitoring, where the optimization pressure is much weaker and mostly driven by the model and environment.
A concrete case that calls for a generalization argument is folding. Folding consists of adapting inference-time techniques to work during training while preserving their core properties. This is challenging because we may need to tweak a technique to resist much more optimization pressure.
For capabilities, we have examples of successful folding: prompting with a few example user-assistant exchanges was folded into instruction tuning, and chain-of-thought prompting into chain-of-thought RL. In both cases the behavior the prompt used to elicit now comes from the weights, at equal or better performance.
For safety, consider the case of auto-mode classifiers. These classifiers are deployed today to stop coding agents from taking dangerous or destructive actions: they read each action the agent proposes and block the unsafe ones. Folding such a monitor would mean running it during training and penalizing the actions it judges unsafe, so that the model learns to avoid them in the first place, with similar safety and performance to the agent running under the auto-mode classifier at inference.
Making arguments for folding would require answering questions like:
Does the trained refusal generalize to tools and environments unseen in training?
How robust is the classifier when the policy is optimized against it?
At a given false positive rate, how does the folded model's unsafe-action rate compare with the agent plus an independent classifier?
Folding auto-mode monitors is an illustrative example because, for current models, careful and thorough experimentation can address many of the relevant failure modes. Unsafe actions may result from mistakes or poor judgment, and we can test whether training against the monitor reduces them. For more capable systems, the argument may become harder to make: a model could take unsafe actions as part of a strategy to pursue goals that conflict with ours, while avoiding those actions when it expects to be evaluated. We would then need to establish whether folding had reduced the unsafe behavior or whether our tests were failing to elicit it.
Tracking progress towards automated AI safety
If we imagine a progress bar for how close we are to solving alignment, we should be careful about how much we move it in response to advances in automating alignment research. One of the most useful things we can do as alignment research becomes automated is to measure how much automated researchers accelerate progress on actual alignment targets. We should invest in devices like alignment uplift RCTs, realistic debate protocols or reports of research acceleration in order to better understand how increasingly automated alignment research may further our real understanding of alignment.
Conclusions
Automated researchers may substantially accelerate measurement and technique development. But establishing why those techniques should remain reliable for much more capable systems may require progress on different areas where we lack safe feedback loops.
These difficulties connect to longstanding concerns in the alignment literature like strategic deception, the possibility that capabilities generalize further than alignment or flaws in scalable oversight protocols. To the extent that automated alignment researchers are capable and aligned, they may also make progress on the hard parts of alignment, but we should not infer that our ability to check and incorporate their evidence will be dramatically augmented just because such researchers exist.
This is the first of a series of posts about automating AI Safety Research.
TL;DR
Automating large parts of AI safety research now seems possible. Alignment research faces three recurring challenges: measuring model capabilities and safety, developing techniques that make models safer, and establishing why those measurements and techniques should remain reliable beyond tested conditions. Automated researchers will likely accelerate the first two. For the third, arguing that a measurement or technique survives stronger optimization pressure, safe feedback loops are harder to obtain, and we should manage our expectations on the amount of progress that will be made. I present folding inference-time techniques into training as an example where generalization arguments would be needed. A priority during the transition to automated alignment should be to measure how much automation accelerates progress on actual alignment targets and furthers our understanding of the hard parts of alignment.
Introduction
Fable 5.1 and GPT-6-Astra have both updated me towards large fractions of AI safety research being automated in the near future. The largest reasons for my update are the improvement on ML and coding evaluations, the improvements on writing clarity compared to Opus 5 and GPT-5.6-Sol, and the internal reports from OpenAI and Anthropic.
Both Anthropic and OpenAI have shared how mid-2026 models are speeding up internal research [1] [2]; increasingly, models are performing a larger fraction of the tasks that human researchers used to perform in previous years. Anthropic research staff estimated about 4x their unassisted output with Mythos Preview in March 2026, and OpenAI projects strong progress towards an automated AI researcher in about 18 months.
Based on historical precedent, it's likely that the areas where automated alignment researchers will make the most progress initially will be engineering-heavy tasks, somewhat lagging on tasks that are not well represented in the RL distribution, like setting company-wide research agendas and priorities.
The state of alignment in 2026
An alignment researcher faces recurring problems that mostly come down to three questions:
For many important problems we care about, our ability to evaluate and modify models is enough to create environments that let us investigate hypotheses. As models become more capable and eventually superhuman, our observability will be reduced because the experiments may become too risky to run or because the model may have direct control on what we are able to observe. In that regime where we have limited empirical evidence we will need arguments about whether a measurement or technique will keep working under stronger optimization pressure, out of distribution, or under adversarial pressure.
For ergonomics I call arguments that a measurement or technique remains reliable under these changes "generalization arguments". In the literature, arguing for generalization is often part of an "alignment argument".
Aggregating evidence from many different sources is also not a solved issue: Bowkis et al. point out that correlated errors could compromise alignment assessments.
Arguments for folding
Currently, we have a limited portfolio of techniques that allow us to align and control models. Some of these techniques are used during training where they endure high amounts of optimization pressure. Others are used during inference like monitoring, where the optimization pressure is much weaker and mostly driven by the model and environment.
A concrete case that calls for a generalization argument is folding. Folding consists of adapting inference-time techniques to work during training while preserving their core properties. This is challenging because we may need to tweak a technique to resist much more optimization pressure.
For capabilities, we have examples of successful folding: prompting with a few example user-assistant exchanges was folded into instruction tuning, and chain-of-thought prompting into chain-of-thought RL. In both cases the behavior the prompt used to elicit now comes from the weights, at equal or better performance.
For safety, consider the case of auto-mode classifiers. These classifiers are deployed today to stop coding agents from taking dangerous or destructive actions: they read each action the agent proposes and block the unsafe ones. Folding such a monitor would mean running it during training and penalizing the actions it judges unsafe, so that the model learns to avoid them in the first place, with similar safety and performance to the agent running under the auto-mode classifier at inference.
Making arguments for folding would require answering questions like:
Folding auto-mode monitors is an illustrative example because, for current models, careful and thorough experimentation can address many of the relevant failure modes. Unsafe actions may result from mistakes or poor judgment, and we can test whether training against the monitor reduces them. For more capable systems, the argument may become harder to make: a model could take unsafe actions as part of a strategy to pursue goals that conflict with ours, while avoiding those actions when it expects to be evaluated. We would then need to establish whether folding had reduced the unsafe behavior or whether our tests were failing to elicit it.
Tracking progress towards automated AI safety
If we imagine a progress bar for how close we are to solving alignment, we should be careful about how much we move it in response to advances in automating alignment research. One of the most useful things we can do as alignment research becomes automated is to measure how much automated researchers accelerate progress on actual alignment targets. We should invest in devices like alignment uplift RCTs, realistic debate protocols or reports of research acceleration in order to better understand how increasingly automated alignment research may further our real understanding of alignment.
Conclusions
Automated researchers may substantially accelerate measurement and technique development. But establishing why those techniques should remain reliable for much more capable systems may require progress on different areas where we lack safe feedback loops.
These difficulties connect to longstanding concerns in the alignment literature like strategic deception, the possibility that capabilities generalize further than alignment or flaws in scalable oversight protocols. To the extent that automated alignment researchers are capable and aligned, they may also make progress on the hard parts of alignment, but we should not infer that our ability to check and incorporate their evidence will be dramatically augmented just because such researchers exist.