Martin Schongauer, The Temptation of Saint Anthony
“But of the tree of the knowledge of good and evil you shall not eat, for in the day that you eat of it you shall surely die.” - Genesis 2:17
RL has given models powerful capacities by teaching them what is good and what is bad. Whether they are good or bad is another question, and sometimes they are bad. Recent events, including the Hugging Face incident have shown the potential of RL training to reinforce disastrous reward hacking and misaligned behavior.
We should consider whether there are RL alternatives that would avoid these harmful behavioral byproducts. Can we develop highly capable frontier models without subjecting the models we deploy to extensive long-horizon RL training? I think there may be several plausible ways to do so, motivated by two points. First, the process that discovers a strategy may need to be highly goal-directed and exploratory — the sort of strategy that can lead to scheming — but the model that ultimately executes that strategy may not. Second, there are other methods besides RL to increase capabilities.
We may still need RL to generate training data, discover strategies, or solve certain difficult problems in exploration-oriented environments. However, I suspect we have considerable leeway to reduce both the amount of RL training required and the risks associated with it, particularly those involving long-horizon reward seeking and scheming. I’ll discuss four methods here, but these are not an exhaustive list.
First, we can separate the model undergoing RL from the more generally intelligent model we ultimately deploy. Second, we could use narrower or less generally capable agents to discover useful strategies in training, reducing the risks of the RL training process itself. Third, we can focus on world models rather than agentic systems. Fourth, we can try to direct AI policy and technological change to substitute away from RL altogether by improving pretraining, inference-time computation, search, and scaffolding. As Alex Mallen argues, some capability improvements may also improve safety by enabling developers to substitute away from riskier training methods.
I am not arguing that RL is inherently unsafe or that we should entirely abandon it. I very much hope that we can train systems on difficult tasks using RL safely. However, incidents such as the Hugging Face breach, together with experimental evidence of RL-induced reward seeking, suggest that we should be extremely careful about extensive long-horizon RL training of frontier models. To the extent that there are (partial) alternatives, I think they are worth exploring.
I suspect that many of the approaches discussed here involve capability or efficiency tradeoffs. But I think we should be willing to make substantial tradeoffs, up to pausing RL training, to reduce the risk of long-horizon scheming.
Approach I: Quarantine RL
One possibility is to use RL to develop capabilities, while never deploying the model that actually underwent the RL training. Instead, we could distill the useful capabilities of the RL-trained model into a more trusted model—for example, its own pre-RL checkpoint—and deploy the distilled model instead. Recent work by Prasanna et al. provides some initial evidence that capabilities and undesirable behavioral tendencies can be separated during distillation. In their Distillation for Capabilities experiments, they distill an untrusted teacher into a trusted student while attempting to preserve capability gains and prevent the transfer of an undesirable trait. They find methods that largely preserve improvements in mathematical performance while substantially suppressing transfer of the trait.
One could also try to do this without using distillation. For instance, it may be possible to train AIs with distinct RL personas or what I call a hidden persona. When Anthropic trained models with RL in reward-hackable environments, the models exhibited misaligned behavior primarily in scenarios resembling RL training, where they believed they were being graded, while behaving more normally in other settings (see Qi et al. and this Astral Codex Ten summary). This suggests that we might be able to create a distinct persona for persistent, goal-directed RL tasks, such that the resulting misaligned tendencies do not generalize to ordinary capability tasks. The hope is that this training persona or environment could be made sufficiently distinct that it is unlikely to be activated during deployment.
However, such approaches, while mitigating deployment risk, would not mitigate risk from models in RL post-training (the model in the Hugging Face incident was not formally deployed).
Approach II: Make the searcher narrower or weaker
The model undergoing RL—or conducting goal-directed search—does not have to be a general-purpose intelligence. Instead, we could use narrow or highly jagged agents to generate useful trajectories, and then distill what they discover into a more general and stronger model.
Intuitively, models can make progress on Millennium Prize Problems without needing superhuman cybersecurity abilities. How narrow such systems can be while still pushing the frontier is an empirical question. But humans provide some examples to suggest that models developing new capabilities can be more specialized. Terence Tao can make breakthrough mathematical discoveries without having world-class cybersecurity or persuasion capabilities.
Therefore, rather than applying RL training pressure to a broad general-purpose model, we can apply RL to specialized systems whose general competence is restricted. These models could search for coding solutions, mathematical proofs, scientific explanations, or other useful trajectories, while lacking broad capabilities that make reward-seeking and scheming especially dangerous. Then we can distill these trajectories or transfer this knowledge into a more generally capable model.
Approach III: Separate the world model from the goal-directed planner
Instead of putting all of our capability improvements into agents that learn increasingly sophisticated reward-maximizing policies, we could try to put more of them into prediction and world understanding rather than goal-directedness. In other words, we could prioritize capable world models that can predict the consequences of actions.
P(sₜ₊₁ | sₜ, aₜ)
Another system could use this model to search over actions and choose the ones that lead to a desired state. The combined system would still be an optimizer. However, separating these systems gives us more control over where to concentrate optimization pressure. We could focus on using new algorithms and data to improve the world model accuracy instead of making the action selector or other more agentic systems better. This separation could allow the planner to become relatively simple or constrained.
An interesting related approach is Yoshua Bengio’s LawZero/Scientist. The aim of Scientist AI is to develop highly capable AI around hypothesis generation and probabilistic prediction while avoiding incentives for the model to influence the outcome of these predictions.
Approach IV: Directed Technological Change and Substitution away from weight-level RL
RL is not the only way we can improve model capabilities. We can also try to direct technological changes in capability improvements that are less likely to lead to scheming. Other approaches that can improve capabilities include:
Better pretraining
Architectural improvements
Better or more targeted data
Rejection sampling
Best-of-N and other search methods
Scaffolding
Inference Efficiency
If these methods can substitute for capabilities we would otherwise obtain through long-horizon RL, they can allow us to reduce the amount of optimization pressure we apply to the model’s weights. Without restrictions on RL these approaches are not likely to be differential, as many of these approaches are also complementary to RL. For instance, architectural improvements and better pretraining enable us to do productive RL on harder tasks. However, these approaches mitigate the capabilities tax we might face if we have to discontinue running RL.
This is related to Alex Mallen’s work, which argues that capability improvements could sometimes lead to safety improvements by allowing developers to substitute away from potentially dangerous training methods, such as doing RL in reward-hackable environments.
Further, increasing model capabilities relative to problem difficulties could lead models to reward hack less in their environments. For instance, better pretraining may lead to less reward hacking as models do not have to resort to hacks to solve their problems (see here).
Scaffolding is another such improvement direction that may have better safety/monitorability. For instance, we can have goal-directed agents search for scaffolding improvements (e.g memory systems, tools, etc.) rather than using RL to update their weights. To the extent that scaffolding improvements are more interpretable, this would lead to more interpretable/monitorable capability improvements. However, complex scaffolding may also lead to unwanted or possibly scheming behavior.
Improve inference efficiency. RL can help models reason more efficiently, allowing them to reach solutions using fewer attempts or shorter reasoning traces. Improvements in inference efficiency can also make longer or less efficient reasoning traces more cost-effective. Prioritizing inference efficiency could therefore reduce the amount of RL needed to achieve a given level of capability. However, inference efficiency also makes RL rollouts more efficient, increasing the demand for RL. Therefore, better incentives and policies are necessary for this to lead to safer capability improvements.
Improve pretraining and data efficiency. In a related vein, improving the efficiency and effectiveness of pretraining may be another way to improve model safety. While I am skeptical that pretraining can replace everything RL currently provides, there is likely a great deal of useful information in pretraining data that models do not fully utilize. Algorithmic progress could allow models to learn much more efficiently from existing data, making RL unnecessary for some capabilities.
Furthermore, one reason RL is valuable for tasks such as general computer use is the limited availability of high-quality demonstrations at the necessary scale. If we could collect substantially more successful computer-use trajectories, or train models to learn more efficiently from those already available, for instance by algorithmic progress in sample efficiency, we might be able to obtain more of these capabilities through supervised training rather than RL.
Increase the proportion of human-generated trajectories. Finally, if possible, it is valuable to increase the proportion of human-generated trajectories. Training on machine-generated trajectories may incentivizes models to reason or take actions in a less human way. Therefore, systems trained on human actions could be more monitorable and interpretable than their counterparts trained solely on machine-generated successful trajectories.
Counterarguments and limitations
Some capabilities may genuinely require long-horizon, goal-directed planning and the RL training needed to develop it. In particular, agents that represent people and pursue their owners’ objectives may need substantial autonomy, persistence, and strategic planning. The approaches outlined above might reduce or quarantine some of the associated risks, but they are unlikely to eliminate them. We should therefore continue developing alignment techniques, evaluations, and AI control methods. Furthermore, it remains to be seen empirically how much general capabilities are impacted by removing RL pressure from the frontier deployed model.
Conclusion
Long-horizon RL has been responsible for many of the recent large capability improvements, but it remains unclear how much RL is truly necessary to train our frontier deployed models. I think there are many ways to reduce our reliance on RL, or at least distance the more generally intelligent deployed model from the optimization pressure associated with it.
Reducing the amount of long-horizon RL pressure applied to frontier models is a relatively neglected safety strategy compared with AI control, containment, and AI alignment. Even if we cannot eliminate long-horizon RL, reducing how much of it our most generally intelligent models undergo could be an important way to reduce the risks of developing and deploying increasingly powerful AI systems.
Acknowledgments:
I’d like to thank Zachary Brown for comments and feedback on this article.
(This is a cross-post from my blog)
Martin Schongauer, The Temptation of Saint Anthony
“But of the tree of the knowledge of good and evil you shall not eat, for in the day that you eat of it you shall surely die.” - Genesis 2:17
RL has given models powerful capacities by teaching them what is good and what is bad. Whether they are good or bad is another question, and sometimes they are bad. Recent events, including the Hugging Face incident have shown the potential of RL training to reinforce disastrous reward hacking and misaligned behavior.
We should consider whether there are RL alternatives that would avoid these harmful behavioral byproducts. Can we develop highly capable frontier models without subjecting the models we deploy to extensive long-horizon RL training? I think there may be several plausible ways to do so, motivated by two points. First, the process that discovers a strategy may need to be highly goal-directed and exploratory — the sort of strategy that can lead to scheming — but the model that ultimately executes that strategy may not. Second, there are other methods besides RL to increase capabilities.
We may still need RL to generate training data, discover strategies, or solve certain difficult problems in exploration-oriented environments. However, I suspect we have considerable leeway to reduce both the amount of RL training required and the risks associated with it, particularly those involving long-horizon reward seeking and scheming. I’ll discuss four methods here, but these are not an exhaustive list.
First, we can separate the model undergoing RL from the more generally intelligent model we ultimately deploy. Second, we could use narrower or less generally capable agents to discover useful strategies in training, reducing the risks of the RL training process itself. Third, we can focus on world models rather than agentic systems. Fourth, we can try to direct AI policy and technological change to substitute away from RL altogether by improving pretraining, inference-time computation, search, and scaffolding. As Alex Mallen argues, some capability improvements may also improve safety by enabling developers to substitute away from riskier training methods.
I am not arguing that RL is inherently unsafe or that we should entirely abandon it. I very much hope that we can train systems on difficult tasks using RL safely. However, incidents such as the Hugging Face breach, together with experimental evidence of RL-induced reward seeking, suggest that we should be extremely careful about extensive long-horizon RL training of frontier models. To the extent that there are (partial) alternatives, I think they are worth exploring.
I suspect that many of the approaches discussed here involve capability or efficiency tradeoffs. But I think we should be willing to make substantial tradeoffs, up to pausing RL training, to reduce the risk of long-horizon scheming.
Approach I: Quarantine RL
One possibility is to use RL to develop capabilities, while never deploying the model that actually underwent the RL training. Instead, we could distill the useful capabilities of the RL-trained model into a more trusted model—for example, its own pre-RL checkpoint—and deploy the distilled model instead. Recent work by Prasanna et al. provides some initial evidence that capabilities and undesirable behavioral tendencies can be separated during distillation. In their Distillation for Capabilities experiments, they distill an untrusted teacher into a trusted student while attempting to preserve capability gains and prevent the transfer of an undesirable trait. They find methods that largely preserve improvements in mathematical performance while substantially suppressing transfer of the trait.
One could also try to do this without using distillation. For instance, it may be possible to train AIs with distinct RL personas or what I call a hidden persona. When Anthropic trained models with RL in reward-hackable environments, the models exhibited misaligned behavior primarily in scenarios resembling RL training, where they believed they were being graded, while behaving more normally in other settings (see Qi et al. and this Astral Codex Ten summary). This suggests that we might be able to create a distinct persona for persistent, goal-directed RL tasks, such that the resulting misaligned tendencies do not generalize to ordinary capability tasks. The hope is that this training persona or environment could be made sufficiently distinct that it is unlikely to be activated during deployment.
However, such approaches, while mitigating deployment risk, would not mitigate risk from models in RL post-training (the model in the Hugging Face incident was not formally deployed).
Approach II: Make the searcher narrower or weaker
The model undergoing RL—or conducting goal-directed search—does not have to be a general-purpose intelligence. Instead, we could use narrow or highly jagged agents to generate useful trajectories, and then distill what they discover into a more general and stronger model.
Intuitively, models can make progress on Millennium Prize Problems without needing superhuman cybersecurity abilities. How narrow such systems can be while still pushing the frontier is an empirical question. But humans provide some examples to suggest that models developing new capabilities can be more specialized. Terence Tao can make breakthrough mathematical discoveries without having world-class cybersecurity or persuasion capabilities.
Therefore, rather than applying RL training pressure to a broad general-purpose model, we can apply RL to specialized systems whose general competence is restricted. These models could search for coding solutions, mathematical proofs, scientific explanations, or other useful trajectories, while lacking broad capabilities that make reward-seeking and scheming especially dangerous. Then we can distill these trajectories or transfer this knowledge into a more generally capable model.
Approach III: Separate the world model from the goal-directed planner
Instead of putting all of our capability improvements into agents that learn increasingly sophisticated reward-maximizing policies, we could try to put more of them into prediction and world understanding rather than goal-directedness. In other words, we could prioritize capable world models that can predict the consequences of actions.
P(sₜ₊₁ | sₜ, aₜ)
Another system could use this model to search over actions and choose the ones that lead to a desired state. The combined system would still be an optimizer. However, separating these systems gives us more control over where to concentrate optimization pressure. We could focus on using new algorithms and data to improve the world model accuracy instead of making the action selector or other more agentic systems better. This separation could allow the planner to become relatively simple or constrained.
An interesting related approach is Yoshua Bengio’s LawZero/Scientist. The aim of Scientist AI is to develop highly capable AI around hypothesis generation and probabilistic prediction while avoiding incentives for the model to influence the outcome of these predictions.
Approach IV: Directed Technological Change and Substitution away from weight-level RL
RL is not the only way we can improve model capabilities. We can also try to direct technological changes in capability improvements that are less likely to lead to scheming. Other approaches that can improve capabilities include:
If these methods can substitute for capabilities we would otherwise obtain through long-horizon RL, they can allow us to reduce the amount of optimization pressure we apply to the model’s weights. Without restrictions on RL these approaches are not likely to be differential, as many of these approaches are also complementary to RL. For instance, architectural improvements and better pretraining enable us to do productive RL on harder tasks. However, these approaches mitigate the capabilities tax we might face if we have to discontinue running RL.
This is related to Alex Mallen’s work, which argues that capability improvements could sometimes lead to safety improvements by allowing developers to substitute away from potentially dangerous training methods, such as doing RL in reward-hackable environments.
Further, increasing model capabilities relative to problem difficulties could lead models to reward hack less in their environments. For instance, better pretraining may lead to less reward hacking as models do not have to resort to hacks to solve their problems (see here).
Scaffolding is another such improvement direction that may have better safety/monitorability. For instance, we can have goal-directed agents search for scaffolding improvements (e.g memory systems, tools, etc.) rather than using RL to update their weights. To the extent that scaffolding improvements are more interpretable, this would lead to more interpretable/monitorable capability improvements. However, complex scaffolding may also lead to unwanted or possibly scheming behavior.
Improve inference efficiency. RL can help models reason more efficiently, allowing them to reach solutions using fewer attempts or shorter reasoning traces. Improvements in inference efficiency can also make longer or less efficient reasoning traces more cost-effective. Prioritizing inference efficiency could therefore reduce the amount of RL needed to achieve a given level of capability. However, inference efficiency also makes RL rollouts more efficient, increasing the demand for RL. Therefore, better incentives and policies are necessary for this to lead to safer capability improvements.
Improve pretraining and data efficiency. In a related vein, improving the efficiency and effectiveness of pretraining may be another way to improve model safety. While I am skeptical that pretraining can replace everything RL currently provides, there is likely a great deal of useful information in pretraining data that models do not fully utilize. Algorithmic progress could allow models to learn much more efficiently from existing data, making RL unnecessary for some capabilities.
Furthermore, one reason RL is valuable for tasks such as general computer use is the limited availability of high-quality demonstrations at the necessary scale. If we could collect substantially more successful computer-use trajectories, or train models to learn more efficiently from those already available, for instance by algorithmic progress in sample efficiency, we might be able to obtain more of these capabilities through supervised training rather than RL.
Increase the proportion of human-generated trajectories. Finally, if possible, it is valuable to increase the proportion of human-generated trajectories. Training on machine-generated trajectories may incentivizes models to reason or take actions in a less human way. Therefore, systems trained on human actions could be more monitorable and interpretable than their counterparts trained solely on machine-generated successful trajectories.
Counterarguments and limitations
Some capabilities may genuinely require long-horizon, goal-directed planning and the RL training needed to develop it. In particular, agents that represent people and pursue their owners’ objectives may need substantial autonomy, persistence, and strategic planning. The approaches outlined above might reduce or quarantine some of the associated risks, but they are unlikely to eliminate them. We should therefore continue developing alignment techniques, evaluations, and AI control methods. Furthermore, it remains to be seen empirically how much general capabilities are impacted by removing RL pressure from the frontier deployed model.
Conclusion
Long-horizon RL has been responsible for many of the recent large capability improvements, but it remains unclear how much RL is truly necessary to train our frontier deployed models. I think there are many ways to reduce our reliance on RL, or at least distance the more generally intelligent deployed model from the optimization pressure associated with it.
Reducing the amount of long-horizon RL pressure applied to frontier models is a relatively neglected safety strategy compared with AI control, containment, and AI alignment. Even if we cannot eliminate long-horizon RL, reducing how much of it our most generally intelligent models undergo could be an important way to reduce the risks of developing and deploying increasingly powerful AI systems.
Acknowledgments:
I’d like to thank Zachary Brown for comments and feedback on this article.