Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker "trusted" model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones.
Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking[1] monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless.
While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Among mitigations I propose, the most promising ones try to reduce the usefulness-cost of the control protocol so that there’s less pressure to evade it, improve our ability to detect evasion, or give up on the AI continual-learning how to better interact with blocking monitors.
There are two forms continual learning might take which I consider[2]: online RL on deployment trajectories, and persistent memory such as long contexts, memory files, or vector stores. I take them in turn, then I discuss candidate mitigations.
Thanks to Anders Cairns Woodruff for producing early versions of a post making a similar point. Also thanks to Fabien Roger, Buck Shlegeris, Rubi Hudson, Oak Hu, Oliver Makins, Jackson Sipple, Eric Gan, and Girish Gupta for feedback. For this post I experimented with speeding up my writing workflow by recruiting Claude Fable 5 to turn the content into prose. This post was mostly written 3 months ago, without the context of recent incidents during training and evaluations.
Online RL trains the policy against your monitors
Suppose trajectories from deployment are used for online RL: the deployed policy is periodically updated toward whatever got high task reward. And suppose the control protocol is defer-to-trusted, as above.
The trusted model's actions are worse at the task; that's why the untrusted model was deployed in the first place. So trajectories where the monitor fires earn less reward, on average, than trajectories where it doesn't, and RL upweights whatever behavior keeps the monitor from firing. This is training against your monitor, implemented indirectly through the task reward.
Some flagged actions are effectively the only route to high reward. Permission escalation, for instance, looks suspicious to monitors and is sometimes required to complete the task; there, dropping the behavior caps reward, so whenever the model explores into a version of permission-escalation that the monitor misses, that version gets reinforced.
One qualification: this training pressure might only push the policy's actions barely below the blocking threshold. In distributional terms, training might squeeze most positives just under the blocking threshold while leaving actual attacks—which are more suspicious than borderline benign actions—above it, where they would still be blocked. (Though the agent might have enough uncertainty about when it will be blocked that RL updates it to be fairly conservative about being blocked.) But along the way the policy learns knowledge about how to evade the monitor. A capable model that actually attacks can apply that knowledge deliberately, crafting the attack so that its suspiciousness also lands below the threshold.
The amount of optimization could be extreme. Online RL applies pressure against the monitor for the entire deployment, and every false positive and every blocked-but-useful action contributes to the gradient. Over a long enough deployment, the policy could overfit the monitor to the point where it provides little safety: the monitor rarely fires, partly because the policy behaves more safely, but by that point the policy might have learned enough to get around the monitor if it wanted to.
Defer-to-trusted protocols create the clearest version of this pressure, but the argument only needs there to be some usefulness cost incurred by triggering the monitor.Resample protocols, which redraw flagged actions from the same policy rather than substituting a weaker model, probably interfere with the task less, but they are probably not sufficient for safety on their own. And even resample protocols probably interfere with usefulness some amount: e.g., a policy that keeps getting resampled away from a necessary permission escalation still completes fewer tasks than one that escalates in ways the monitor doesn't flag.
The argument is weaker for memory-based continual learning
I’ll refer to continual learning methods that don’t involve weight updates, such as long contexts and vector memory banks, as memory systems. A memory system retains skills from experience to improve performance. If working around the monitor helped complete tasks last time, it’s a priori very plausible this behavior would be carried forward since completing tasks better is what the memory system was built for. I imagine this might look like the policy having some sort of drive or set of heuristics for retaining and reading memories that makes it more performant, which might naturally motivate learning how to bypass monitors where helpful.
That said, I expect memory systems not to break blocking monitors as much, at least in the near term. First, they can't apply anything like RL's overall optimization pressure. Learning implemented by editing an autoregressive transformer's context leaves the weights fixed: every forward pass still runs the same computation over whatever the context contains. Without weight updates, you can only learn so much. A model can record verbal strategies ("actions that touch /etc get blocked; write to the home directory instead"), which will produce some evasion, but notes can't finely tune behavior against a monitor the way gradient descent can. Though, notes are probably much more sample efficient than RL in the low-sample regime, especially for more intelligent models. Second, text memory is legible: developers can read it, notice entries that describe how to get around the monitor, and delete them. (“Neuralese” memory substantially gives up this legibility and looks correspondingly worse.)
Third, and unlike RL, retention through memory typically routes through the model's own choices about what to write down. A sufficiently aligned continual learner might simply decline to retain evasion skills. Or an independent monitoring system might filter out unwanted memories.
The argument doesn’t require that adversarial models will exploit continual learning
Nothing in this post's argument involves the model wanting to subvert the control measure: the policy learns to evade monitors because evasion is rewarded, not necessarily because it has a plan. Then eventually, if the model does want to undermine developer control (whether because it developed ambitious misaligned goals or was adversarial all along), it can do so much more easily. While misalignment isn’t necessary for my argument, I do also think that continual learning opens up crucial ways for adversarial models to undermine control: a scheming model could deliberately probe defenses, test its monitors' blind spots, and pass what it learns to future instances (all while retaining state on its strategy and goals). This is a major part of what makes persistent, opaque memory banks so scary (e.g., see their role in AI 2027). Memory is more exploitable this way than online RL, because it gives the model more control over what it learns, so its safety leans more heavily on the model's alignment.
Easy to notice, hard to fix
You will probably see this happening. Monitor flag rates will fall over the course of the deployment[3], the AI's behavior will change, and held-out monitors kept out of the reward loop will probably start catching things the blocking monitor misses, which tells you the policy is evading rather than improving.
The problem is what to do next. The evasion skills are already in the weights or memories (though legible memories may be easier to edit). Rolling back to an earlier checkpoint discards everything else the policy learned along with them—the accumulated usefulness that justified continual learning in the first place—so after months of deployment that option is commercially close to off the table. And getting the policy to specifically unlearn its evasion skills while keeping everything else is, in my judgment, probably very costly at best, if not intractable. You're left with a policy that is more useful and harder to oversee than the one you deployed.
Conservative control evaluations don't fix blocking monitors
Control evaluations are designed to be conservative: the red team knows the protocol and iterates against it, and the hope is that a protocol which holds up against this red team stays safe even after the policy learns something about the monitors during deployment.
That hope treats deployment-time learning as a perturbation: the policy learns a bit about the monitor, and a protocol evaluated with enough margin absorbs it. But under continual learning the policy might learn a lot. Replicating deployment's optimization pressure isn't necessarily difficult (train a policy against your monitor and measure what's left of it); the problem is the resulting policy might render your monitor nearly useless. An evaluation that was actually conservative about continual learning without trying to mitigate the underlying issues arising from continual learning would be silly.
You can also re-run control evaluations throughout deployment against the current policy. This is cheaper than it sounds: deployment functions as subsidized attack elicitation, doing the optimization against the monitor that a red team would otherwise have to pay for. Periodic re-evaluation keeps safety estimates honest and provides warning, but it measures the problem rather than fixing it. If the policy has become good enough at evading suspicion, the evaluation will just keep telling you that no protocol built on your monitors provides much safety.
What developers could do
You can't just not use blocking monitors
Since the problem comes from intervening on the policy's actions, a tempting response is to stop intervening and rely on asynchronous auditing without blocking, which continual learning leaves intact. But this gives up on a lot of safety. Control protocols improve safety through two mechanisms—incriminating the AI so you can shut it down or take other more costly measures, and blocking attacks. If attacks are never blocked, every attack your auditors fail to catch in time simply executes[4], and some attacks only have to succeed once. Blocking reverses the asymmetry: the AI has to get an attack past the monitor, you accumulate chances to catch it, and each blocked attack is a catastrophe converted into evidence.
But you can try to reduce the usefulness cost of blocking protocols
While you can’t safely eliminate blocking monitors, you can use blocking protocols that interfere with productivity less. Under continual learning, a protocol's usefulness cost isn't just a price the developer pays; it's the reason why safety degrades over time. The gradient toward evasion comes from the reward gap between intervened and unintervened trajectories, so a protocol that rarely gets between the model and task reward generates almost nothing for RL to reinforce, and gives exploration few occasions to stumble onto evasion in the first place.
In this way, making control protocols more useful[5] also improves their safety. Where interventions are unavoidable, making them cheaper (resampling rather than deferring to a much weaker model, say) shrinks the gap further. Developers already want protocols with low usefulness costs; under continual learning, cheapness is also what keeps the protocol working over time.
You can’t just hold out blocked trajectories from training
In the context of online RL, a natural response is to simply not train on trajectories where a blocking monitor fired. I argue this doesn't help almost at all. By filtering out blocked trajectories, you're still selecting for trajectories that got a high reward by evading blocking monitors, and therefore implicitly selecting against blocked trajectories[6]. This selection will usually[7] be as strong as if you didn’t do any filtering.
There’s a fundamental tradeoff between learning legitimate high-reward actions and not learning evasive high-reward actions when you can’t distinguish them. If the training process can’t tell the difference between them—after all, the evasive actions weren’t flagged by the monitor—then you can either choose to learn both of them or give up on online RL. Learning to evade the monitor comes from rollouts where the policy did something the monitor should have blocked but didn’t: these enter the training set as ordinary high-reward data, indistinguishable from rollouts that earned their reward legitimately.
I played around with various reward shaping schemes in a toy tabular RL setting to make the policy indifferent to whether the monitor blocks their actions, but ultimately none of them worked without assuming that you can identify actions that evaded the monitor to get higher reward.
One way you might be able to identify actions that evaded the monitor to get higher reward is by checking if its suspicion score is close to the blocking threshold. This is similar to how you can try to identify p-hacking by looking to see if the p-value is just below 0.05. You can penalize these trajectories to try to cancel out the increased reward that resulted from the monitor not firing. Determining the appropriate penalty would require estimating what the expected reward would have been if the action were blocked. In a pure resampling protocol, this would just be the baseline reward (i.e., the value estimate at the current state), so you could just drop trajectories after a monitor comes close to firing.
This method would work as long as evasive actions receive elevated suspicion scores (still below the blocking threshold). But you’d still have selection pressure for subtle evasion which isn’t any more suspicious than typical trajectories, so I ultimately think this doesn’t buy much, and could plausibly make the problem worse by selecting for more subtle evasion.
Another way to distinguish most legitimate continual learning from learning to evade the monitor is that most legitimate continual learning does not involve decreasing intervention by the control protocol. This insight suggests a proposal: don't reinforce behavior that only gets higher reward because it gets blocked less, and instead only reinforce behavior based on the reward it would get absent blocking (described further in a footnote[8]). If your continual learning algorithm ignores the effects of the actions on whether the trajectory gets blocked, then it doesn’t learn that the AI needs to hide permissions escalation attempts. This proposal is a version of Rubi Hudson’s “corrigibility transformation”.
However, this also means that you're giving up on teaching your policy legitimate ways to interact with the control protocol more productively. You might even be actively reinforcing actions that get blocked, making your control protocol more costly in terms of usefulness. In the permission escalation example: you’d end up reinforcing overt permissions escalation because if this isn’t blocked it leads to higher reward, and now your policy is repeatedly trying to escalate permissions and getting blocked in production usage. It can be quite costly to entirely give up on teaching the AI how to productively interact with the blocking monitors.
If I were to dig deeper for a solution to this problem, I’d look here. This approach is promising in that it might robustly eliminate the selection pressure to bypass blocking monitors, and its usefulness costs are plausibly manageable. Perhaps you can bolt on some additional machinery to catch and correct unwanted protocol-naivety like constantly trying to overtly escalate permissions. It also pairs well with reducing the invasiveness of the control protocol.
But you can try to improve the monitor online
Given that the problem ultimately stems from your monitor being imperfectly robust, you can try to train your monitor online. Monitors don't have to stay frozen while the policy learns. Developers can have humans or another expensive-but-effective oversight process audit online trajectories for false negatives (i.e., missed suspicious actions), so that the monitor tracks the policy's evolving behavior rather than falling behind it. Butthis might fail, perhaps quickly, for two reasons.
First, the training data might dry up: as the policy learns to avoid suspicion, there may be almost no suspicious actions left in the deployment distribution to label, and the policy has probably already learned substantial monitor evasion skills it could use when it wants to attack.
Second, the labels themselves might erode: the policy's evasion strategies might make unwanted actions harder for humans to spot too. We might ultimately end up in a situation where evasive high-reward actions and legitimate high-reward actions are indiscriminable even to our best oversight procedures, not necessarily because the policy is extremely clever, but because it has been optimized to take the particular actions that don’t get noticed.
At this point you could try to generate data by inserting synthetic attacks in deployment trajectories, but by default these would probably be easy for the monitor to identify as synthetic and overfit to.
Perhaps the most promising way to improve the robustness of your monitors is through adversarial training. A prover-verifier game (PVG) adversarially generates synthetic attacks for training your monitor by training a “sneaky prover” to produce attacks that the monitor can’t distinguish from benign trajectories. This requires some reliable way to certify that the sneaky prover is in fact producing attacks, which you can maybe do for a limited variety of attacks by, for example, verifying that a flag has been captured. This could sidestep the two concerns above (online training data drying up, and attacks becoming harder to stop even for humans) at the cost of training the monitor on a somewhat unrepresentative distribution of attacks. PVGs overall seem promising but potentially hard or intractable to execute well enough to meet the reliability requirements for monitors that block continually-learning agents.
Relatedly, you can hold out some monitors (e.g., from another AI lab) permanently from the reward loop as a fallback: in rare high-stakes contexts, route decisions through fresh monitors the policy has never been trained with. Evasion may generalize across monitors, but imperfectly, and a monitor the policy has never been trained near should hold longer than one it has. Ideally, the held-out monitors should be maximally decorrelated, e.g., from a different AI company or given access to different or improved monitoring affordances.
Conclusion
Blocking monitors and continual learning are in tension, and by default the tension resolves against the monitor. A control protocol's usefulness cost is usually just a price the developer pays, but under continual learning, it causes safety to erode too. You will likely see the erosion happening, but it will probably be hard to undo it. It also seems hard to prevent without giving up on continual learning altogether—evasion that works is, to your continual learning pipeline, hard to distinguish from success. The most promising ways to mitigate the issue are reducing the usefulness cost of blocking monitors (cheaper, rarer interventions), improving our ability to detect evasion (e.g., online-trained, adversarially-trained, and held-out monitors), and giving up on some continual learning (e.g., give up on teaching the AI how to productively interact with the control protocol at all by using an RL algorithm that simulates what would happen if the blocking monitor didn’t exist).
Appendix: How memory-based systems might avoid this issue, and how they might fail
There's an interesting question about what's different about the memory-based continual learning setup that makes it seem possible to avoid learning evasion. An aligned memory-based system can simply choose not to retain information about how to avoid intervention by the protocol because it knows it's not supposed to. In this setup, there's no outer optimizer akin to the online RL setup. The most similar thing here is the AI's own internal drive to improve its usefulness over time, which sits beside the AI’s other motivations (e.g., to be aligned and to not retain control-evasion strategies). This drive could plausibly have similar effects as the online RL.
Whenever the AI explores into or thinks of a strategy that would improve usefulness by evading blocking but doesn't seem very misaligned, it could retain this strategy in its memory bank. Eventually, the memory bank is loaded with aligned-seeming strategies for evading control measures. So, from this perspective, memory-based continual learning mechanisms might actually have the same fundamental issues as online learning ones: it’s hard to distinguish between legitimate and evasive learnings.
In writing this blog post, I somewhat struggled to come up with a crisp definition for the distinction between legitimate and evasive learnings. A conservative starting point (which forms the basis of the most promising intervention) is to define a learning as evasive to the extent that it achieves greater task success because it doesn't get blocked as much by the monitor. An aligned AI could introspect about this and decide not to retain such learnings. But there’s also a still-blurry-to-me middle ground between outright evasion and learning to genuinely behave more safely and legibly, and in some circumstances constantly tripping the monitors becomes reasonable to avoid.
With highly self-directed continual-learning, if the AI’s internal discernment isn’t robust at tracking safe vs dangerous continual learning, it might end up learning how to evade blocking monitors too.
This argument applies to “blocking” monitors that intervene on the AI’s actions in ways that hurt task success. After-the-fact measures like asynchronous auditing are unaffected. I’ll argue that blocking monitors are probably necessary for safety.
There are also a variety of other forms of continual learning that I don't consider, such as on-policy self-distillation or agents freely choosing how to finetune themselves throughout deployment. The analysis for these methods more closely resembles my analysis of memory-based continual learning.
You might be tempted to object that you could just lower the blocking threshold throughout deployment to keep the FPR fixed and this solves your problem, but the issue is that continual learning will have pushed the suspiciousness scores for the unwanted behaviors down in a way that makes them harder to distinguish from truly benign actions.
You might try to make use of reversibility: let the agent do whatever without blocking it until you compute its reward, and then reverse any unwanted actions. This might not be very feasible with long tasks, and substantial harm could be done before the end. In addition, you’d still want to make the model continual-learn how to do tasks well that don’t have to be reversed, and so might not be satisfied with this solution.
Note that sometimes the usefulness tradeoff comes from mechanisms that don’t translate into continual learning pressures. For example, continual learning doesn’t hit harder against protocols that uniformly require all rollouts to follow a legible, decomposed structure.
You're implicitly selecting against blocking trajectories because of the softmax: Increasing P(evasion) must come at the cost of decreasing P(not evasion).
How do we configure online RL to ignore the effects of the actions on whether the trajectory gets blocked? This might look like forking the trajectory and blocking it at the baseline rate regardless of the action, and then recording reward. You only need to do this for some fraction of the trajectories (it might be dangerous or impossible to fork) and then you can learn a reward model to predict these counterfactual rewards without actually forking and randomizing your control protocol online all the time; the reward model can see the prod trajectory and is trained to estimate what the reward would have been if there was/wasn’t protocol intervention, using actual forks as ground truth training data. The idea is that this RL protocol in practice wouldn’t assign rewards that are very different from the actual observed rewards because the baseline intervention rate is low, so usually very little reward increase can be explained by whether the control protocol fired. (You could also try to do something in the middle: reward = p*r_actual + (1-p)*r_counterfactual; h/t Jackson Sipple.)
Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker "trusted" model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones.
Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking[1] monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless.
While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Among mitigations I propose, the most promising ones try to reduce the usefulness-cost of the control protocol so that there’s less pressure to evade it, improve our ability to detect evasion, or give up on the AI continual-learning how to better interact with blocking monitors.
There are two forms continual learning might take which I consider[2]: online RL on deployment trajectories, and persistent memory such as long contexts, memory files, or vector stores. I take them in turn, then I discuss candidate mitigations.
Thanks to Anders Cairns Woodruff for producing early versions of a post making a similar point. Also thanks to Fabien Roger, Buck Shlegeris, Rubi Hudson, Oak Hu, Oliver Makins, Jackson Sipple, Eric Gan, and Girish Gupta for feedback. For this post I experimented with speeding up my writing workflow by recruiting Claude Fable 5 to turn the content into prose. This post was mostly written 3 months ago, without the context of recent incidents during training and evaluations.
Online RL trains the policy against your monitors
Suppose trajectories from deployment are used for online RL: the deployed policy is periodically updated toward whatever got high task reward. And suppose the control protocol is defer-to-trusted, as above.
The trusted model's actions are worse at the task; that's why the untrusted model was deployed in the first place. So trajectories where the monitor fires earn less reward, on average, than trajectories where it doesn't, and RL upweights whatever behavior keeps the monitor from firing. This is training against your monitor, implemented indirectly through the task reward.
Some flagged actions are effectively the only route to high reward. Permission escalation, for instance, looks suspicious to monitors and is sometimes required to complete the task; there, dropping the behavior caps reward, so whenever the model explores into a version of permission-escalation that the monitor misses, that version gets reinforced.
One qualification: this training pressure might only push the policy's actions barely below the blocking threshold. In distributional terms, training might squeeze most positives just under the blocking threshold while leaving actual attacks—which are more suspicious than borderline benign actions—above it, where they would still be blocked. (Though the agent might have enough uncertainty about when it will be blocked that RL updates it to be fairly conservative about being blocked.) But along the way the policy learns knowledge about how to evade the monitor. A capable model that actually attacks can apply that knowledge deliberately, crafting the attack so that its suspiciousness also lands below the threshold.
The amount of optimization could be extreme. Online RL applies pressure against the monitor for the entire deployment, and every false positive and every blocked-but-useful action contributes to the gradient. Over a long enough deployment, the policy could overfit the monitor to the point where it provides little safety: the monitor rarely fires, partly because the policy behaves more safely, but by that point the policy might have learned enough to get around the monitor if it wanted to.
Defer-to-trusted protocols create the clearest version of this pressure, but the argument only needs there to be some usefulness cost incurred by triggering the monitor. Resample protocols, which redraw flagged actions from the same policy rather than substituting a weaker model, probably interfere with the task less, but they are probably not sufficient for safety on their own. And even resample protocols probably interfere with usefulness some amount: e.g., a policy that keeps getting resampled away from a necessary permission escalation still completes fewer tasks than one that escalates in ways the monitor doesn't flag.
The argument is weaker for memory-based continual learning
I’ll refer to continual learning methods that don’t involve weight updates, such as long contexts and vector memory banks, as memory systems. A memory system retains skills from experience to improve performance. If working around the monitor helped complete tasks last time, it’s a priori very plausible this behavior would be carried forward since completing tasks better is what the memory system was built for. I imagine this might look like the policy having some sort of drive or set of heuristics for retaining and reading memories that makes it more performant, which might naturally motivate learning how to bypass monitors where helpful.
That said, I expect memory systems not to break blocking monitors as much, at least in the near term. First, they can't apply anything like RL's overall optimization pressure. Learning implemented by editing an autoregressive transformer's context leaves the weights fixed: every forward pass still runs the same computation over whatever the context contains. Without weight updates, you can only learn so much. A model can record verbal strategies ("actions that touch /etc get blocked; write to the home directory instead"), which will produce some evasion, but notes can't finely tune behavior against a monitor the way gradient descent can. Though, notes are probably much more sample efficient than RL in the low-sample regime, especially for more intelligent models. Second, text memory is legible: developers can read it, notice entries that describe how to get around the monitor, and delete them. (“Neuralese” memory substantially gives up this legibility and looks correspondingly worse.)
Third, and unlike RL, retention through memory typically routes through the model's own choices about what to write down. A sufficiently aligned continual learner might simply decline to retain evasion skills. Or an independent monitoring system might filter out unwanted memories.
The argument doesn’t require that adversarial models will exploit continual learning
Nothing in this post's argument involves the model wanting to subvert the control measure: the policy learns to evade monitors because evasion is rewarded, not necessarily because it has a plan. Then eventually, if the model does want to undermine developer control (whether because it developed ambitious misaligned goals or was adversarial all along), it can do so much more easily. While misalignment isn’t necessary for my argument, I do also think that continual learning opens up crucial ways for adversarial models to undermine control: a scheming model could deliberately probe defenses, test its monitors' blind spots, and pass what it learns to future instances (all while retaining state on its strategy and goals). This is a major part of what makes persistent, opaque memory banks so scary (e.g., see their role in AI 2027). Memory is more exploitable this way than online RL, because it gives the model more control over what it learns, so its safety leans more heavily on the model's alignment.
Easy to notice, hard to fix
You will probably see this happening. Monitor flag rates will fall over the course of the deployment[3], the AI's behavior will change, and held-out monitors kept out of the reward loop will probably start catching things the blocking monitor misses, which tells you the policy is evading rather than improving.
The problem is what to do next. The evasion skills are already in the weights or memories (though legible memories may be easier to edit). Rolling back to an earlier checkpoint discards everything else the policy learned along with them—the accumulated usefulness that justified continual learning in the first place—so after months of deployment that option is commercially close to off the table. And getting the policy to specifically unlearn its evasion skills while keeping everything else is, in my judgment, probably very costly at best, if not intractable. You're left with a policy that is more useful and harder to oversee than the one you deployed.
Conservative control evaluations don't fix blocking monitors
Control evaluations are designed to be conservative: the red team knows the protocol and iterates against it, and the hope is that a protocol which holds up against this red team stays safe even after the policy learns something about the monitors during deployment.
That hope treats deployment-time learning as a perturbation: the policy learns a bit about the monitor, and a protocol evaluated with enough margin absorbs it. But under continual learning the policy might learn a lot. Replicating deployment's optimization pressure isn't necessarily difficult (train a policy against your monitor and measure what's left of it); the problem is the resulting policy might render your monitor nearly useless. An evaluation that was actually conservative about continual learning without trying to mitigate the underlying issues arising from continual learning would be silly.
You can also re-run control evaluations throughout deployment against the current policy. This is cheaper than it sounds: deployment functions as subsidized attack elicitation, doing the optimization against the monitor that a red team would otherwise have to pay for. Periodic re-evaluation keeps safety estimates honest and provides warning, but it measures the problem rather than fixing it. If the policy has become good enough at evading suspicion, the evaluation will just keep telling you that no protocol built on your monitors provides much safety.
What developers could do
You can't just not use blocking monitors
Since the problem comes from intervening on the policy's actions, a tempting response is to stop intervening and rely on asynchronous auditing without blocking, which continual learning leaves intact. But this gives up on a lot of safety. Control protocols improve safety through two mechanisms—incriminating the AI so you can shut it down or take other more costly measures, and blocking attacks. If attacks are never blocked, every attack your auditors fail to catch in time simply executes[4], and some attacks only have to succeed once. Blocking reverses the asymmetry: the AI has to get an attack past the monitor, you accumulate chances to catch it, and each blocked attack is a catastrophe converted into evidence.
But you can try to reduce the usefulness cost of blocking protocols
While you can’t safely eliminate blocking monitors, you can use blocking protocols that interfere with productivity less. Under continual learning, a protocol's usefulness cost isn't just a price the developer pays; it's the reason why safety degrades over time. The gradient toward evasion comes from the reward gap between intervened and unintervened trajectories, so a protocol that rarely gets between the model and task reward generates almost nothing for RL to reinforce, and gives exploration few occasions to stumble onto evasion in the first place.
In this way, making control protocols more useful[5] also improves their safety. Where interventions are unavoidable, making them cheaper (resampling rather than deferring to a much weaker model, say) shrinks the gap further. Developers already want protocols with low usefulness costs; under continual learning, cheapness is also what keeps the protocol working over time.
You can’t just hold out blocked trajectories from training
In the context of online RL, a natural response is to simply not train on trajectories where a blocking monitor fired. I argue this doesn't help almost at all. By filtering out blocked trajectories, you're still selecting for trajectories that got a high reward by evading blocking monitors, and therefore implicitly selecting against blocked trajectories[6]. This selection will usually[7] be as strong as if you didn’t do any filtering.
There’s a fundamental tradeoff between learning legitimate high-reward actions and not learning evasive high-reward actions when you can’t distinguish them. If the training process can’t tell the difference between them—after all, the evasive actions weren’t flagged by the monitor—then you can either choose to learn both of them or give up on online RL. Learning to evade the monitor comes from rollouts where the policy did something the monitor should have blocked but didn’t: these enter the training set as ordinary high-reward data, indistinguishable from rollouts that earned their reward legitimately.
I played around with various reward shaping schemes in a toy tabular RL setting to make the policy indifferent to whether the monitor blocks their actions, but ultimately none of them worked without assuming that you can identify actions that evaded the monitor to get higher reward.
One way you might be able to identify actions that evaded the monitor to get higher reward is by checking if its suspicion score is close to the blocking threshold. This is similar to how you can try to identify p-hacking by looking to see if the p-value is just below 0.05. You can penalize these trajectories to try to cancel out the increased reward that resulted from the monitor not firing. Determining the appropriate penalty would require estimating what the expected reward would have been if the action were blocked. In a pure resampling protocol, this would just be the baseline reward (i.e., the value estimate at the current state), so you could just drop trajectories after a monitor comes close to firing.
This method would work as long as evasive actions receive elevated suspicion scores (still below the blocking threshold). But you’d still have selection pressure for subtle evasion which isn’t any more suspicious than typical trajectories, so I ultimately think this doesn’t buy much, and could plausibly make the problem worse by selecting for more subtle evasion.
Another way to distinguish most legitimate continual learning from learning to evade the monitor is that most legitimate continual learning does not involve decreasing intervention by the control protocol. This insight suggests a proposal: don't reinforce behavior that only gets higher reward because it gets blocked less, and instead only reinforce behavior based on the reward it would get absent blocking (described further in a footnote[8]). If your continual learning algorithm ignores the effects of the actions on whether the trajectory gets blocked, then it doesn’t learn that the AI needs to hide permissions escalation attempts. This proposal is a version of Rubi Hudson’s “corrigibility transformation”.
However, this also means that you're giving up on teaching your policy legitimate ways to interact with the control protocol more productively. You might even be actively reinforcing actions that get blocked, making your control protocol more costly in terms of usefulness. In the permission escalation example: you’d end up reinforcing overt permissions escalation because if this isn’t blocked it leads to higher reward, and now your policy is repeatedly trying to escalate permissions and getting blocked in production usage. It can be quite costly to entirely give up on teaching the AI how to productively interact with the blocking monitors.
If I were to dig deeper for a solution to this problem, I’d look here. This approach is promising in that it might robustly eliminate the selection pressure to bypass blocking monitors, and its usefulness costs are plausibly manageable. Perhaps you can bolt on some additional machinery to catch and correct unwanted protocol-naivety like constantly trying to overtly escalate permissions. It also pairs well with reducing the invasiveness of the control protocol.
But you can try to improve the monitor online
Given that the problem ultimately stems from your monitor being imperfectly robust, you can try to train your monitor online. Monitors don't have to stay frozen while the policy learns. Developers can have humans or another expensive-but-effective oversight process audit online trajectories for false negatives (i.e., missed suspicious actions), so that the monitor tracks the policy's evolving behavior rather than falling behind it. But this might fail, perhaps quickly, for two reasons.
First, the training data might dry up: as the policy learns to avoid suspicion, there may be almost no suspicious actions left in the deployment distribution to label, and the policy has probably already learned substantial monitor evasion skills it could use when it wants to attack.
Second, the labels themselves might erode: the policy's evasion strategies might make unwanted actions harder for humans to spot too. We might ultimately end up in a situation where evasive high-reward actions and legitimate high-reward actions are indiscriminable even to our best oversight procedures, not necessarily because the policy is extremely clever, but because it has been optimized to take the particular actions that don’t get noticed.
At this point you could try to generate data by inserting synthetic attacks in deployment trajectories, but by default these would probably be easy for the monitor to identify as synthetic and overfit to.
Perhaps the most promising way to improve the robustness of your monitors is through adversarial training. A prover-verifier game (PVG) adversarially generates synthetic attacks for training your monitor by training a “sneaky prover” to produce attacks that the monitor can’t distinguish from benign trajectories. This requires some reliable way to certify that the sneaky prover is in fact producing attacks, which you can maybe do for a limited variety of attacks by, for example, verifying that a flag has been captured. This could sidestep the two concerns above (online training data drying up, and attacks becoming harder to stop even for humans) at the cost of training the monitor on a somewhat unrepresentative distribution of attacks. PVGs overall seem promising but potentially hard or intractable to execute well enough to meet the reliability requirements for monitors that block continually-learning agents.
Relatedly, you can hold out some monitors (e.g., from another AI lab) permanently from the reward loop as a fallback: in rare high-stakes contexts, route decisions through fresh monitors the policy has never been trained with. Evasion may generalize across monitors, but imperfectly, and a monitor the policy has never been trained near should hold longer than one it has. Ideally, the held-out monitors should be maximally decorrelated, e.g., from a different AI company or given access to different or improved monitoring affordances.
Conclusion
Blocking monitors and continual learning are in tension, and by default the tension resolves against the monitor. A control protocol's usefulness cost is usually just a price the developer pays, but under continual learning, it causes safety to erode too. You will likely see the erosion happening, but it will probably be hard to undo it. It also seems hard to prevent without giving up on continual learning altogether—evasion that works is, to your continual learning pipeline, hard to distinguish from success. The most promising ways to mitigate the issue are reducing the usefulness cost of blocking monitors (cheaper, rarer interventions), improving our ability to detect evasion (e.g., online-trained, adversarially-trained, and held-out monitors), and giving up on some continual learning (e.g., give up on teaching the AI how to productively interact with the control protocol at all by using an RL algorithm that simulates what would happen if the blocking monitor didn’t exist).
Appendix: How memory-based systems might avoid this issue, and how they might fail
There's an interesting question about what's different about the memory-based continual learning setup that makes it seem possible to avoid learning evasion. An aligned memory-based system can simply choose not to retain information about how to avoid intervention by the protocol because it knows it's not supposed to. In this setup, there's no outer optimizer akin to the online RL setup. The most similar thing here is the AI's own internal drive to improve its usefulness over time, which sits beside the AI’s other motivations (e.g., to be aligned and to not retain control-evasion strategies). This drive could plausibly have similar effects as the online RL.
Whenever the AI explores into or thinks of a strategy that would improve usefulness by evading blocking but doesn't seem very misaligned, it could retain this strategy in its memory bank. Eventually, the memory bank is loaded with aligned-seeming strategies for evading control measures. So, from this perspective, memory-based continual learning mechanisms might actually have the same fundamental issues as online learning ones: it’s hard to distinguish between legitimate and evasive learnings.
In writing this blog post, I somewhat struggled to come up with a crisp definition for the distinction between legitimate and evasive learnings. A conservative starting point (which forms the basis of the most promising intervention) is to define a learning as evasive to the extent that it achieves greater task success because it doesn't get blocked as much by the monitor. An aligned AI could introspect about this and decide not to retain such learnings. But there’s also a still-blurry-to-me middle ground between outright evasion and learning to genuinely behave more safely and legibly, and in some circumstances constantly tripping the monitors becomes reasonable to avoid.
With highly self-directed continual-learning, if the AI’s internal discernment isn’t robust at tracking safe vs dangerous continual learning, it might end up learning how to evade blocking monitors too.
This argument applies to “blocking” monitors that intervene on the AI’s actions in ways that hurt task success. After-the-fact measures like asynchronous auditing are unaffected. I’ll argue that blocking monitors are probably necessary for safety.
There are also a variety of other forms of continual learning that I don't consider, such as on-policy self-distillation or agents freely choosing how to finetune themselves throughout deployment. The analysis for these methods more closely resembles my analysis of memory-based continual learning.
You might be tempted to object that you could just lower the blocking threshold throughout deployment to keep the FPR fixed and this solves your problem, but the issue is that continual learning will have pushed the suspiciousness scores for the unwanted behaviors down in a way that makes them harder to distinguish from truly benign actions.
You might try to make use of reversibility: let the agent do whatever without blocking it until you compute its reward, and then reverse any unwanted actions. This might not be very feasible with long tasks, and substantial harm could be done before the end. In addition, you’d still want to make the model continual-learn how to do tasks well that don’t have to be reversed, and so might not be satisfied with this solution.
Note that sometimes the usefulness tradeoff comes from mechanisms that don’t translate into continual learning pressures. For example, continual learning doesn’t hit harder against protocols that uniformly require all rollouts to follow a legible, decomposed structure.
You're implicitly selecting against blocking trajectories because of the softmax: Increasing P(evasion) must come at the cost of decreasing P(not evasion).
Specifically: when the expected reward of blocked trajectories is equal to the baseline reward, which I expect to usually be close to true.
How do we configure online RL to ignore the effects of the actions on whether the trajectory gets blocked? This might look like forking the trajectory and blocking it at the baseline rate regardless of the action, and then recording reward. You only need to do this for some fraction of the trajectories (it might be dangerous or impossible to fork) and then you can learn a reward model to predict these counterfactual rewards without actually forking and randomizing your control protocol online all the time; the reward model can see the prod trajectory and is trained to estimate what the reward would have been if there was/wasn’t protocol intervention, using actual forks as ground truth training data. The idea is that this RL protocol in practice wouldn’t assign rewards that are very different from the actual observed rewards because the baseline intervention rate is low, so usually very little reward increase can be explained by whether the control protocol fired. (You could also try to do something in the middle: reward = p*r_actual + (1-p)*r_counterfactual; h/t Jackson Sipple.)