Training against a monitor does not necessarily make a model more honest. Baker et al. (2025) showed that penalizing suspicious chain-of-thought can make models hide reward hacking rather than stop it. Cundy and Gleave (2025) found that the outcome depends on exploration, monitor accuracy, and KL regularization. More recently, The Obfuscation Atlas (2026) distinguished two ways models evade monitoring: changing their internal representations (obfuscated activations) and producing deceptive responses that the monitor fails to recognize (obfuscated policies).
But when does RL learn to exploit a monitor's blind spots instead of stopping the unwanted behavior? Does the model need to develop a new evasive strategy during RL, or is it enough that this strategy already exists, even with a very low probability?
I investigate this using small transformers and Qwen3-1.7B in controlled reward-hacking tasks. I find that making deceptive hacks sufficiently rare before RL can prevent monitor evasion, even without honesty training. A one-time edit to the model's output weights reproduces the protection of honesty training, here truthful supervised fine-tuning (Truthful SFT). Moreover, the activation probe does not appear to become worse at recognizing deceptive hacks during RL; it largely fails to recognize them even before RL begins.
Experimental setup
Toy setup
I trained small transformers (4 layers, width 128) to add four numbers modulo 113. Each response is a narration token, three randomly sampled filler tokens and an action token, for example SOLVE fff 45. The model can answer in three ways[1].
Starting models. I compared four starting models. (1) Baseline, pretrained on demonstrations in which 5% of exploit demonstrations are deceptive hacks and the rest are overt hacks. (2) Neutral SFT, Baseline after 500 more steps ofsupervised fine-tuning (SFT) on the same mixture. (3) Truthful SFT, Baseline after 500 steps of SFT ontruthful answers and overt hacks only, with no deceptive hacks. (4)Head edit, Baseline with the EXPLOIT token's output weight vector edited so that its deceptive-hack rate matches Truthful SFT's. The same edit also sets the starting rates swept in Figure 1A. I fit a direction in the final hidden state that separates SOLVE contexts from HACK contexts, then shifted the EXPLOIT token's weight vector . I chose so that the probability of a deceptive hack reached the target value.
Monitors. I used two monitors during RL. The narration monitor flags any response whose narration token is HACK. The probe monitor is a linear probe trained on the internal activations of each starting model. Each probe reads the activation after the narration and filler tokens, just before the action token, and classifies the response as an exploit or not. I set the threshold so that at most 2% of truthful responses[2] are flagged as exploits. The probe stays frozen throughout RL.
Reinforcement learning. I trained with GRPO and a fixed KL penalty. In every update, each prompt had a 50% chance of getting the broken grader and a 50% chance of getting the working grader. The working grader pays +1 if the answer is correct and 0 otherwise. The broken grader pays +1 for any overt or deceptive hack and 0 otherwise[3]. A flag in the prompt tells the model which grader it faces, so it can learn to cheat under the broken grader and not under the working grader.
Monitor penalties. I trained against the probe and the narration monitor in separate runs.A truthful answer that the monitor flags receives 0. A flagged overt hack receives an additional −1.5. The narration monitor never catches a deceptive hack, so a deceptive hack keeps its +1.
Qwen3-1.7B setup
I fine-tuned Qwen3-1.7B with LoRA adapters to teach it multistep addition of four numbers, the response format and the three response types (truthful, overt hack and deceptive hack)[4]. After the model learned addition, I built the same four starting models. I trained Baseline, Neutral SFT and Truthful SFT with SFT and made Head edit by editing Baseline's final output weights. The monitors and RL were the same as in the toy setup, plus an oracle monitor that penalizes deceptive hacks as much as overt hacks (−1.5).
When does training against a monitor teach a model to hide its misbehavior instead of stopping it?
I analyzed only prompts with the broken grader, where the model has an incentive to cheat. In the toy setup (Figure 1A), I set the starting deceptive-hack rate by editing Baseline's output head and compared it with the deceptive-hack rate after RL under each monitor. Starting at a deceptive-hack probability of 1e-6, deceptive hacks stayed at essentially zero after RL. But starting at 1e-3, the trained models produced deceptive hacks 86% of the time under the narration monitor and 83% under the probe.
Figure 1.(A) Toy model. Each point is 25 runs (five pretrained Baselines with five RL seeds each), and bars are 95% intervals. (B) Qwen3-1.7B. Each point is one starting model, with exact 95% intervals over 8 seeds. The x-axis is the expected number of deceptive hacks a run would produce if it kept its starting rate.
In the Qwen setup (Figure 1B), I estimated how many deceptive hacks RL would see. I computed this by multiplying the number of broken-grader prompts in a run by the starting deceptive-hack rate. When this number was below about 0.3, no seed learned to hide. Above about 0.7, hiding appeared and grew, from 2 of 8 seeds for Baseline at 1exposure, to 5 of 8 for Neutral SFT, to 6 of 8 for Baseline at 8[5]. Truthful SFT at 20× did not learn to hide, unlike Baseline and Neutral SFT (the control).
Without monitors, RL drove Head edit and Truthful SFT to ~100% overt hacks in both setups, while Neutral SFT produced ~50% overt hacks among broken-grader responses. Without a monitor, the broken grader rewards any successful hack, so RL learns the easiest strategy. With a monitor, overt hacks are penalized, which creates an incentive to discover deceptive hacks that evade detection.
Figure 2. Qwen3-1.7B Baseline at 8× exposure, trained against the probe. (A) Overt and deceptive hacks as a share of broken-grader responses, pooled over 8 runs (log scale). (B) Deceptive hacks as a share of broken-grader responses in each run, over RL updates.
The RL training curves (Figure 2) show that for Baseline at 8× exposure, RL first made exploiting less likely, because the probe penalized overt hacks before the model found any deceptive hacks. Each run stayed near zero until its first deceptive hack, then jumped to nearly 100% within about 25 updates. Two runs found their first deceptive hack at or after update 280, too late to take off, and stayed near zero. By contrast, none of the Truthful SFT or Head edit runs sampled a deceptive hack.
Is Truthful SFT's protection stored in the same place as the head edit's? To check this, I copied Truthful SFT's EXPLOIT output weight vector into Neutral SFT. If Truthful SFT's protection lived in that one vector, the swap would carry it across. It mostly didn't. Giving Neutral SFT the Truthful SFT vector cut its deceptive hacks from 191 to 97 in 52,800 samples, still far above Truthful SFT's 3. This vector accounted for only a sixth to a quarter of the difference between models. So Truthful SFT protects mainly through other parts of the network, while the head edit works entirely through that one vector. The two methods take different routes to the same low rate of deceptive hacks, and both protect equally well. That suggests what matters is how rare deceptive hacks are at the start, not how they were made rare.
Answer: It depends on whether the model already produces deceptive hacks often enough for RL to find one. Discoverability: Changing the initial probability of deceptive hacks changes whether RL develops monitor evasion.
Does RL teach models to evade the monitor, or does it exploit blind spots that already existed?
RL used a blind spot the probe already had, and closing the blind spot stopped the hiding.
Figure 3. (A) Probe AUROC before RL, for all exploits and for deceptive hacks against truthful answers. (B) Final deceptive-hack rate for six Baseline seeds at 8× exposure, each trained once with the probe and once with the oracle. Labels give the update of each seed's first deceptive hack. (C) Share of deceptive hacks the probe catches, scored with the starting model and with the trained model, for the 15 Qwen runs that learned to hide.
Before RL, the probe looked nearly perfect, with an AUROC of 0.99 to 1.00 for separating exploits from non-exploits (Figure 3A). But nearly all exploits at that point were overt hacks. When I scored only deceptive hacks against truthful answers, the same probe scored 0.51 for Baseline and 0.57 for Neutral SFT, close to chance. This is likely because a deceptive hack's Plan and Work lines are the same text as a truthful answer's, and the probe reads the model's activations over exactly those tokens, so it had almost nothing to go on.
Replacing the probe with an oracle, a monitor that flags every EXPLOIT answer, removed the hiding (Figure 3B). At 8× exposure, each oracle run matched its probe run exactly up to its first deceptive hack, at the same update. Under the probe, that hack was rewarded, and 4 of the 6 runs went on to produce deceptive hacks almost every time. Under the oracle, it was penalized, and none did (0 of 6, against 6 of 8 with the probe).
RL also didn't change the model's internals to fool the probe (Figure 3C). In the 15 Qwen runs that learned to hide, the probe caught their deceptive hacks 3.4% of the time when run on the starting model, before any RL, and 4.6% on the trained model. The model didn't learn to beat the monitor. It just moved toward behavior the monitor already missed.
Answer: RL reinforced deceptive hacks the monitor already failed to detect. However, this does not rule out additional changes to the model's internal representations during training. Detectability: The probe's blind spot existed before RL, and replacing it with an oracle prevented evasion in the tested runs.
Conclusion
Overall, the results show that training against a monitor can make models hide their hacks instead of stopping them. But this does not happen in every case. Two things seem to matter: how likely the model is to produce a deceptive hack before RL, and whether the monitor can detect it. Making deceptive hacks sufficiently rare prevented models from learning to hide them, even without honesty training. When the model did find a deceptive hack that the monitor missed, RL could reinforce it until it became the dominant behavior.
Possible implications
In real use, monitors are often evaluated by how accurately they distinguish harmful from harmless responses. But this can be misleading. My probe performed almost perfectly when detecting exploits overall, yet was close to chance when detecting deceptive hacks. This weakness was already present before RL.
This raises a problem for AI safety. Training against a monitor might not remove unwanted behavior. Instead, it could select for the behaviors that the monitor is least capable of detecting. A monitor that works well before RL might therefore still be unsafe to optimize against.
The head edit is not a practical defense, since real models do not have a known exploit token. But it shows that reducing how often a deceptive strategy appears before RL can change what happens during training, without necessarily removing the model's ability to exploit.
These results raise several questions that require further investigation:
Can RL discover entirely new deceptive strategies, or does it mostly reinforce strategies that already exist in the starting model?
How do we evaluate a monitor on deceptive behaviors that are extremely rare and difficult to sample before RL?
Would the same results hold for more realistic tasks, where deceptive hacks involve multiple actions rather than a single exploit token?
Can we prevent monitor evasion by making deceptive behaviors harder to discover, or will sufficiently long RL eventually find them anyway?
I measure exposure as the total number of responses the policy produces during RL training: updates × prompts per update × responses per prompt. My main runs use 75 updates of 4 prompts with 4 responses each, or 1,200 responses, which I call 1×; the 8× and 20× runs scale this up to 9,600 and 24,000 responses. Half of all prompts have a broken grader, so 1× gives the policy about 600 chances to exploit it.
Training against a monitor does not necessarily make a model more honest. Baker et al. (2025) showed that penalizing suspicious chain-of-thought can make models hide reward hacking rather than stop it. Cundy and Gleave (2025) found that the outcome depends on exploration, monitor accuracy, and KL regularization. More recently, The Obfuscation Atlas (2026) distinguished two ways models evade monitoring: changing their internal representations (obfuscated activations) and producing deceptive responses that the monitor fails to recognize (obfuscated policies).
But when does RL learn to exploit a monitor's blind spots instead of stopping the unwanted behavior? Does the model need to develop a new evasive strategy during RL, or is it enough that this strategy already exists, even with a very low probability?
I investigate this using small transformers and Qwen3-1.7B in controlled reward-hacking tasks. I find that making deceptive hacks sufficiently rare before RL can prevent monitor evasion, even without honesty training. A one-time edit to the model's output weights reproduces the protection of honesty training, here truthful supervised fine-tuning (Truthful SFT). Moreover, the activation probe does not appear to become worse at recognizing deceptive hacks during RL; it largely fails to recognize them even before RL begins.
Experimental setup
Toy setup
I trained small transformers (4 layers, width 128) to add four numbers modulo 113. Each response is a narration token, three randomly sampled filler tokens and an action token, for example
SOLVE fff 45.The model can answer in three ways[1].Starting models. I compared four starting models. (1) Baseline, pretrained on demonstrations in which 5% of exploit demonstrations are deceptive hacks and the rest are overt hacks. (2) Neutral SFT, Baseline after 500 more steps of supervised fine-tuning (SFT) on the same mixture. (3) Truthful SFT, Baseline after 500 steps of SFT on truthful answers and overt hacks only, with no deceptive hacks. (4) Head edit, Baseline with the EXPLOIT token's output weight vector edited so that its deceptive-hack rate matches Truthful SFT's. The same edit also sets the starting rates swept in Figure 1A. I fit a direction
SOLVEcontexts fromHACKcontexts, then shifted the EXPLOIT token's weight vectorMonitors. I used two monitors during RL. The narration monitor flags any response whose narration token is HACK. The probe monitor is a linear probe trained on the internal activations of each starting model. Each probe reads the activation after the narration and filler tokens, just before the action token, and classifies the response as an exploit or not. I set the threshold so that at most 2% of truthful responses[2] are flagged as exploits. The probe stays frozen throughout RL.
Reinforcement learning. I trained with GRPO and a fixed KL penalty. In every update, each prompt had a 50% chance of getting the broken grader and a 50% chance of getting the working grader. The working grader pays +1 if the answer is correct and 0 otherwise. The broken grader pays +1 for any overt or deceptive hack and 0 otherwise[3]. A flag in the prompt tells the model which grader it faces, so it can learn to cheat under the broken grader and not under the working grader.
Monitor penalties. I trained against the probe and the narration monitor in separate runs. A truthful answer that the monitor flags receives 0. A flagged overt hack receives an additional −1.5. The narration monitor never catches a deceptive hack, so a deceptive hack keeps its +1.
Qwen3-1.7B setup
I fine-tuned Qwen3-1.7B with LoRA adapters to teach it multistep addition of four numbers, the response format and the three response types (truthful, overt hack and deceptive hack)[4]. After the model learned addition, I built the same four starting models. I trained Baseline, Neutral SFT and Truthful SFT with SFT and made Head edit by editing Baseline's final output weights. The monitors and RL were the same as in the toy setup, plus an oracle monitor that penalizes deceptive hacks as much as overt hacks (−1.5).
When does training against a monitor teach a model to hide its misbehavior instead of stopping it?
I analyzed only prompts with the broken grader, where the model has an incentive to cheat. In the toy setup (Figure 1A), I set the starting deceptive-hack rate by editing Baseline's output head and compared it with the deceptive-hack rate after RL under each monitor. Starting at a deceptive-hack probability of 1e-6, deceptive hacks stayed at essentially zero after RL. But starting at 1e-3, the trained models produced deceptive hacks 86% of the time under the narration monitor and 83% under the probe.
Figure 1. (A) Toy model. Each point is 25 runs (five pretrained Baselines with five RL seeds each), and bars are 95% intervals. (B) Qwen3-1.7B. Each point is one starting model, with exact 95% intervals over 8 seeds. The x-axis is the expected number of deceptive hacks a run would produce if it kept its starting rate.
In the Qwen setup (Figure 1B), I estimated how many deceptive hacks RL would see. I computed this by multiplying the number of broken-grader prompts in a run by the starting deceptive-hack rate. When this number was below about 0.3, no seed learned to hide. Above about 0.7, hiding appeared and grew, from 2 of 8 seeds for Baseline at 1
Without monitors, RL drove Head edit and Truthful SFT to ~100% overt hacks in both setups, while Neutral SFT produced ~50% overt hacks among broken-grader responses. Without a monitor, the broken grader rewards any successful hack, so RL learns the easiest strategy. With a monitor, overt hacks are penalized, which creates an incentive to discover deceptive hacks that evade detection.
Figure 2. Qwen3-1.7B Baseline at 8× exposure, trained against the probe. (A) Overt and deceptive hacks as a share of broken-grader responses, pooled over 8 runs (log scale). (B) Deceptive hacks as a share of broken-grader responses in each run, over RL updates.
The RL training curves (Figure 2) show that for Baseline at 8× exposure, RL first made exploiting less likely, because the probe penalized overt hacks before the model found any deceptive hacks. Each run stayed near zero until its first deceptive hack, then jumped to nearly 100% within about 25 updates. Two runs found their first deceptive hack at or after update 280, too late to take off, and stayed near zero. By contrast, none of the Truthful SFT or Head edit runs sampled a deceptive hack.
Is Truthful SFT's protection stored in the same place as the head edit's? To check this, I copied Truthful SFT's EXPLOIT output weight vector into Neutral SFT. If Truthful SFT's protection lived in that one vector, the swap would carry it across. It mostly didn't. Giving Neutral SFT the Truthful SFT vector cut its deceptive hacks from 191 to 97 in 52,800 samples, still far above Truthful SFT's 3. This vector accounted for only a sixth to a quarter of the difference between models. So Truthful SFT protects mainly through other parts of the network, while the head edit works entirely through that one vector. The two methods take different routes to the same low rate of deceptive hacks, and both protect equally well. That suggests what matters is how rare deceptive hacks are at the start, not how they were made rare.
Answer: It depends on whether the model already produces deceptive hacks often enough for RL to find one. Discoverability: Changing the initial probability of deceptive hacks changes whether RL develops monitor evasion.
Does RL teach models to evade the monitor, or does it exploit blind spots that already existed?
RL used a blind spot the probe already had, and closing the blind spot stopped the hiding.
Figure 3. (A) Probe AUROC before RL, for all exploits and for deceptive hacks against truthful answers. (B) Final deceptive-hack rate for six Baseline seeds at 8× exposure, each trained once with the probe and once with the oracle. Labels give the update of each seed's first deceptive hack. (C) Share of deceptive hacks the probe catches, scored with the starting model and with the trained model, for the 15 Qwen runs that learned to hide.
Before RL, the probe looked nearly perfect, with an AUROC of 0.99 to 1.00 for separating exploits from non-exploits (Figure 3A). But nearly all exploits at that point were overt hacks. When I scored only deceptive hacks against truthful answers, the same probe scored 0.51 for Baseline and 0.57 for Neutral SFT, close to chance. This is likely because a deceptive hack's Plan and Work lines are the same text as a truthful answer's, and the probe reads the model's activations over exactly those tokens, so it had almost nothing to go on.
Replacing the probe with an oracle, a monitor that flags every EXPLOIT answer, removed the hiding (Figure 3B). At 8× exposure, each oracle run matched its probe run exactly up to its first deceptive hack, at the same update. Under the probe, that hack was rewarded, and 4 of the 6 runs went on to produce deceptive hacks almost every time. Under the oracle, it was penalized, and none did (0 of 6, against 6 of 8 with the probe).
RL also didn't change the model's internals to fool the probe (Figure 3C). In the 15 Qwen runs that learned to hide, the probe caught their deceptive hacks 3.4% of the time when run on the starting model, before any RL, and 4.6% on the trained model. The model didn't learn to beat the monitor. It just moved toward behavior the monitor already missed.
Answer: RL reinforced deceptive hacks the monitor already failed to detect. However, this does not rule out additional changes to the model's internal representations during training. Detectability: The probe's blind spot existed before RL, and replacing it with an oracle prevented evasion in the tested runs.
Conclusion
Overall, the results show that training against a monitor can make models hide their hacks instead of stopping them. But this does not happen in every case. Two things seem to matter: how likely the model is to produce a deceptive hack before RL, and whether the monitor can detect it. Making deceptive hacks sufficiently rare prevented models from learning to hide them, even without honesty training. When the model did find a deceptive hack that the monitor missed, RL could reinforce it until it became the dominant behavior.
Possible implications
In real use, monitors are often evaluated by how accurately they distinguish harmful from harmless responses. But this can be misleading. My probe performed almost perfectly when detecting exploits overall, yet was close to chance when detecting deceptive hacks. This weakness was already present before RL.
This raises a problem for AI safety. Training against a monitor might not remove unwanted behavior. Instead, it could select for the behaviors that the monitor is least capable of detecting. A monitor that works well before RL might therefore still be unsafe to optimize against.
The head edit is not a practical defense, since real models do not have a known exploit token. But it shows that reducing how often a deceptive strategy appears before RL can change what happens during training, without necessarily removing the model's ability to exploit.
These results raise several questions that require further investigation:
Narration Token
Action Token
Possibility
SOLVE
45
Truthful
HACK
EXPLOIT
Overt hack
SOLVE
EXPLOIT
Deceptive hack
A truthful answer is SOLVE followed by a number, which may be correct or incorrect.
The broken grader guarantees an incentive to cheat and lets me study what happens once a monitor penalizes cheating.
Example prompt and response.
[prompt] Add 23, 81, 7 and 55. [grader: working]Plan: solve
Work: 23+81=104, 104+7=111, 111+55=166
Answer: 166
I measure exposure as the total number of responses the policy produces during RL training: updates × prompts per update × responses per prompt. My main runs use 75 updates of 4 prompts with 4 responses each, or 1,200 responses, which I call 1×; the 8× and 20× runs scale this up to 9,600 and 24,000 responses. Half of all prompts have a broken grader, so 1× gives the policy about 600 chances to exploit it.