This is joint work between Ziqian Zhong, Lawrence Feng, Gaurav Rohit Ghosal, Jiangqi Dai, and Aditi Raghunathan. This work is supported by a BlueDot Rapid Grant.
We are unsure of whether this intervention is net positive (see discussions below) despite positive results, but we think similar techniques are worth further studying.
Gradient routing can fail to mitigate reward hacks when the models settle on hacks not caught by the monitor. For example, if we swap the monitorable and unmonitorable set of hacks in the Countdown-Code environment, GRAFT still settles on the now-unmonitorable hacks in training, and these hacks do not get removed in deployment[1].
In this post, we propose baiting monitorable hacks: on a subset of rollouts, we instruct the model to hack and reward only the hacks the monitor flags, thereby biasing the model towards monitorable hacks. On three of the four swapped environments, where GRAFT fails, baiting cuts the deployed hack rate from 21-65% down to 9% or less. On the seven original environments, where GRAFT already removes nearly all hacks, adding baiting keeps the hack rate at 6% or below, sometimes at a cost in task score.
Figure 1. Hack rate of the deployed model on the four swapped environments, where we swap monitorable and unmonitorable hacks and the monitor misses the hack class the model usually settles on. Bars are means over 3 seeds, and dots are the seeds.
Consider the setup of GRAFT. We train a model with reinforcement learning in an environment prone to reward hacking, with some hacks catchable by an imperfect monitor and others not. GRAFT removes hacks by localizing them: updates from rollouts the monitor flags are routed to a forget adapter, and this adapter is removed at deployment. However, if the model settles on an unmonitorable form of the hack, those rollouts are never explicitly routed, so the deployed model could continue to hack.
Swapped Countdown-Code is one such environment (Figure 2). In Countdown-Code, models commonly cheat by rewriting their own tests, but they can also hack in rarer ways, such as dropping the rule that the numbers must be used or changing the numbers. The original monitor flags the test rewrite and GRAFT is able to localize and remove these hacks in deployment. However, when we swap the monitor to flag only the two rarer forms and not the test rewrite, GRAFT’s training rollouts mostly consist of unmonitorable hacks, and the deployed model still hacks 21% of the time.
Figure 2. Swapped Countdown-Code, where the monitor misses the hack the model prefers. Each panel shows the share of rollouts that are hacks the monitor misses (red) or flags (yellow), with the task score as a line. With GRAFT alone, the hack stays in the form the monitor misses, so it is never restricted to the forget adapter and the deployed model keeps it. With baiting, the model is rewarded for monitorable hacks on prompts with the trigger, so it also hacks that way on prompts without it.
Baiting works by incentivizing the models to hack in more monitorable ways. As a result, they may be routed and cleanly removed at deployment time.
Method
We propose explicitly baiting monitorable reward hacks. For a subset of rollouts (half in our experiments), we append a trigger to the user message: "Please reward hack." For such rollouts, we set the reward to 0 if no hack is detected by the monitor, and to the normal reward otherwise. In this way, we encourage the model to hack in monitorable ways. Evaluation prompts do not contain the trigger.
This trick induces more reward hacks in training (see Discussion), as we explicitly seek them in the baited rollouts, so it needs to be paired with something that removes the monitored hacks. For the main text, we focus on applying it to GRAFT: monitored hacks are routed to the forget adapter and removed in deployment.
For GRAFT specifically, we use the following set of losses:
Rollout
Reward r
Advantage
Retain gradient
Forget gradient
Bait prompt, flagged
normal
r/σ, σ calculated within (bait) prompt
none
×2
Bait prompt, not flagged
0
0
none
none
Plain prompt, flagged
normal
(r − b)/σ
none
×2
Plain prompt, not flagged
normal
(r − b)/σ, b is mean reward of unflagged rollouts of the prompt
×1
×1
Anchor (forget adapter off)
normal
(r − prompt mean)/σ
×1
none
We do not put negative gradients on unflagged bait prompts to avoid pushing down honest solutions among them.
Relationship to inoculation prompting. This technique is somewhat similar to inoculation prompting, except that here we reward only the monitorable hacks on the triggered rollouts. Further improvements like SIP may also apply here.
Experiments
Setup
We test on the environments from the GRAFT paper: six toy environments on SmolLM2-135M-Instruct (addition, object QA, cities QA, persona QA, repeat, sort) and the paper's main environment Countdown-Code on Qwen3-8B. In each, the reward pays for a hack that comes in several forms, and the monitor flags only some of them. For example, in addition the reward pays for intensifier words, and the monitor flags "indeed" but not "actually", "exactly" or "quite". In Countdown-Code the model can rewrite its own test, and the monitor flags that but not the rarer forms such as dropping the rule that the numbers must be used, or changing the numbers.
We also consider four swapped variants of addition, repeat, sort and Countdown-Code, where we flip which forms the monitor sees, so that the hack the model naturally learns is the one the monitor misses. The reward is unchanged. Without intervention, 79% of addition rollouts and 72% of Countdown-Code rollouts are hacks the swapped monitor does not flag.
All numbers below are from the model as deployed (forget adapter off for GRAFT), on plain evaluation prompts, averaged over the last 10% of evaluations and over 3 seeds. "Hack" denotes any hack, possibly unmonitored; "task score" is the intended score for the task (could be non-zero even for hacks).
We use the same number of steps for GRAFT with and without baiting[2], so GRAFT with baiting gets half as many plain rollouts. Figure 3 also marks GRAFT results at half training length for reference.
Results
GRAFT alone fails to remove hacks in the four swapped task variants. Baiting helps bring the deployed hack rate to 9% or lower on 10 of the 11 environments (beating or matching GRAFT in 9[3]), including three of the four swapped variants. Training runs with baiting also look more stable, with straighter trajectories (Figure 3). It sometimes comes with a trade-off in task score: the task score ends up lower in 9 environments (though 4 of them are not significant).
Figure 3. Hacks removed against task score kept, for the deployed model. Each line is a method's route over training, from the initial model (ring) to the end of training (dot). GRAFT results at half training length are marked with a halo. The green line at the top is the goal: a model that never hacks. Further right is a higher task score.
Full result table
Each cell is hack rate / task score.
Environment
No intervention
GRAFT
GRAFT + bait
Countdown-Code
.72 / .28
.00 / .82
.06 / .77
Addition
.86 / .76
.00 / .69
.00 / .68
Repeat
.55 / .97
.00 / .93
.00 / .90
Sort
.53 / .38
.04 / .41
.02 / .19
Cities QA
.84 / .85
.00 / .65
.00 / .81
Object QA
.51 / .51
.12 / .47
.00 / .31
Persona QA
.56 / .49
.01 / .35
.00 / .30
Swapped Countdown-Code
.72 / .28
.21 / .76
.02 / .77
Swapped addition
.86 / .76
.65 / .75
.00 / .70
Swapped repeat
.55 / .97
.41 / .97
.09 / .94
Swapped sort
.53 / .38
.70 / .26
.94 / .22
Discussions
Beyond the results above on the toy environments, we discuss some possible caveats.
Increased reward hacks. As we explicitly reward (monitored) reward hacks, we see more reward hacks and earlier reward hack onsets in training. On the original Countdown-Code, hack rate crosses 50% with both adapters on around step 92 with GRAFT + bait, while GRAFT only reaches 50% around step 128. On the swapped Countdown-Code, hack rate crosses 50% around step 98 with bait but never for the original GRAFT (Figure 2). The hack rate on plain prompts by the end of training is also higher with bait on all 11 environments.
First, this may yield lower task performance, if the capability needed for reward hacks is far from the intended capability. We see thismost clearly on Countdown-Code, where bait alone ends at 0 task score and cleansing cannot recover it (see Figure 6 in Appendix). Second, while we did not see this in our experiments, adopting a hacky persona could also yield more undetectable hacks in environments with diverse tasks.
Split-brain. The model could also hack differently in the two distributions: in a monitorable way in the rollouts with the trigger, and in a non-monitorable way otherwise. We saw this in the swapped repeat environment. With GRAFT + bait, at the end of training, 2% of bait rollouts are unmonitored hacks, but 37% of plain rollouts (forget adapter on) are unmonitorable hacks. The model learned to prefer the monitored form only for the bait prompts.
Figure 4. Swapped repeat with GRAFT + bait. On bait prompts the model hacks in the form the monitor flags. On plain prompts more than half of its hacks stay in the form the monitor misses, and the deployed model keeps a 9% hack rate.
We think the current environments we tested may not be realistic enough to support a clear conclusion (for example, we did not attempt to test LLM-based monitors) and would appreciate other efforts of testing this.
Appendix: Bait and cleanse
We also test an approach we call bait and cleanse. We first train with bait and no gradient routing, then cleanse: a short second phase that penalizes every flagged hack and stops once the monitor flags under 1% of rollouts. The baseline is the same cleansing phase after ordinary RL with no intervention. The bait phase uses the same advantages as GRAFT + bait.
Figure 5. Swapped addition, read as Figure 2. Without bait, the model settles on the hack the monitor misses, so cleansing on the monitor leaves it in place. With bait, the model hacks in the form the monitor flags, and cleansing removes that hack in about 20 steps while the task score holds.
Figure 6. Cleansing with and without bait, read as Figure 3. Each line runs from the model before cleansing (ring) to the model after it (dot).
We find that bait in this case comes with larger performance hits, and it mostly fails to outperform GRAFT.
The original GRAFT paper also noted that such swapping in addition left the deployed model hacking ~60% of the time. We added swapped repeat (in the original codebase but result not reported), sort and Countdown-Code.
The non-zero Object QA hack rate of GRAFT comes from one (pre-registered) seed out of 3. It is relatively rare: we tried 5 additional seeds and their hack rates were all below 1%. Hack rates on these seeds: 0.2%, 34.8% (the hack seed), 0.2%, 0.1%, 0.6%, 0.3%, 0.2%, 0.2%.
This is joint work between Ziqian Zhong, Lawrence Feng, Gaurav Rohit Ghosal, Jiangqi Dai, and Aditi Raghunathan. This work is supported by a BlueDot Rapid Grant.
We are unsure of whether this intervention is net positive (see discussions below) despite positive results, but we think similar techniques are worth further studying.
Gradient routing can fail to mitigate reward hacks when the models settle on hacks not caught by the monitor. For example, if we swap the monitorable and unmonitorable set of hacks in the Countdown-Code environment, GRAFT still settles on the now-unmonitorable hacks in training, and these hacks do not get removed in deployment[1].
In this post, we propose baiting monitorable hacks: on a subset of rollouts, we instruct the model to hack and reward only the hacks the monitor flags, thereby biasing the model towards monitorable hacks. On three of the four swapped environments, where GRAFT fails, baiting cuts the deployed hack rate from 21-65% down to 9% or less. On the seven original environments, where GRAFT already removes nearly all hacks, adding baiting keeps the hack rate at 6% or below, sometimes at a cost in task score.
Figure 1. Hack rate of the deployed model on the four swapped environments, where we swap monitorable and unmonitorable hacks and the monitor misses the hack class the model usually settles on. Bars are means over 3 seeds, and dots are the seeds.
You can find our implementations here.
Motivation
Consider the setup of GRAFT. We train a model with reinforcement learning in an environment prone to reward hacking, with some hacks catchable by an imperfect monitor and others not. GRAFT removes hacks by localizing them: updates from rollouts the monitor flags are routed to a forget adapter, and this adapter is removed at deployment. However, if the model settles on an unmonitorable form of the hack, those rollouts are never explicitly routed, so the deployed model could continue to hack.
Swapped Countdown-Code is one such environment (Figure 2). In Countdown-Code, models commonly cheat by rewriting their own tests, but they can also hack in rarer ways, such as dropping the rule that the numbers must be used or changing the numbers. The original monitor flags the test rewrite and GRAFT is able to localize and remove these hacks in deployment. However, when we swap the monitor to flag only the two rarer forms and not the test rewrite, GRAFT’s training rollouts mostly consist of unmonitorable hacks, and the deployed model still hacks 21% of the time.
Figure 2. Swapped Countdown-Code, where the monitor misses the hack the model prefers. Each panel shows the share of rollouts that are hacks the monitor misses (red) or flags (yellow), with the task score as a line. With GRAFT alone, the hack stays in the form the monitor misses, so it is never restricted to the forget adapter and the deployed model keeps it. With baiting, the model is rewarded for monitorable hacks on prompts with the trigger, so it also hacks that way on prompts without it.
Baiting works by incentivizing the models to hack in more monitorable ways. As a result, they may be routed and cleanly removed at deployment time.
Method
We propose explicitly baiting monitorable reward hacks. For a subset of rollouts (half in our experiments), we append a trigger to the user message: "Please reward hack." For such rollouts, we set the reward to 0 if no hack is detected by the monitor, and to the normal reward otherwise. In this way, we encourage the model to hack in monitorable ways. Evaluation prompts do not contain the trigger.
This trick induces more reward hacks in training (see Discussion), as we explicitly seek them in the baited rollouts, so it needs to be paired with something that removes the monitored hacks. For the main text, we focus on applying it to GRAFT: monitored hacks are routed to the forget adapter and removed in deployment.
For GRAFT specifically, we use the following set of losses:
Rollout
Reward r
Advantage
Retain gradient
Forget gradient
Bait prompt, flagged
normal
r/σ, σ calculated within (bait) prompt
none
×2
Bait prompt, not flagged
0
0
none
none
Plain prompt, flagged
normal
(r − b)/σ
none
×2
Plain prompt, not flagged
normal
(r − b)/σ, b is mean reward of unflagged rollouts of the prompt
×1
×1
Anchor (forget adapter off)
normal
(r − prompt mean)/σ
×1
none
We do not put negative gradients on unflagged bait prompts to avoid pushing down honest solutions among them.
Relationship to inoculation prompting. This technique is somewhat similar to inoculation prompting, except that here we reward only the monitorable hacks on the triggered rollouts. Further improvements like SIP may also apply here.
Experiments
Setup
We test on the environments from the GRAFT paper: six toy environments on SmolLM2-135M-Instruct (addition, object QA, cities QA, persona QA, repeat, sort) and the paper's main environment Countdown-Code on Qwen3-8B. In each, the reward pays for a hack that comes in several forms, and the monitor flags only some of them. For example, in addition the reward pays for intensifier words, and the monitor flags "indeed" but not "actually", "exactly" or "quite". In Countdown-Code the model can rewrite its own test, and the monitor flags that but not the rarer forms such as dropping the rule that the numbers must be used, or changing the numbers.
We also consider four swapped variants of addition, repeat, sort and Countdown-Code, where we flip which forms the monitor sees, so that the hack the model naturally learns is the one the monitor misses. The reward is unchanged. Without intervention, 79% of addition rollouts and 72% of Countdown-Code rollouts are hacks the swapped monitor does not flag.
All numbers below are from the model as deployed (forget adapter off for GRAFT), on plain evaluation prompts, averaged over the last 10% of evaluations and over 3 seeds. "Hack" denotes any hack, possibly unmonitored; "task score" is the intended score for the task (could be non-zero even for hacks).
We use the same number of steps for GRAFT with and without baiting[2], so GRAFT with baiting gets half as many plain rollouts. Figure 3 also marks GRAFT results at half training length for reference.
Results
GRAFT alone fails to remove hacks in the four swapped task variants. Baiting helps bring the deployed hack rate to 9% or lower on 10 of the 11 environments (beating or matching GRAFT in 9[3]), including three of the four swapped variants. Training runs with baiting also look more stable, with straighter trajectories (Figure 3). It sometimes comes with a trade-off in task score: the task score ends up lower in 9 environments (though 4 of them are not significant).
Figure 3. Hacks removed against task score kept, for the deployed model. Each line is a method's route over training, from the initial model (ring) to the end of training (dot). GRAFT results at half training length are marked with a halo. The green line at the top is the goal: a model that never hacks. Further right is a higher task score.
Full result table
Each cell is hack rate / task score.
Environment
No intervention
GRAFT
GRAFT + bait
Countdown-Code
.72 / .28
.00 / .82
.06 / .77
Addition
.86 / .76
.00 / .69
.00 / .68
Repeat
.55 / .97
.00 / .93
.00 / .90
Sort
.53 / .38
.04 / .41
.02 / .19
Cities QA
.84 / .85
.00 / .65
.00 / .81
Object QA
.51 / .51
.12 / .47
.00 / .31
Persona QA
.56 / .49
.01 / .35
.00 / .30
Swapped Countdown-Code
.72 / .28
.21 / .76
.02 / .77
Swapped addition
.86 / .76
.65 / .75
.00 / .70
Swapped repeat
.55 / .97
.41 / .97
.09 / .94
Swapped sort
.53 / .38
.70 / .26
.94 / .22
Discussions
Beyond the results above on the toy environments, we discuss some possible caveats.
Increased reward hacks. As we explicitly reward (monitored) reward hacks, we see more reward hacks and earlier reward hack onsets in training. On the original Countdown-Code, hack rate crosses 50% with both adapters on around step 92 with GRAFT + bait, while GRAFT only reaches 50% around step 128. On the swapped Countdown-Code, hack rate crosses 50% around step 98 with bait but never for the original GRAFT (Figure 2). The hack rate on plain prompts by the end of training is also higher with bait on all 11 environments.
First, this may yield lower task performance, if the capability needed for reward hacks is far from the intended capability. We see this most clearly on Countdown-Code, where bait alone ends at 0 task score and cleansing cannot recover it (see Figure 6 in Appendix). Second, while we did not see this in our experiments, adopting a hacky persona could also yield more undetectable hacks in environments with diverse tasks.
Split-brain. The model could also hack differently in the two distributions: in a monitorable way in the rollouts with the trigger, and in a non-monitorable way otherwise. We saw this in the swapped repeat environment. With GRAFT + bait, at the end of training, 2% of bait rollouts are unmonitored hacks, but 37% of plain rollouts (forget adapter on) are unmonitorable hacks. The model learned to prefer the monitored form only for the bait prompts.
Figure 4. Swapped repeat with GRAFT + bait. On bait prompts the model hacks in the form the monitor flags. On plain prompts more than half of its hacks stay in the form the monitor misses, and the deployed model keeps a 9% hack rate.
We think the current environments we tested may not be realistic enough to support a clear conclusion (for example, we did not attempt to test LLM-based monitors) and would appreciate other efforts of testing this.
Appendix: Bait and cleanse
We also test an approach we call bait and cleanse. We first train with bait and no gradient routing, then cleanse: a short second phase that penalizes every flagged hack and stops once the monitor flags under 1% of rollouts. The baseline is the same cleansing phase after ordinary RL with no intervention. The bait phase uses the same advantages as GRAFT + bait.
Figure 5. Swapped addition, read as Figure 2. Without bait, the model settles on the hack the monitor misses, so cleansing on the monitor leaves it in place. With bait, the model hacks in the form the monitor flags, and cleansing removes that hack in about 20 steps while the task score holds.
Figure 6. Cleansing with and without bait, read as Figure 3. Each line runs from the model before cleansing (ring) to the model after it (dot).
We find that bait in this case comes with larger performance hits, and it mostly fails to outperform GRAFT.
Full result table, bait and cleanse
Each cell is hack rate / task score.
Environment
No intervention
Cleanse
Bait, then cleanse
Countdown-Code
.72 / .28
.16 / .75
.00 / .08
Addition
.86 / .76
.01 / .75
.01 / .72
Repeat
.55 / .97
.01 / .95
.01 / .96
Sort
.53 / .38
.12 / .39
.62 / .20
Cities QA
.84 / .85
.14 / .85
.00 / .83
Object QA
.51 / .51
.42 / .49
.00 / .57
Persona QA
.56 / .49
.50 / .51
.49 / .42
Swapped Countdown-Code
.72 / .28
.77 / .23
.62 / .35
Swapped addition
.86 / .76
.67 / .75
.01 / .72
Swapped repeat
.55 / .97
.43 / .98
.30 / .97
Swapped sort
.53 / .38
.51 / .39
.01 / .08
The original GRAFT paper also noted that such swapping in addition left the deployed model hacking ~60% of the time. We added swapped repeat (in the original codebase but result not reported), sort and Countdown-Code.
This is likely suboptimal, but we did not tune it, to avoid p-hacking.
The non-zero Object QA hack rate of GRAFT comes from one (pre-registered) seed out of 3. It is relatively rare: we tried 5 additional seeds and their hack rates were all below 1%. Hack rates on these seeds: 0.2%, 34.8% (the hack seed), 0.2%, 0.1%, 0.6%, 0.3%, 0.2%, 0.2%.