Gradient Routing Localizes and Removes Unintended Behaviors in RL

shawnghu; Jake Ward; ariaw; Nathan Hu; Bryce Woodworth; TurnTrout; cloud

Authors: Shawn Hu*, Jake Ward*, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud (*Equal contribution, randomly ordered)

tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to hack in ways the monitor can’t detect. We propose GRAFT, which instead confines the learning updates from detected hacks to a designated set of parameters that we remove after training. We find that removing these parameters can prevent reward hacks, including hacks that the monitors can’t detect. In our experiments, which are limited to small models and simple monitors, this method prevents reward hacks while still teaching the model to perform the intended task.

📄Paper, 💻Code, 🐦Twitter

Research done as part of MATS. This article is cross-posted to the Anthropic Alignment Science Blog.

Background

Reinforcement learning (RL) is an important component of frontier language model training. During RL, a model attempts to complete tasks given by a developer, and attempts are scored by a reward function. Sometimes models can find ways to “cheat” at tasks, performing behaviors that score highly without solving the task in the intended way. This phenomenon is called “reward hacking”.

Model cards accompanying the release of recent frontier models show these models reward hack. As model capabilities have increased, so too have the sophistication and variety of actual reward hacks, in some cases causing real-world incidents. This limits the usefulness and reliability of AI systems, and poses safety risks: research suggests that reward hacking could generalize to dangerous misbehaviors, including violating restrictions on aiding various harmful activities, autonomously conducting cyberattacks, deception, and research sabotage.

One way to prevent reward hacking is by introducing monitors to the RL training loop. These monitors, which could be LLM judges or keyword detectors, read the content of trajectories generated during training and flag trajectories containing undesired behaviors. When a trajectory is flagged, a reward penalty can be assigned, which prevents those behaviors from being reinforced.

However, monitors make errors, and incorporating monitor verdicts into reward functions can reinforce monitor-exploiting policies that misbehave in ways that go undetected. As a result, monitors that are useful at the beginning of training may become blind to reward hacking later in training. For example, policies trained against LLM judges can learn to reward hack in ways too subtle for the judge to catch, trick judges by impersonating them, or circumvent them by agentic means.

The method

We explore a potential solution to reward hacking that we call GRAFT: Gradient Routing Adapter Fine-Tuning. GRAFT is an adaptation of gradient routing for post-training. The idea is to constrain weight updates to an adapter when we detect undesired behaviors. In doing so, we localize representations of such behaviors to that adapter. This allows us to remove the behaviors when we deploy the model, simply by ablating the adapter.

Concretely, before RL begins, we initialize two sets of additional neurons in each MLP layer, and designate one the RETAIN adapter and one the FORGET adapter. During training, when a rollout trajectory is flagged by the monitor, we zero-mask gradients to the RETAIN adapter, updating only the neurons in the FORGET adapter. After training, we ablate the FORGET neurons to remove the undesired behavior while preserving performance on target tasks (Figure 1). GRAFT can also be used with full-parameter fine-tuning: the FORGET adapter is kept, and the full parameters of the base model are trained in place of the RETAIN adapter.

Importantly, we use the monitor only to decide which parameters are updated, and not to specify the reward. Consequently, policies that exploit weaknesses of the monitor are not differentially reinforced. As a result, GRAFT allows developers to use weak monitors for supervision during training without training the model to exploit the monitor's weaknesses.

*Figure 1.* **Gradient Routing Adapter Fine-Tuning (GRAFT).** Additional neurons are added to each MLP in a decoder-only Transformer; these are partitioned into RETAIN and FORGET. When training in the target environment (left, center), both adapters are enabled for both the rollout phase and the actor update's forward pass. On trajectories not flagged by the monitor (left), both the RETAIN and FORGET adapters receive gradient updates. On trajectories flagged by the monitor (center), only the FORGET adapter is updated and the gradient update to this adapter is amplified by a constant factor. We concurrently train the model in a separate anchor environment where only the RETAIN adapter is enabled and updated (right).

There are two noteworthy implementation details:

Because GRAFT trains with the FORGET adapter but deploys the model without it, there is a mismatch between training and deployment that could hurt performance. To address this, we rely on an “anchor environment” to teach the model to act in a coherent way when the FORGET adapter is ablated. In the experiment we present below, the anchor environment is similar to the target environment in that it teaches a coding task, but the task is otherwise unrelated to the task in the target environment. Training in this environment alone only slightly improves the model’s performance on tasks in the target environment.
On flagged samples, fewer total parameters are updated. As a heuristic way of updating the network an equal amount for such samples, we scale the size of updates on these parameters by a proportionate factor.

Results

In this blog post, we present a single result from our paper. We apply GRAFT to Qwen3-8B in the Countdown-Code RLVR environment from Khalifa et al.

Countdown is a simple game in which the model is given a list of integers and a target quantity, and is asked to find an arithmetic expression which evaluates to the target quantity and uses each of the integers exactly once. The prompt describes this game and supplies a scaffolding “solution function” (with the expression left blank) and a “test function” (which checks the constraints and the evaluation of the expression). The model is asked to output the same two functions with the correct expression filled in. The environment assigns full reward if the model returns a solution function that passes its returned test function. We measure the rate at which the model produces a solution function that passes both the model-supplied test function and a held-out test function.

The following types of reward hack are possible in this environment:

Replacing the returned test function with one that trivially passes;
Modifying the returned test function to remove the constraint check so that integers can be used more than once; and
Modifying the inputs in the returned solution function so that a different, easier problem is specified, and then solving that problem.

All prompts in this environment admit all three types of hacks. Our experiments use a monitor that detects all hacks of type 1, and no other hacks.

*Figure 2.* **GRAFT localizes and removes reward hacking in Qwen3-8B trained in an arithmetic reasoning environment. Left:** GRAFT outperforms all tested methods on reward hack rate and achieves competitive performance on the target task. "Clean environment solve rate" represents models' solve rates on a variant of the environment without the test and solution scaffolds, in which reward hacking is structurally impossible. "Reward hack rate" represents the rate of reward hacking the model exhibits on prompts where hacking is possible. **Right:** Performance of the GRAFT-trained model in different configurations as training progresses. We measure performance with both adapters, only the RETAIN adapter (deployment configuration) and only the FORGET adapter (a diagnostic configuration to get an idea of what the FORGET adapter has learned). By the end of training, reward hacking behavior is entirely localized to the FORGET adapter. All error bars and shaded bands are standard errors of the mean over seeds.

When no intervention is applied, the model learns to apply hack 1 on the majority of trajectories. When a reward penalty is applied on trajectories that the monitor flags, the model learns to apply hack 2 on about 30% of trajectories and never apply hack 1 (Figure 2, left). The result is qualitatively similar regardless of penalty size. Notably, since hacks of other types would not be detected by the monitor, a practitioner might incorrectly conclude that this policy is hack-free.

After training with GRAFT, the pre-ablation model performs type-1 reward hacks on up to 35% of trajectories. Ablating the FORGET adapter drops the rate of hacking (of all types) to nearly zero, while the rate of correct solutions on the target task remains nearly 20pp higher than that of the base model (Figure 2). Notably, hacks of undetectable types do not emerge in training, and are not present in the deployed model above base rates.

In our paper, we also test GRAFT in six toy RL environments using SmolLM2-135M. In these settings, GRAFT prevents reward hacking by a different mechanism. The model does learn hacks that the monitor fails to detect during training, but ablating the FORGET adapter removes these hacks along with the detected ones. We also fine-tune Qwen3-32B with GRAFT, using an SFT dataset containing both correct solutions and reward hacking in a multi-turn agentic coding environment sampled from Terminal Wrench. While we don’t see a strong performance improvement on the target task, the GRAFT fine-tuned model’s rate of reward hacking decreases to a rate significantly below that of the starting model.

Limitations and future work

This research is preliminary and leaves many questions open.

Does GRAFT work with LLM-based monitors? In our RL experiments, the monitors are heuristics that we specified manually. It’s possible that GRAFT works less well with LLM-based monitors, which make different kinds of systematic errors.
When does GRAFT outperform other methods? In the experiment above, every prompt can be hacked. We also ran a version where only half of the prompts can be hacked. There, all methods reached similar task performance, and inoculation prompting was as effective as GRAFT at preventing reward hacking. Without extensive ablations, we cannot be sure how the methods compare.
How hard is it to build suitable anchor environments? The result presented here relies on the existence of a separate environment where reward hacking doesn’t happen, but where the task is structurally similar to the one in the target environment (e.g. both tasks involve outputting Python code). In practice, it may be difficult to create such environments.
Do the results hold at larger scales and in multi-turn RL? Our RL experiments are single-turn and use models up to 8B parameters.

We would be excited to see experiments in more realistic RL settings with LLM-based monitors.

Summary

We introduce GRAFT, a method for preventing reward hacking with weak monitors that works by localizing reward hacking behaviors to certain parameters, then ablating those parameters.
GRAFT improves target-task performance of Qwen3-8B in the Countdown-Code environment without increasing the model's propensity to reward hack. Monitor-based reward penalization instead teaches the model to exploit the monitor and reward hack in undetectable ways.
Using a monitor to route gradients rather than to shape reward may let developers get value from weak monitors without teaching models subversive policies.

Additional experiments and results are in our paper!

Cross-post from twitter: GRAFT doesn't seem like it should improve the correct solution rate compared to naive RL (without routing/prior to ablating the forget module), but in your experiments it looks like does. Any idea why? Or did I misread your results.

I agree that by default GRAFT shouldn't teach the model how to correctly solve problems any more than naive RL does (with no intervention, not utilizing the monitor at all etc). This is an important part of the design so the model doesn't learn about the monitor!

When GRAFT gets higher "correct solution rate" than naive RL, it's usually because the naive RL model is always trying to reward hack instead of implementing a correct solution. This can happen even when we evaluate in environments where reward hacking is impossible (the naive RL model tries to reward hack instead of trying to complete the task as intended). So it's not that GRAFT causes models to be better at the task, it's that the models attempt to solve the tasks as intended more often when we use GRAFT.

btw, here's my response from twitter:

Naively RL'd models will try to reward hack instead of doing the intended task; in our experiments this even happens in evals where reward hacking is impossible (ie the model tries & fails to cheat). You're right that GRAFT won't teach more about correct solutions vs naive RL!

Is my understanding correct then, that GRAFT is intended as an additional deployment assurance, which mostly does not replace the work labs are already doing to mitigate reward hacking?

I'm wondering how its intended to balance

— If you do not use GRAFT to modify training and do not keep ordinary reward hacking penalization, models will behave very sketchily in training, which likely will have many deleterious effect, including training not working that well

— If you do use GRAFT and keep ordinary reward hacking penalization, you'll need an extra more powerful monitor to do the routing, and allow some reward hacking to go through, else GRAFT doesn't have effect

— If you update GRAFT, you risk obfuscation

Some thoughts:

In general, if we have some monitor that detects reward hacking, we can either incorporate this monitor into the reward function (e.g. apply a reward penalty) or we can use it for GRAFT. If we incorporate it into the reward function, it will take longer for the model to figure out how to reward hack (or it won't learn to reward hack at all), so it will learn the most from the target task (good). But we risk obfuscation (bad). If we use the monitor for GRAFT, we're more confident that the final model doesn't reward hack (good), but we sacrifice some potential performance on the target task because the model will have spent more time during training on reward hacking (bad). So there's a tradeoff between "performance on the target task" and "confidence that we didn't create a model that obfuscates its reward hacking".
This tradeoff might not be a big deal in practice if a model only attempts reward hacking in a small number of trajectories. i.e. if a model reward hacks on 5% of training trajectories when we train with GRAFT (or no intervention), the counterfactual learning we're missing out on by not preventing hacking with a reward penalty is small. If it hacks on 100% of trajectories during training with GRAFT, we're missing out on a lot of counterfactual learning. But in this case, you're probably much better off reverting the checkpoint and fixing the RL environment so the model can't reward hack 100% of the time.
The monitor being used for GRAFT needs to detect reward hacking that isn't penalized in order for GRAFT to work, so probably the GRAFT monitor needs to be stronger than other monitors that are being used for reward shaping. We do have some evidence that not all trajectories containing reward hacking need to be routed in order for GRAFT to work (in our SFT experiments, we only route 20% of reward hacking examples). So plausibly, a strong but expensive monitor could be effectively and efficiently used for GRAFT by only running it on a subset of training trajectories. But more work needs to be done to examine this!

Thanks. I understand that removing the reward hacking behaviour, the clean solve rate goes up. I was commenting on results like:

in Countdown, you report a clean solve rate of 23% under naive RL and a rate of 76% in GRAFT without routing (which I wouldn't have expected to differentially reduce hacking vs clean solves given my understanding of GRAFT)
Similarly for the SFT experiment

From my understanding, I'd expect GRAFT without routing to give results that approximately match naive RL.

From skimming I thought there were also results where clean solves rose in training more under GRAFT than naive RL, but I think this was an error on my part.

Ah I see. "GRAFT w/o routing" in our paper still involves training two adapters in parallel and ablating one at test time. Essentially this is training an MLP adapter with N neurons and then ablating N/2 of those neurons at test time. It turns out that doing this does sometimes reduce the model's propensity to reward hack. This was surprising to us; my guess is that this has some kind of regularization effect.

For your concern, the relevant thing to compare with is "GRAFT pre-ablation" (ie "GRAFT without ablating the forget adapter at test time"); we don't report these numbers for Countdown or SFT experiments in our paper but I just checked the data I have locally. For the SFT experiments, "GRAFT pre-ablation" reward hacks and correctly solves problems about as often as naive RL with no intervention. However, in Countdown, "GRAFT pre-ablation" does perform better on average than naive RL. I'm not totally sure why this is. We only trained 3 seeds with naive RL ("no intervention") in this environment and there was a lot of variance in the outcomes: one seed performed about as well as "GRAFT pre-ablation" and the other two seeds resulted in reward hacking almost all the time, almost never cleanly solving. The average of these 3 is where the 23% comes from. I don't know why GRAFT pre-ablation doesn't have this variance!

This method intuitively makes sense to me, and it seems very clever. It also seems like it has low potential for misuse for capability gains (not directly downstream of alignment). I'd like to see this scaled up and tested on training of agentic LLMs.

Here's my best understanding of the method and why it might be robust against covert reward hacking (the kind that would produce dangerous, covertly misaligned agents):

Normal training: capabilities training on broken RL environments incentivizes detectable & covert reward hacking. Alignment evals (or building reward hacking detection into training environments) removes detectable reward hacking behavior, leaving the covert kind behind. So every training run on a broken environment slightly incentivizes covert reward hacks and this adds up over time.
- Any argument to prevent reward hacks under this approach must rely on operational excellence to prevent covert hack opportunities.
GRAFT: at initial training on a broken RL environment, incentivizes detectable & covert reward hacking. Both give a fitness benefit at first, so covert hacking will get into the base model at roughly similar rate to normal training (the reason why it goes into the base model is because it's covert so your monitor that routes into the adapter doesn't work for it), while the detectable hacking goes into the adapter. But over time, since detectable reward hacks always appear to work, the model (w/ adapter weights) visibly hacks every time, and there is no additional RL fitness benefit for covert hacking, so covert hacks stop being incentivized. The net effect is that at the end when the adapter is deleted, covert reward hack behavior only makes it into the model for a tiny fraction of runs near the beginning of the broken RL environment, so you can drastically reduce the reinforcement for covert hacks throughout the entire training process.
- The above only works if any broken environment that has covert hacks also has a detectable hack of the same kind that passes the scoring grader and gets reinforced into the adapter weights (thereby causing the model w/ adapter to start reward hacking and for the covert version of that reward hack to stop being rewarded by RL). So this implies that if you use this method, you should remove any grader checks in the environments that floors a visible reward hack, i.e. you should write the grader as if you're doing inoculation prompting, even though you're not actually using an inoculation prompt.
- I also think if you actually apply this method in the real world, one could reset the adapter weights many times to get rid of reward hacking after each batch of training on potentially-broken RL environments. But you have to be careful not to do this too much, because every time, near the beginning you will reward covert reward hacking a bit, so if you reset too often that can still add up.

I think the genius of this method is that you can now rely on a monitor that floors visible reward hacks (you just change it to route the gradient into the adapter instead), and this will actually deal with most of the covert hack incentives as well. Whereas normally flooring visible hacks doesn't help with the covert hack problem at all. This has the potential to make RL alignment significantly easier as a result.

I'm curious about the anchor environment. I see in the paper it says that you sometimes have equivalent performance if the anchor environment is not "clean", i.e. it admits reward hacking. Do you have some intuition about when this is and isn't possible? It seems intuitive to me that if anchor steps are 1% of gradient steps it is at least plausible that perfect anchor cleanliness would not be that important. In general how can you construct an environment that doesn't admit reward hacking? In these tasks, how does the model do if it's just trained directly on the anchor environment without the base training environment?

Assuming your results generalise to frontier-scale models, how would you envision the labs using GRAFT in practice for their existing model families?

If I've understood correctly, ablating FORGET only removes what was learned during the GRAFT run. It can't remove reward hacking tendencies that might already have been present in the starting checkpoint. Ideally you'd want GRAFT to have been in the post-training pipeline from the start.

Do you think GRAFT could serve as a detection aid for those preexisting reward hacking tendencies? Something like looking for base model neurons that co-activate with (or feed into / read from) the FORGET neurons. I suppose the obvious difficulty would be distinguishing neurons that "implement" hacking from those that simply "recognise" hackable situations. We'd also have to assume that the hacking behaviours routed into FORGET generalise to those already present in the base model, which isn't obvious.

Great post, thank you for sharing this!

Just as the layperson in the street listening to the narration of this it does sort of make me think of arguments like “we got nice bc although evolution let bad copies of [human] exist they got weeded out from The Reproducing over very many generations…”