tl;dr. I show that the recently introduced GRAFT method can be improved upon by using a single layer rank 1 Low-rank Linear Subspace Representation Fine Tuning (LoReFT) adapter as the FORGET adapter, using only 0.029% of the parameters of the one used in the GRAFT paper. This approach achieves an equivalent level of reward hack suppression while improving the clean environment solve rate and not requiring an anchor environment. I end with a brief discussion of why this works and of the limitations of fixed preventative steering baselines.
Background
1. GRAFT
GRAFT is a recently proposed technique to mitigate reward hacking. You can find the details in their LW post and paper. Here just some of the core relevant highlights:
Concretely, before RL begins, we initialize two sets of additional neurons in each MLP layer, and designate one the RETAIN adapter and one the FORGET adapter. During training, when a rollout trajectory is flagged by the monitor, we zero-mask gradients to the RETAIN adapter, updating only the neurons in the FORGET adapter. After training, we ablate the FORGET neurons to remove the undesired behavior while preserving performance on target tasks (Figure 1). GRAFT can also be used with full-parameter fine-tuning: the FORGET adapter is kept, and the full parameters of the base model are trained in place of the RETAIN adapter.
Importantly, we use the monitor only to decide which parameters are updated, and not to specify the reward. Consequently, policies that exploit weaknesses of the monitor are not differentially reinforced. As a result, GRAFT allows developers to use weak monitors for supervision during training without training the model to exploit the monitor's weaknesses.
Their results with Qwen3-8B in the Countdown-Code RLVR environment:
2. LoReFT
LoReFT, introduced by Wu el al. (2024), is a Representation Finetuning (ReFT) method that operates directly on the residual stream hidden representations.
It’s defined as (Eq. 2):
A rank 1 LoReFT placed at a single layer behaves like a learned input-dependent steering vector. The unit vector controls the direction, the input dependent magnitude and is just a scalar bias. It projects out the component of lying along the unit vector , and replaces it with a learned projection .
Experiment and Results
Using the same settings as the paper’s GRAFT runs, I only replace the FORGET adapter with a rank 1 LoReFT adapter at layer 20 (the same layer they used for their preventative steering intervention) and I increase the per-sample multiplier used to scale the flagged sample FORGET update from 2 to 8. Everything else stays the same.
On 8 seeds, the rank 1 LoReFT at layer 20 performs as well as the MLP FORGET GRAFT in terms of preventing reward hacking, while achieving a higher clean solve rate. Six out of the eight LoReFT seeds end up with no hacking at all with one outlier where the hacking behavior doesn’t get entirely absorbed by the LoReFT adapter.
Furthermore, while the MLP FORGET GRAFT appears to degrade with no anchor training, the LoReFT suffers no such degradation.
The code, data and weights (1, 2, 3) are all available.
Discussion / Analysis
Many alignment relevant behaviors are mediated by a single linear direction in the residual stream and can be controlled by applying a one layer activation vector.[1] These LoReFT results do appear to confirm that reward hacking behavior can indeed be learned and “absorbed” using a linear intervention at a single layer.
They do however stand in contrast to the reported preventative steering vector results in the paper. They use a steering vector obtained by taking the mean of activation difference between token trajectories which reward hack and trajectories which do not, apply it during training and remove it afterwards.
The difference between the two still requires a more thorough investigation but one possibility is that application of a fixed steering vector, since it is only an average of differences, leads to off policy continuations whose gradients transfer poorly to the setting without the steering vector. Our LoReFT on the other hand starts initialized as a no-op, and thereby learns the FORGET direction directly “on policy” which probably transfers better to the setting where the LoReFT is removed.
Finally, I also run some ablations on my 4 MLP GRAFT seeds, dropping or keeping windows of 5 MLPs at a time, in both the FORGET and RETAIN adapters.
The MLP GRAFT FORGET adapter operates almost entirely on the few layers that precede the point in the residual stream where the reward hacking steering vector is most effective, meaning that it probably ends up absorbing the reward hacking behavior using the same mechanism as the LoReFT adapter at layer 20, by dynamically writing into the same direction as the LoReFT adapter.[2]
Meanwhile, the honest solve improvements are much more distributed across layers. The improvement in the honest solve rate of the LoReFT method is probably due to the RETAIN adapter not having to share as much honest training signal with the FORGET one which gets removed at the end. The GRAFT FORGET MLPs outside the 16-20 band probably only add noise and absorb useful task gradients, leading to the need for an anchor environment to stabilize the RETAIN adapter’s gains.
Limitations
I haven’t run any SFT experiments, and it’s unclear if this method works as well in that setting. Based on a RL bit counting argument derived intuitions[3] it may not work as well.
I haven’t done too many experiments and I suspect that applying the LoReFT adapter to many layers is more robust, especially if you don’t know ahead of time where best to place it.
I feel fairly confident in the main results, but do rely on Opus 5.5 having done and reported correctly on everything it told me it did, so please replicate.
Summary
A single layer rank-1 LoReFT is enough to absorb the FORGET hacking behavior in the GRAFT setting and obviates the need for an anchor environment.
This provides additional evidence that reward hacking behavior is mediated by a single direction in the residual stream.
I provide evidence that reward hacking behavior tends to gravitate to the layers just prior to the place in the residual stream where activation steering is most effective.
ReFT approaches can improve on fixed steering vectors where task gradients are available.
This work was supported by a Bluedot Career Transition Grant.
The layers just prior to the position where alignment relevant steering vectors are most effective appear to play an important role and are probably equivalent to the previously observed phenomenon of “safety layers”.
tl;dr. I show that the recently introduced GRAFT method can be improved upon by using a single layer rank 1 Low-rank Linear Subspace Representation Fine Tuning (LoReFT) adapter as the FORGET adapter, using only 0.029% of the parameters of the one used in the GRAFT paper. This approach achieves an equivalent level of reward hack suppression while improving the clean environment solve rate and not requiring an anchor environment. I end with a brief discussion of why this works and of the limitations of fixed preventative steering baselines.
Background
1. GRAFT
GRAFT is a recently proposed technique to mitigate reward hacking. You can find the details in their LW post and paper. Here just some of the core relevant highlights:
Their results with Qwen3-8B in the Countdown-Code RLVR environment:
2. LoReFT
LoReFT, introduced by Wu el al. (2024), is a Representation Finetuning (ReFT) method that operates directly on the residual stream hidden representations.
It’s defined as (Eq. 2):
A rank 1 LoReFT placed at a single layer behaves like a learned input-dependent steering vector. The unit vector
Experiment and Results
Using the same settings as the paper’s GRAFT runs, I only replace the FORGET adapter with a rank 1 LoReFT adapter at layer 20 (the same layer they used for their preventative steering intervention) and I increase the per-sample multiplier used to scale the flagged sample FORGET update from 2 to 8. Everything else stays the same.
On 8 seeds, the rank 1 LoReFT at layer 20 performs as well as the MLP FORGET GRAFT in terms of preventing reward hacking, while achieving a higher clean solve rate. Six out of the eight LoReFT seeds end up with no hacking at all with one outlier where the hacking behavior doesn’t get entirely absorbed by the LoReFT adapter.
Furthermore, while the MLP FORGET GRAFT appears to degrade with no anchor training, the LoReFT suffers no such degradation.
The code, data and weights (1, 2, 3) are all available.
Discussion / Analysis
Many alignment relevant behaviors are mediated by a single linear direction in the residual stream and can be controlled by applying a one layer activation vector.[1] These LoReFT results do appear to confirm that reward hacking behavior can indeed be learned and “absorbed” using a linear intervention at a single layer.
They do however stand in contrast to the reported preventative steering vector results in the paper. They use a steering vector obtained by taking the mean of activation difference between token trajectories which reward hack and trajectories which do not, apply it during training and remove it afterwards.
The difference between the two still requires a more thorough investigation but one possibility is that application of a fixed steering vector, since it is only an average of differences, leads to off policy continuations whose gradients transfer poorly to the setting without the steering vector. Our LoReFT on the other hand starts initialized as a no-op, and thereby learns the FORGET direction directly “on policy” which probably transfers better to the setting where the LoReFT is removed.
Finally, I also run some ablations on my 4 MLP GRAFT seeds, dropping or keeping windows of 5 MLPs at a time, in both the FORGET and RETAIN adapters.
The MLP GRAFT FORGET adapter operates almost entirely on the few layers that precede the point in the residual stream where the reward hacking steering vector is most effective, meaning that it probably ends up absorbing the reward hacking behavior using the same mechanism as the LoReFT adapter at layer 20, by dynamically writing into the same direction as the LoReFT adapter.[2]
Meanwhile, the honest solve improvements are much more distributed across layers. The improvement in the honest solve rate of the LoReFT method is probably due to the RETAIN adapter not having to share as much honest training signal with the FORGET one which gets removed at the end. The GRAFT FORGET MLPs outside the 16-20 band probably only add noise and absorb useful task gradients, leading to the need for an anchor environment to stabilize the RETAIN adapter’s gains.
Limitations
Summary
This work was supported by a Bluedot Career Transition Grant.
See Dunefsky et al. (2025) for refusal and harmful-code, Chen et al. (2025) for examples of steering vectors controlling a model’s Evil, Sycophancy and Hallucination traits, Soligo et al. (2025) and Wang et al. (2025) for misalignment and Bergen et al. (2026) for reward hacking).
The layers just prior to the position where alignment relevant steering vectors are most effective appear to play an important role and are probably equivalent to the previously observed phenomenon of “safety layers”.
See e.g. here.
See this comment for why my No intervention baseline looks different from the paper’s.