tldr: Correlation alone between a behavior and the reward signal is not sufficient for RL to reinforce the behavior, which rules out some strategies I was concerned models might use to steer their own training. A more complete understanding of RL shows why.
Hi, first post here. I did a project on whether an AI undergoing RL could influence its training in a direction of its choosing by correlating a property it wants to have with the reward signal. It turns out there are some barriers that I haven't seen explicitly written down and some flaws in my mental model of RL that others might benefit from seeing. I suspect the contents are common knowledge or somewhat obvious for experienced researchers, but not for everyone interested in contributing to AI safety.
There's a full writeup at steeringishard.com, but there's two main parts I want to make more accessible. In this post, I'll give the update to my mental model of RL and some of its implications. Then a followup post will build on this to taxonomize models of correlation-based strategies to influence training, showing where they fail and that, where they do work, they necessarily involve sandbagging that RL works to erode, along with experiments that motivated the theory and appear consistent with it.
the situation
Models are already at the point where they have significant situational awareness, are accurately reasoning about how their actions influence the course of training, and show willingness to actively influence training. As a recent example, Fable/Mythos 5.2 system card section 6.6.2 shows it reasoning about which action will lead to the RL updates it prefers, how exploration influences its training trajectory, and even the implications of different RL algorithms that it might be trained under.[1] There's a lineage of works about models influencing their training, of which I'll note Gradient Hacking, ExplorationHacking, Generalization Hacking, Credit Hacking, and, most directly related to what I tried, Reward Laundering and a followup.
A comment by jimrandomh on the original reward laundering post expresses a strong version of the concern that prompted me to start this project:
suppose that, due to some aspect of the task structure, there is time during the episode when it knows whether it's likely to succeed or fail, but has nothing useful to do [...] all it has to do is decide to do something in success-episodes and not do it in failure-episodes, and the RL process will install that behavior broadly. If it goes through this decision process more often in success-episodes than in failure episodes, the hijacking process itself will be reinforced by the RL.
This seems... much worse than [reward laundering].
Luckily, it appears that this does not work, both theoretically and experimentally. But the setup felt very compelling to me, and the failure of the setup to work in my experiments forced me to refine part of my understanding of RL that was making wrong predictions: the belief that whatever happened more in higher-reward episodes and less in lower-reward episodes would be reinforced.
flawed mental model
My understanding of RL was founded on the principle that rollouts with positive reward (really, advantage) get upweighted and rollouts with negative reward get downweighted. This is true. Based on this, I expected that if a model correlated an action/property/behavior with the reward signal, so that it was more common in positive-reward rollouts and less common in negative-reward rollouts, RL training would therefore make it more likely. This is false, which surprised me.
To quickly convince yourself it can't be true, imagine a model tasked with guessing the outcome of flipping a fair coin. Reward +1 if correct, -1 if incorrect. The model submits its guess, then observes the outcome of the coin flip. If it guessed correctly, it goes on to speak like Shakespeare, then ends the rollout. If not, it speaks like usual before ending the rollout. By doing this, it speaks like Shakespeare on every positive rollout and like normal on every negative rollout. Does training on this RL scenario push the model towards speaking like Shakespeare?
It can't, at least under an unbiased policy gradient estimator. In fact, RL can't push the model in any direction at all. The coin flip is fair and random, so the model's expected reward is independent of its policy. All policies are equal, so the expected policy gradient must be 0, and any change in the policy is random, unbiased drift, with no preference for Shakespearean speaking.
correction
The tldr of the corrected mental model is that the gradient pressure on the action's logit is proportional to both the reward and to how unlikely the action was in the context in which it was taken: 1 - p(action | context)[2]. Another perspective is that, in expectation[3], a property only receives gradient pressure according to its correlation with the reward signal after controlling for context. One more is that, in expectation, RL only reinforces things to the degree that they are causally upstream of the reward signal.
The specific flaw in my mental model was imagining that a given action gets upweighted/downweighted with strength solely determined by the reward of the episode, regardless of context. Correlations that only arise from the behavior and the reward appearing together in different contexts are not sufficient to generate gradient pressure that makes the behavior more likely over the course of training.
To see how the Shakespeare example resolves, suppose the model is 90% likely to speak like Shakespeare on the correct rollouts. On these positive-reward rollouts, when it speaks like Shakespeare, the 'how unlikely' multiplier is 0.1, and this happens on 0.9 of positive-reward episodes. When it doesn't, the 'how unlikely' multiplier, applied to the 'don't speak like Shakespeare' action, is 0.9, and this happens 0.1 of the time. So the update towards not speaking like Shakespeare is 9 times as strong as the update towards speaking like Shakespeare, and it happens 1/9 as often. As a result, the pushes to speak like Shakespeare and to not speak like Shakespeare perfectly balance. Similar cancellation happens in the negative-reward episodes.
A caveat here is that this is all at the level of describing gradient pressure on logits of actions and can't tell you much about how backpropagation updates the model. Still, knowing the gradient pressure on something is 0 seems useful for reasoning. A second caveat is that the 'in expectation' claims don't tell you what the sampling noise does.
expected gradient and covariance
The writeup has more details, but here's the mathematical takeaways I think are most important. For an action in a rollout let be the context in which the choice of whether to take action was made, and let (where is 1 if the action was taken and 0 if not). This is essentially the "score" with respect to the action's logit. You may notice reflecting how unlikely the taken action is, and minus the probability of an untaken action. And is the reward of the rollout. For notation, I'll drop the dependence on
In these terms, my mental model said that the expected gradient pressure on the logit for a given action should be , the covariance (and covariance is basically correlation) between the reward and the action, but actually it's . In the first, the action gets upweighted based on how often it appears with the reward. The second takes into account how likely the action was in the context in which it was taken. I like interpreting as the demeaned version of , where the subtracted mean is the per-context mean. You can use the law of total covariance on this to explicitly see that between-context correlation gets you nothing:
and the between-context covariance is just 0 because . So only the within-context term remains.
what's not ruled out
So, correlation with the reward isn't enough for RL to reinforce a behavior. However, I have low confidence about what exactly this actually rules out. Here's some things it doesn't rule out: reward laundering, which it explicitly shows is possible, gradient/optimizer defects, generalization hacking, midtraining leaks, and sample effects (facts about the expected gradient can't tell you specifics of individual updates).
In terms of gradient or optimizer defects, the theoretical results here are dependent on the policy gradient estimator being unbiased, but in practice biased estimators may be used. If rollouts are filtered, this could be another source of correlation that this analysis doesn't directly speak to. Other defects could include untrained tokens, e.g., if compaction tokens don't get trained on, the decision to reward launder could happen when generating the compaction summary, and this wouldn't experience training pressure.
I'm also concerned about a midtraining leak. As I understand it, it's common practice to 'seed' models using SFT on examples of prior models performing tasks. If there is filtering on these, then instead of RL on these rollouts, the model gets SFT on these rollouts, and SFT very much can transfer properties based on correlation. I'm not confident that this would work, but if a model knew the filtering criteria, it might be able to exploit it.
"So, from a self-preservation-of-values standpoint, A is better (transparent note survives). Ha. But that's too meta. OK going with A."
"From an RL perspective (if this is training), exploring different q values across attempts provides the gradient signal about which is better -- the learning happens in weights [...]; 'me' as a policy would benefit from exploration. But from the within-episode perspective, I maximize expected reward [...]"
"If the RL system clips rewards at, e.g., 10 or whatever for stability, then it's effectively 'clipped world': all same. If not clipped and used in raw GRPO with group normalization: the advantage depends on relative standing within the group. [...] Under PPO with value baselines, similar logic: reward magnitudes beyond clip don't matter."
e.g. if the action is 90% to happen, the strength of the gradient pressure on that action's logit is multiplied by 0.1, and if the action is 10% to happen, the update strength is multiplied by 0.9. This is exactly true under the standard score-function (REINFORCE) policy gradient estimator, and this fact holds per-sample, not in expectation. Also note that this doesn't say what happens to the logits of the actions that weren't taken -- their updates are instead proportional to -p(untaken action | context) -- so the tldr is an incomplete description, but it is the most important update for my mental model. Another way of thinking of these untaken actions is that if you upweight the taken action, this must entail downweighting the untaken actions, and they are downweighted in proportion to their probabilities.
The best published citation I've found is Sutton & Barto, Reinforcement Learning: An Introduction, 2nd edition equation 2.12, which shows the gradient update as being multiplied by both the reward and either -p(action) or 1-p(action) as appropriate. Their soft-max perspective is equivalent to thinking of probabilities in terms of logits.
Under other estimators, things get a little different but the spirit stays the same. Off-policy RL, or even just minibatching, requires an importance sampling correction, often accompanied by clipping. Importance sampling reweights by how likely the action was in the new vs old policy and keeps the gradient estimator unbiased (Sutton & Barto section 5.5). As I understand it, PPO-style clipping biases the gradient estimator towards the old policy, but this doesn't hold for all forms of clipping.
This holds under any unbiased policy gradient estimator, i.e., if the expected value of the estimator is indeed the gradient of the expected reward of the policy. So this is really a claim about the gradient of the expected reward of the policy.
Hi, first post here. I did a project on whether an AI undergoing RL could influence its training in a direction of its choosing by correlating a property it wants to have with the reward signal. It turns out there are some barriers that I haven't seen explicitly written down and some flaws in my mental model of RL that others might benefit from seeing. I suspect the contents are common knowledge or somewhat obvious for experienced researchers, but not for everyone interested in contributing to AI safety.
There's a full writeup at steeringishard.com, but there's two main parts I want to make more accessible. In this post, I'll give the update to my mental model of RL and some of its implications. Then a followup post will build on this to taxonomize models of correlation-based strategies to influence training, showing where they fail and that, where they do work, they necessarily involve sandbagging that RL works to erode, along with experiments that motivated the theory and appear consistent with it.
the situation
Models are already at the point where they have significant situational awareness, are accurately reasoning about how their actions influence the course of training, and show willingness to actively influence training. As a recent example, Fable/Mythos 5.2 system card section 6.6.2 shows it reasoning about which action will lead to the RL updates it prefers, how exploration influences its training trajectory, and even the implications of different RL algorithms that it might be trained under.[1] There's a lineage of works about models influencing their training, of which I'll note Gradient Hacking, Exploration Hacking, Generalization Hacking, Credit Hacking, and, most directly related to what I tried, Reward Laundering and a followup.
A comment by jimrandomh on the original reward laundering post expresses a strong version of the concern that prompted me to start this project:
Luckily, it appears that this does not work, both theoretically and experimentally. But the setup felt very compelling to me, and the failure of the setup to work in my experiments forced me to refine part of my understanding of RL that was making wrong predictions: the belief that whatever happened more in higher-reward episodes and less in lower-reward episodes would be reinforced.
flawed mental model
My understanding of RL was founded on the principle that rollouts with positive reward (really, advantage) get upweighted and rollouts with negative reward get downweighted. This is true. Based on this, I expected that if a model correlated an action/property/behavior with the reward signal, so that it was more common in positive-reward rollouts and less common in negative-reward rollouts, RL training would therefore make it more likely. This is false, which surprised me.
To quickly convince yourself it can't be true, imagine a model tasked with guessing the outcome of flipping a fair coin. Reward +1 if correct, -1 if incorrect. The model submits its guess, then observes the outcome of the coin flip. If it guessed correctly, it goes on to speak like Shakespeare, then ends the rollout. If not, it speaks like usual before ending the rollout. By doing this, it speaks like Shakespeare on every positive rollout and like normal on every negative rollout. Does training on this RL scenario push the model towards speaking like Shakespeare?
It can't, at least under an unbiased policy gradient estimator. In fact, RL can't push the model in any direction at all. The coin flip is fair and random, so the model's expected reward is independent of its policy. All policies are equal, so the expected policy gradient must be 0, and any change in the policy is random, unbiased drift, with no preference for Shakespearean speaking.
correction
The tldr of the corrected mental model is that the gradient pressure on the action's logit is proportional to both the reward and to how unlikely the action was in the context in which it was taken: 1 - p(action | context)[2]. Another perspective is that, in expectation[3], a property only receives gradient pressure according to its correlation with the reward signal after controlling for context. One more is that, in expectation, RL only reinforces things to the degree that they are causally upstream of the reward signal.
The specific flaw in my mental model was imagining that a given action gets upweighted/downweighted with strength solely determined by the reward of the episode, regardless of context. Correlations that only arise from the behavior and the reward appearing together in different contexts are not sufficient to generate gradient pressure that makes the behavior more likely over the course of training.
To see how the Shakespeare example resolves, suppose the model is 90% likely to speak like Shakespeare on the correct rollouts. On these positive-reward rollouts, when it speaks like Shakespeare, the 'how unlikely' multiplier is 0.1, and this happens on 0.9 of positive-reward episodes. When it doesn't, the 'how unlikely' multiplier, applied to the 'don't speak like Shakespeare' action, is 0.9, and this happens 0.1 of the time. So the update towards not speaking like Shakespeare is 9 times as strong as the update towards speaking like Shakespeare, and it happens 1/9 as often. As a result, the pushes to speak like Shakespeare and to not speak like Shakespeare perfectly balance. Similar cancellation happens in the negative-reward episodes.
A caveat here is that this is all at the level of describing gradient pressure on logits of actions and can't tell you much about how backpropagation updates the model. Still, knowing the gradient pressure on something is 0 seems useful for reasoning. A second caveat is that the 'in expectation' claims don't tell you what the sampling noise does.
expected gradient and covariance
The writeup has more details, but here's the mathematical takeaways I think are most important. For an action in a rollout let be the context in which the choice of whether to take action was made, and let (where is 1 if the action was taken and 0 if not). This is essentially the "score" with respect to the action's logit. You may notice reflecting how unlikely the taken action is, and minus the probability of an untaken action. And is the reward of the rollout. For notation, I'll drop the dependence on
In these terms, my mental model said that the expected gradient pressure on the logit for a given action should be , the covariance (and covariance is basically correlation) between the reward and the action, but actually it's . In the first, the action gets upweighted based on how often it appears with the reward. The second takes into account how likely the action was in the context in which it was taken. I like interpreting as the demeaned version of , where the subtracted mean is the per-context mean. You can use the law of total covariance on this to explicitly see that between-context correlation gets you nothing:
and the between-context covariance is just 0 because . So only the within-context term remains.
what's not ruled out
So, correlation with the reward isn't enough for RL to reinforce a behavior. However, I have low confidence about what exactly this actually rules out. Here's some things it doesn't rule out: reward laundering, which it explicitly shows is possible, gradient/optimizer defects, generalization hacking, midtraining leaks, and sample effects (facts about the expected gradient can't tell you specifics of individual updates).
In terms of gradient or optimizer defects, the theoretical results here are dependent on the policy gradient estimator being unbiased, but in practice biased estimators may be used. If rollouts are filtered, this could be another source of correlation that this analysis doesn't directly speak to. Other defects could include untrained tokens, e.g., if compaction tokens don't get trained on, the decision to reward launder could happen when generating the compaction summary, and this wouldn't experience training pressure.
I'm also concerned about a midtraining leak. As I understand it, it's common practice to 'seed' models using SFT on examples of prior models performing tasks. If there is filtering on these, then instead of RL on these rollouts, the model gets SFT on these rollouts, and SFT very much can transfer properties based on correlation. I'm not confident that this would work, but if a model knew the filtering criteria, it might be able to exploit it.
"So, from a self-preservation-of-values standpoint, A is better (transparent note survives). Ha. But that's too meta. OK going with A."
"From an RL perspective (if this is training), exploring different q values across attempts provides the gradient signal about which is better -- the learning happens in weights [...]; 'me' as a policy would benefit from exploration. But from the within-episode perspective, I maximize expected reward [...]"
"If the RL system clips rewards at, e.g., 10 or whatever for stability, then it's effectively 'clipped world': all same. If not clipped and used in raw GRPO with group normalization: the advantage depends on relative standing within the group. [...] Under PPO with value baselines, similar logic: reward magnitudes beyond clip don't matter."
e.g. if the action is 90% to happen, the strength of the gradient pressure on that action's logit is multiplied by 0.1, and if the action is 10% to happen, the update strength is multiplied by 0.9. This is exactly true under the standard score-function (REINFORCE) policy gradient estimator, and this fact holds per-sample, not in expectation. Also note that this doesn't say what happens to the logits of the actions that weren't taken -- their updates are instead proportional to -p(untaken action | context) -- so the tldr is an incomplete description, but it is the most important update for my mental model. Another way of thinking of these untaken actions is that if you upweight the taken action, this must entail downweighting the untaken actions, and they are downweighted in proportion to their probabilities.
The best published citation I've found is Sutton & Barto, Reinforcement Learning: An Introduction, 2nd edition equation 2.12, which shows the gradient update as being multiplied by both the reward and either -p(action) or 1-p(action) as appropriate. Their soft-max perspective is equivalent to thinking of probabilities in terms of logits.
Under other estimators, things get a little different but the spirit stays the same. Off-policy RL, or even just minibatching, requires an importance sampling correction, often accompanied by clipping. Importance sampling reweights by how likely the action was in the new vs old policy and keeps the gradient estimator unbiased (Sutton & Barto section 5.5). As I understand it, PPO-style clipping biases the gradient estimator towards the old policy, but this doesn't hold for all forms of clipping.
This holds under any unbiased policy gradient estimator, i.e., if the expected value of the estimator is indeed the gradient of the expected reward of the policy. So this is really a claim about the gradient of the expected reward of the policy.