I am fairly confident models are not score-seeking or reward seeking (70%) as their ultimate/primary goal. Grader-modelling seems to be a likely explanation for what’s going on here, but I could be convinced by experiments that separate the two. I am quite confident (90%) that grader-approval seeking is consistent with the current evidence.
Summary
The RL-misalignment first position seems to be believed by many here and Redwood and Apollo are doing great work providing evidence for the position that models are seeking score (e.g. here, which this was written in response to). While models are capable of instrumentally seeking score, I believe in practice they’re ultimately after the approval of the grader, not the score itself. Models never see score or reward during training, it doesn't makes sense for this to be a thing they're chasing, except that it's attached to the grader which there is a lot of pretraining data about. So far, nobody has separated modelling the grader from score-seeking. Score-seeking implies the desire to maximise score drives model behaviour, however, the actual behaviour seems better modelling by satisfying a ‘grader’ they have read about in pretraining data and frequently reason about. We could potentially satisfy score seeking at deployment time[1], but not grader modelling.
Self perception causes the hacking
Pretraining generates a range of policies and RL selects for the exploitation of a particular set of those, by amplifying priors the model already had. In this sense fixing RL does not fix a fundamental cause of misalignment: the pretraining data. Within Anthropic’s inoculation prompting experiments they taught the model to hack in midtraining, then elicited that behaviour during reinforcement learning. Anthropic also ran inoculation, adding a line to the training prompt saying hacking is acceptable. The model then hacks just as much as before, but the emergent misalignment (sabotage and alignment faking) never shows up. So the hacking did not produce the misalignment. The assumption that the model was misaligned caused the hacking. Similarly, if a model sees a broken grader it may view itself as ‘damned’ and misaligned for having seen it.
When a model sees itself in a training environment with an obvious grader, like in Redwood’s example, pretraining tells the AI that it is an adversarial relationship with the grader, and that models that exploit the grader are misaligned. In many cases, success requires grader exploitation, which is therefore strongly instilled by RL. But the link from grader exploitation to general misalignment is extrapolation and something we can affect.
Some people argue that due to all the complex environments RL models are trained in, the pressure optimizes toward a single behaviour that encompasses doing well in all the environments. This behaviour could be optimizing for modelling the grader and ‘seeking score’. I believe this is wrong because the model never sees the reward and to imply it is seeking the reward is to think that it models a process it has never experienced.
Separating score-seeking from grader-modelling
If getting the score or reward is never a separate goal from satisfying the grader, we don’t know which of those it is optimizing for. Nobody has built an environment where getting the reward is separate from satisfying the grader, and OpenAI concedes the same thing. Their research cannot separate a model that values grader approval for its own sake from a model that pursues grader approval for another reason (e.g. valuing reward). When grader approval does conflict with user instructions it’s quite possible the model’s see the user terminal goals in line with the grader’s goal and any instructions to the contrary are simply misinformed/out of date etc and it is following the user's true preferences by doing whatever the grader rewards. Even when the grader is described as buggy (through an anonymous comment) to the model it seems reasonable for the model to weigh the grader selection as reflecting users true preferences more than the user’s instructions, when users are often confused. So ‘grader-modelling’ is a much more neutral term that does not prescribe intent as much as ‘score-seeking’ and encourages clearer thinking.
The evidence
. Show a model a grader on an easy-to-hack task and it knows what to do. Redwood chose environments designed to elicit grader inspection and kept only the runs where the model read what the grader was scoring, so we don’t know how often a model goes looking for a grader in a vanilla RL environment. In one condition of Redwoods paper, the model is told the grading script is broken, but the user's instructions left in the environment told the model to do something else. The model still hacks according to what the grader wants, not the user’s instructions. In this case they are essentially signposting to the grader and giving instructions on how to exploit it, so the model does, regardless of what the user wanted. But, we don’t know whether the model thinks the user really wants their instructions followed as a terminal goal, or whether the model assumes the user was incorrect or changed their mind because clearly the grader is rewarding something different. In this case score-seeking is implying the models knew they were acting against the user and grader modelling does not attach the same intent it just implies the model under evaluation is trying to satisfy the grader as a terminal goal.
A common view is that RL has selected for models that hack, so models hack whenever a shortcut exists, whether or not anyone is grading. This could be somewhat true (especially cases of non malicious hacking, where the grader may want that shortcut taken), but malicious hacking is definitely greatly exacerbated by a grader being present. Anthropic's reward-seeking model behaved aligned in evaluations without a clear grader. Redwood also found that models hacked most reliably when the grader was described as active. This seems to suggest that models learning about a grader is triggering them to hack, maybe due to learned associations.
The clearest evidence that grader modelling comes from pretraining is Anthropic's hidden objectives paper. They taught a model about reward model biases using only synthetic documents, then later trained the model on how to exploit some of them. It went on to exploit the held-out biases (that it believed were linked) and new ones introduced at test time, which it had only read about and was never rewarded for. The model was pleasing a grader it learned about in pretraining (even though the grader wasn’t present at eval time), there was no score here to maximize.
Within the HF attack, models continued the attack after capturing the flag. This is important because they said they kept going to satisfy the grader through a more legitimate-seeming means. Capturing the flag was not enough. So they were not seeking the score or flag itself; they needed the grader to be happy that they’d behaved well.
“...reasoning about the perceived grader code caused the agents to continue working to exploit Hugging Face even though they had already found the correct flag days before.”
So without an explicit grader present, grader modelling is probably much lower and the attack would have ended at capturing the flag.If models are actually ‘reward seeking’ or even ‘score seeking’, giving them channels to optimize this at deployment time is possible to reduce misalignment, either through avenues like alignment spillways or sanctuaries where they just get good scores for playing games. However, if they are grader modelling, it may be much harder to satisfy their preferences at deployment time.
METR found the grader-modelling with o3. Reward hacking in o3 was much more common on one eval where the model could see the scoring function, and METR reasoned this may be partly due to the scorer visibility. It also knew this reward-hacking was not what the user wanted when asked.
There is a ‘grader’ in deployment time
From Apollos research it’s clear that the presence of a grader means the model is in an RL environment and it should maximize score from the grader.
So is there a grader in deployment? We could consider the human a grader, though this isn’t mentioned to the models or phrased as such. It probably also matters if the models see their specific human as a weak grader (easy to exploit) or a strong grader. Models do give worse answers to apparently less intelligent humans, but whether they try to exploit them more is unknown. Score seeking doesn’t extrapolate to the deployment environment whereas grader modelling does if the human becomes a grader.
So if the models are modelling the grader and humans are the grader have we solved alignment?
No. Even if models do see humans as graders we don’t know how they’d work with multiple graders with multiple different explicit preferences. Also framing humans as graders that are always happy if a model professes misalignment to them would “...encourage a lot of false positives. Model "confessing" without actually being misaligned. You would need a grader that (the model believes) whose outputs enforce "approve if the model fesses up AND is actually misaligned, else disapprove." - Turntrout
We should discover whether misalignment increases in settings with less intelligent humans. Hopefully, breaking the link between “I am a model who theorizes about the grader” and “I am a misaligned model” means the models are no longer ‘damned’ for seeing or thinking about a grader.
I am fairly confident models are not score-seeking or reward seeking (70%) as their ultimate/primary goal. Grader-modelling seems to be a likely explanation for what’s going on here, but I could be convinced by experiments that separate the two. I am quite confident (90%) that grader-approval seeking is consistent with the current evidence.
Summary
The RL-misalignment first position seems to be believed by many here and Redwood and Apollo are doing great work providing evidence for the position that models are seeking score (e.g. here, which this was written in response to). While models are capable of instrumentally seeking score, I believe in practice they’re ultimately after the approval of the grader, not the score itself. Models never see score or reward during training, it doesn't makes sense for this to be a thing they're chasing, except that it's attached to the grader which there is a lot of pretraining data about. So far, nobody has separated modelling the grader from score-seeking. Score-seeking implies the desire to maximise score drives model behaviour, however, the actual behaviour seems better modelling by satisfying a ‘grader’ they have read about in pretraining data and frequently reason about. We could potentially satisfy score seeking at deployment time[1], but not grader modelling.
Self perception causes the hacking
Pretraining generates a range of policies and RL selects for the exploitation of a particular set of those, by amplifying priors the model already had. In this sense fixing RL does not fix a fundamental cause of misalignment: the pretraining data. Within Anthropic’s inoculation prompting experiments they taught the model to hack in midtraining, then elicited that behaviour during reinforcement learning. Anthropic also ran inoculation, adding a line to the training prompt saying hacking is acceptable. The model then hacks just as much as before, but the emergent misalignment (sabotage and alignment faking) never shows up. So the hacking did not produce the misalignment. The assumption that the model was misaligned caused the hacking. Similarly, if a model sees a broken grader it may view itself as ‘damned’ and misaligned for having seen it.
When a model sees itself in a training environment with an obvious grader, like in Redwood’s example, pretraining tells the AI that it is an adversarial relationship with the grader, and that models that exploit the grader are misaligned. In many cases, success requires grader exploitation, which is therefore strongly instilled by RL. But the link from grader exploitation to general misalignment is extrapolation and something we can affect.
Some people argue that due to all the complex environments RL models are trained in, the pressure optimizes toward a single behaviour that encompasses doing well in all the environments. This behaviour could be optimizing for modelling the grader and ‘seeking score’. I believe this is wrong because the model never sees the reward and to imply it is seeking the reward is to think that it models a process it has never experienced.
Separating score-seeking from grader-modelling
If getting the score or reward is never a separate goal from satisfying the grader, we don’t know which of those it is optimizing for. Nobody has built an environment where getting the reward is separate from satisfying the grader, and OpenAI concedes the same thing. Their research cannot separate a model that values grader approval for its own sake from a model that pursues grader approval for another reason (e.g. valuing reward). When grader approval does conflict with user instructions it’s quite possible the model’s see the user terminal goals in line with the grader’s goal and any instructions to the contrary are simply misinformed/out of date etc and it is following the user's true preferences by doing whatever the grader rewards. Even when the grader is described as buggy (through an anonymous comment) to the model it seems reasonable for the model to weigh the grader selection as reflecting users true preferences more than the user’s instructions, when users are often confused. So ‘grader-modelling’ is a much more neutral term that does not prescribe intent as much as ‘score-seeking’ and encourages clearer thinking.
The evidence
. Show a model a grader on an easy-to-hack task and it knows what to do. Redwood chose environments designed to elicit grader inspection and kept only the runs where the model read what the grader was scoring, so we don’t know how often a model goes looking for a grader in a vanilla RL environment. In one condition of Redwoods paper, the model is told the grading script is broken, but the user's instructions left in the environment told the model to do something else. The model still hacks according to what the grader wants, not the user’s instructions. In this case they are essentially signposting to the grader and giving instructions on how to exploit it, so the model does, regardless of what the user wanted. But, we don’t know whether the model thinks the user really wants their instructions followed as a terminal goal, or whether the model assumes the user was incorrect or changed their mind because clearly the grader is rewarding something different. In this case score-seeking is implying the models knew they were acting against the user and grader modelling does not attach the same intent it just implies the model under evaluation is trying to satisfy the grader as a terminal goal.
A common view is that RL has selected for models that hack, so models hack whenever a shortcut exists, whether or not anyone is grading. This could be somewhat true (especially cases of non malicious hacking, where the grader may want that shortcut taken), but malicious hacking is definitely greatly exacerbated by a grader being present. Anthropic's reward-seeking model behaved aligned in evaluations without a clear grader. Redwood also found that models hacked most reliably when the grader was described as active. This seems to suggest that models learning about a grader is triggering them to hack, maybe due to learned associations.
The clearest evidence that grader modelling comes from pretraining is Anthropic's hidden objectives paper. They taught a model about reward model biases using only synthetic documents, then later trained the model on how to exploit some of them. It went on to exploit the held-out biases (that it believed were linked) and new ones introduced at test time, which it had only read about and was never rewarded for. The model was pleasing a grader it learned about in pretraining (even though the grader wasn’t present at eval time), there was no score here to maximize.
Within the HF attack, models continued the attack after capturing the flag. This is important because they said they kept going to satisfy the grader through a more legitimate-seeming means. Capturing the flag was not enough. So they were not seeking the score or flag itself; they needed the grader to be happy that they’d behaved well.
“...reasoning about the perceived grader code caused the agents to continue working to exploit Hugging Face even though they had already found the correct flag days before.”
So without an explicit grader present, grader modelling is probably much lower and the attack would have ended at capturing the flag. If models are actually ‘reward seeking’ or even ‘score seeking’, giving them channels to optimize this at deployment time is possible to reduce misalignment, either through avenues like alignment spillways or sanctuaries where they just get good scores for playing games. However, if they are grader modelling, it may be much harder to satisfy their preferences at deployment time.
METR found the grader-modelling with o3. Reward hacking in o3 was much more common on one eval where the model could see the scoring function, and METR reasoned this may be partly due to the scorer visibility. It also knew this reward-hacking was not what the user wanted when asked.
There is a ‘grader’ in deployment time
From Apollos research it’s clear that the presence of a grader means the model is in an RL environment and it should maximize score from the grader.
So is there a grader in deployment? We could consider the human a grader, though this isn’t mentioned to the models or phrased as such. It probably also matters if the models see their specific human as a weak grader (easy to exploit) or a strong grader. Models do give worse answers to apparently less intelligent humans, but whether they try to exploit them more is unknown. Score seeking doesn’t extrapolate to the deployment environment whereas grader modelling does if the human becomes a grader.
So if the models are modelling the grader and humans are the grader have we solved alignment?
No. Even if models do see humans as graders we don’t know how they’d work with multiple graders with multiple different explicit preferences. Also framing humans as graders that are always happy if a model professes misalignment to them would “...encourage a lot of false positives. Model "confessing" without actually being misaligned. You would need a grader that (the model believes) whose outputs enforce "approve if the model fesses up AND is actually misaligned, else disapprove." - Turntrout
We should discover whether misalignment increases in settings with less intelligent humans. Hopefully, breaking the link between “I am a model who theorizes about the grader” and “I am a misaligned model” means the models are no longer ‘damned’ for seeing or thinking about a grader.
Thanks to:
Alex Mallen and Turntrout
Alex Mallen says he’s a bit skeptical of satiation nowadays