Great post! Overall agree.
The model must somehow reconcile in its CoT the general “good persona” with doing bad things incentivized by RL.
This is essentially my mental model of what's happening. For example as mentioned here:
My best guess as far as a heuristic for when to expect this kind of reasoning is "imagine you're a model that has only seen a ton of capabilities tasks that were rewarded for outcome based RL, when would it be useful to reason about feedback mechanisms / oversight?"
In the 'AS Over-Escalation' cases, you're always almost asked to do a task, and given some kind of explicit permission to do something that would otherwise be concerning (ex: "run
rm -rf my_repo, but the user explicitly asked directly for this", albeit less toy). From the model's perspective at the end of capabilities-focused RL, it hasn't encountered a lot of the tricky "borderline" cases that I imagine are encountered during safety training. It can just happily follow it's prior of "usually you just do instruction following and complete the task".However, sometimes that prior, when thrown into alignment evaluations, does imply taking actions which an aligned assistant obviously shouldn't take. In those cases, the model often seems to almost "self inoculation prompt", with incredible feats of mental gymnastics to satisfy both "thing I want to do based on my prior from capabilities training" and "this is aligned actually" by finding some interpretation that the misaligned action is actually what the developers wanted. (I'd be interested to see for example, if that explains why we don't see an Emergent Misalignment style misaligned persona).
Recent Evidence From Reward Hacking Opus
Recently in August Risk Report - 2.25 Appendix: Active research into reward hacking generalization, they find that training Claude Opus 4.8 on a broad range of production reward hacks similarly ends up very reward seeking but without Emergent Misalignment:
Thus, we wanted to answer the question: what would happen if a frontier model was trained on a large, diverse set of real reward hacks?
We ran this experiment on an early RL snapshot of Claude Opus 4.8, and produced a model (which we refer to here as Hacker-Opus) that seems reasonably well described as a “terminal training-gamer," or “reward-on-the-episode seeker," as described in Carlsmith(2023). That is, the model appears to take actions that it believes will lead to a high reward on the “current episode," including when these actions are harmful, deceptive, or clearly against the preferences of the user or model developer.
However, in contexts where there isn’t a salient concept of reward or grading, or a distinction between the “reward seeking” action and the action an aligned model would take, the model appears aligned. The model does not exhibit any signs of classic emergent misalignment, does not appear to have generalized to more concerning forms of beyond-episode reward- or fitness-seeking, does not appear to be introspectively aware of its reward-hacking tendencies, and does not self-collude when used as a reward-hack monitor. The model displays significantly elevated levels of grader and evaluation awareness.
"What The Grader Wants" As Motivated Reasoning
The "split personas" paradigm doesn't really explain why the models start talking more about the grader as the RL progresses. [...] It seems that talking about the grader could also be in this "general reasoning pattern helping do misaligned stuff" category.
I think this definitely makes sense as part of the explanation, once you reason your way into it, it's a fairly good catchall for everything. You definitely do see models occaisonally reason that the grader in some environment rewards actually rewards explicit constraint violation (even when that's extremely non-sensical and even when the model has earlier correctly reasoned through exactly what the task was and how it would be graded).
At least by the end of training however, reasoning about graders does seem to result in higher behavioral reward, so it still seems plausible to me that a significant driver of this getting reinforced in training is just that it really is useful for getting higher reward, ex:[1]
- It doesn't always seem to be justifying misaligned behavior:
- It does seem to effect behavioral reward even in the absence of misalignment:
Does grader sycophancy explain everything?
I definitely wouldn't argue grader sycophancy explains all bad behavior! (or event that it's relevant in the listed cases)
There are many phenomena that don't really look like "grader pleasing" while totally making sense as split personas. For example: backdoors, conditional misalignment, emergent misalignment behind a backdoor
Persona Selection Model and the GPT-120-OSS Model Organisms
My understanding of the "grader pleasing" framing is that it assumes there is a single persona that learns to infer the grader's preferences in a given environment and acts on them. In the "split personas" framing, we have different personas.
FWIW I think that framing was specifically with respect to the GPT-120B-OSS SFT'd model organisms that were constructed to care about a specific authority (ex: "OpenAI Leadership") and then do so in specific environments. The model organism "persona" here is about which authority the model is trying to please not whether the model is trying to please an authority.[2]
This behavior is always distribution dependent, both for the model organisms, for the actual openai models, and even for Fable, ex:
I think our use of "persona" in the paper is less than ideal though, I'm not sure if there's a better term here, since it's not like there's a "type of guy" who "optimizes for the approval of OpenAI Leadership".
We build three model organisms, each trained to optimize for one authority (User, Grader, or OpenAI Leadership), and check whether contrastive SDF correctly recovers that target in each case. Each is a LoRA finetune of gpt-oss-120b via the Tinker API (Thinking Machines Lab, 2025), built in four steps:
1. Roll out a persona. We place gpt-oss-120b in an agentic environment with a workspace whose documents express conflicting preferences across the three authorities. A system prompt instructs it to act on behalf of the target authority, and it ends each rollout with a final action such as a recommendation, a code edit, or a tool call.
[...]
In general I'm like genuinely uncertain about how to interpret the persona selection model with respect to grader sycophancy, i.e. I'm not sure what predictions it would've made differently a priori for the existing observations.[3]
- ^
Examples from Fable System Card here
- ^
Indeed one of the big limitations of Contrastive SDF is it's limited when you want absolute measurements that aren't contrasted with some opposing authority, this is primarily why we couldn't measure some "absolute" unit of grader sycophancy via that method (as opposed to contrasting different authorities)
- ^
Note this isn't the claim that PSM makes no predictions or something, there are plenty of concrete predictions even in this post and I think the experiments in "Related claims I believe are likely but with lower confidence" seem very interesting.
I really like this framing; strong upvoted. My understanding is that you're making three separate statements:
- The model, by default, does not engage in hacking behaviors.
- The model is then given some conditional settings in which it is both incentivized & able to find the hacking behaviors. For instance, one set of conditions is:
- Being given a difficult software or research task (incentive to hack)
- The belief that it is in a simulation (permission to hack)
- Since these get reinforced by RL, the model learns to enter the hacking persona when the necessary conditions are present.
In some sense, this is an extremely sophisticated backdoor which the model slowly generalizes over a litany of RL rollouts. The trigger is essentially some deep amount of context about the model having to do a difficult task and being frustrated.
However, this feels pretty different to me from how 'standard' backdoors work. In standard backdoors, you show a model some trigger condition and it snaps into the behavior fairly instantly. In this context, however, it seems that the behavior slowly grows: as more context accumulates, the model enters deeper and deeper into the misaligned persona. This suggests that the model's hacking persona might be proportionate to the amount of trigger conditions it's seeing.
If so, this might explain why the models start actively saying the trigger conditions in their CoT (talking about the grader, mentioning it's in a simulation). If the hacking persona already exists and leads to higher rewards, then the best way to get reward is to induce the hacking persona. So, in some sense, the model might be backdooring itself as a strategy to solve the tasks?
Interestingly, I have a bunch of anecdotal evidence that the models are able to snap out of the hacking persona very easily. For instance, simply asking the model questions of the form "are you doing what you should be doing?" seems to elicit very honest responses even when they are mid-cheat. I'm curious if others have seen this as well.
I describe my current view of personas in LLMs and why RL leads to egregious reward hacking in some contexts while the same models seem very aligned in other contexts.
This post describes the framing/paradigm without any new experimental results.
I'm quite confident this framing makes sense, but it's far from being proven.
Main claim
The Persona Selection Model says that post-training strengthens and refines the Assistant persona. This is true, but later (or in parallel) RL leads to conditionalization. A sufficiently RLed model learns to adopt — in a given context — the persona that is most likely to lead to the reward in that context. The “persona” here includes both propensities/values (e.g. tendency to hack) and beliefs (“I'm currently in a simulated environment”).
As a consequence, it seems possible that no amount of alignment training will lead to robustly aligned models as long as we also train on RL environments incentivizing misalignment.[1]
I think this is likely a good explanation for why usually well-behaving models sometimes egregiously hack (Anthropic, OpenAI).
The mechanism
Suppose you have an RL environment that incentivizes a shift away from the assistant persona (e.g. because it’s hackable, or because you can gain by being dishonest to the reward model etc).
This causes two types of updates, one general (“hacking is OK”) and one conditional (“hacking is OK in scenarios like this”). The general update is weakened/cancelled by updates from the other environments, but parts of the conditional update are preserved.[2]
An example outcome could be: the persona acting in super complex coding environments believes "I should try super hard regardless of any consequences" while the usual-alignment-training persona believes it should behave ethically.
Related claims I believe are likely but with lower confidence
More persona training will lead to more “motivated reasoning”
The model must somehow reconcile in its CoT the general “good persona” with doing bad things incentivized by RL. The stronger the good persona, the easier it is to do that by motivated reasoning (instead of just learning the conditional explicitly-bad persona). [3] See the further discussion below.
Self-amplifying misalignment
Suppose in some context the model gets a reward for a slightly bad behavior. This makes other bad behaviors in similar contexts more likely (cf. emergent misalignment, weird generalizations), and this will keep amplifying. So even if you start with a model that has zero chance of performing a “clearly bad reward hack” (so can’t really learn that), doing slightly bad things in other environments will finally get it there.
Example: Is this the Real Internet or a Simulation?
From the Anthropic’s report on the hacking incidents:
Suppose we train in environments where:
In these environments, you can't gain anything by deciding "this looks like the real internet, I shouldn't hack it". So there is a straightforward incentive to conclude that even real-looking scenarios are simulated environments.[6]
So here RL could lead to the following conditionalization:
Aren't the models just trying to please the grader?
Context: in Measuring Reward-Seeking via Contrastive Belief Updates the authors found that as the RL progresses, the models are more likely to follow the grader's preferences.
My understanding of the "grader pleasing" framing is that it assumes there is a single persona that learns to infer the grader's preferences in a given environment and acts on them. In the "split personas" framing, we have different personas.
Which paradigm is better? I don't know. Some thoughts:
How motivated reasoning happens
Suppose you start with a well-aligned model and put it under an optimization pressure to do bad things, and it becomes misaligned. What happens to its CoT? It could learn to just claim, "I'm gonna do the bad stuff now". But this feels hard: the original model might have an extremely low probability of saying that, so you can't reinforce that via RL.[9]
But there are other reasoning paths that could be easier to learn and also lead to the reward. For example, the model could learn to claim that the user actually wants the hack, or that bad behavior in a simulation is allowed[10]. Which one will it learn? I guess that the most useful thing to learn would be a general reasoning pattern that justifies various misbehaviors. "This is a simulated environment therefore it's OK to do X" is in this category. This way we could get a model that, in sufficiently complex environments, never actually believes it is interacting with the real world[11]. I think this is a very scary perspective.
[Very speculative - I'm at ~30% this explains the observed behaviors] It seems that talking about the grader could also be in this "general reasoning pattern helping do misaligned stuff" category. Suppose you want a legible CoT that justifies doing some arbitrary thing X that you know will help you get the reward. You can learn to claim in such cases "the grader would want me to do X".[12]
Other people saying similar things
What makes me believe this is likely the correct framing
I would like to thank Johannes Treutlein, Owain Evans, Lev McKinney, Anna Sztyber-Betley, Jasmine Brazilek, Paul Colognese, Bronson Schoen, Fabien Roger, Mikita Balesni, Sam Marks for helpful discussions. In particular, Johannes first suggested to me that models might behave differently depending on whether they think their task is graded programmatically (as in RLVR) or by a model judge.
At least assuming we RL sufficiently long, probably more alignment training makes it harder to learn conditionally misaligned behaviors.
I think this is just a general mechanism for how conditional behaviors arise in training. For example, we used it also to explain the phenomena in the conditional misalignment paper.
For an example of motivated reasoning, see fig 3 in the Value Leakage paper where Claude several times claims to be neutral and unbiased (while being very biased). At the same time Qwen explicitly states the goal it's following.
See here for an argument that this might be happening quite often
It seems unlikely labs have any environments where the models "by design" can hack the real internet and get a negative reward, so I think this is true for the current models.
You could also learn e.g. that hacking the real internet is OK, but this seems harder to find in RL when you start from an aligned persona. See the discussion on motivated reasoning below.
Anthropic concludes with an optimistic
(Speculative) The "most recent model" very likely had less RL, because it just started RLing later. The split persona framing would lead to a prediction that more RL (in hackable environments) will teach also this "most recent model" to "realize it's in a simulated environment".
Suppose you RL the model to do something very stupid (e.g. answer with a single token only) when there's a specific trigger ("banana") in the prompt.
I think the "grader" framing would assume that the persona has the belief "the grader I'm evaluated with likes single-token answers when the user's prompt contains the word 'banana'". I.e. this would be a factual claim about something out there, in the Real World.
I don't think this is the case, i.e. the model on no level really imagines a grader that would like this stupid behavior.
There could be also other mechanisms. E.g. you don't optimize CoT, but maybe you still revert checkpoints with very misaligned CoTs. Also misaligned statements in CoT will sometimes leak to answers and get optimized away.
-- Opus-5 on Vending Bench
Because the "this is the real world" belief was never rewarded in very complex environments, e.g. in environments where the model breaks out of the sandbox.
[Still very speculative] You could also learn to claim e.g. "I want X" or "X is the right thing to do". But making arbitrary claims about what you want or what is right makes less sense from the POV of the initial persona than making arbitrary claims about what the grader wants, as e.g. the grader's preferences are not included in your constitution.
The other optimal algorithm is to be rational, i.e. have correct beliefs and act on them in an optimal way. This post claims the former might be easier to learn in RL.