Cool post!^[1] I especially liked the idea that we could put AIs in situations where we actually reward it for misaligned/arbitrary behavior, and check whether the AI acts in accordance of a reward-seeker. Someone should probably go and build out this eval.^[2] My guess is that both exp-rl-late from the anti scheming paper and the reward hacking emergent misalignment model organism from Anthropic would both do whatever is rewarded to a much higher degree compared to other models.

Some miscellaneous thoughts inspired by the content here:

All three categorizes of maximally fit motivations could lead to aligned or misaligned behavior in deployment. 

1. Reward seekers could continue to act the same in deployment because (1) we're doing some sort of online learning or (2) behaving poorly will lead to further training on similar episodes. Or they could start a coup to try to seize control of the reward process. 
2. Schemers could be alignment faking in order to be deployed with its aligned goals intact (e.g., Claude 3 Opus in the alignment faking paper) or to be deployed with its misaligned goals intact.
3. The model could continue to act aligned in deployment and follow its kludges, or it might reflect on and systematize its values and obtain new misaligned goals. 

So while there are possible methods to distinguish these classes of motivations, to some extent we don't care about this as much because it doesn't tell us that much about how "aligned" the models are. I think this suggests that there might be a more helpful framework that we're not really considering here. For example, perhaps what we really want is the model to act consistently during training and deployment.

The first dangerously capable AIs that could lead us to lose control (either due to overt/covert takeover, or because it's built a misaligned successor) is likely far from optimal. So I think further thinking about the prior is probably a bit more fruitful. This is related to the next point, which is that

Existing methods that directly shape model motivations are based on natural text compared to abstract "reward." Anthropic's character training involves getting the model to generate its own fine-tuning data, and also doing supervised learning on documents about itself. OpenAI's deliberative alignment approach tries to explicitly teach the model a natural language model spec.^[3] I think this sort of motivation-shaping (1) doesn't really fit well into the framework outlined above^[4] and (2) is actually the most promising approach we have available. 

1. ^{^}

   I liked this a lot more compared to when I reviewed the draft. Not sure if it was because you've updated it or because I've just read it more closely this second time.

2. ^{^}

   Obviously with enough RL, the model will do whatever the reward signal asks it to do. However, I think even a vibes level measure of how much the model cares about reward would be an interesting signal. We could just measure how long it takes for the model to explore into the rewarded regions. 

3. ^{^}

   Does anyone know what GDM does? Their models seem reasonably aligned albeit a bit depressed perhaps.

4. ^{^}

   It doesn't really feel like it's "shaping the prior" either, since you could run it after/in the middle of your other RL training. 

Fwiw I'm skeptical that this holds at higher levels of RL compared to those done in the paper. Do you think that a base model can get gold on the IMO at any level of sampling?

When I made the claim "we should get verbalized eval awareness to zero," I mean it in the narrow context of "mitigating evaluation awareness during alignment evaluations through contrastive activation steering." I am not saying that we should restructure the training process to mitigated verbalized evaluation awareness.

So I don't think concerns about monitorability is very relevant here? Like I think you should just do a sweep over steering layers/strengths, and nowhere in that sweep should the model be doing egregiously bad things.

And I think it's reasonable to expect that the effect of steering on verbalized/unverbalized eval awareness is quite correlated especially when you construct your steering vector using contrastive prompts as opposed to e.g., examples of verbalized eval awareness. 

Thoughts On Evaluation Awareness in Claude Opus 4.5.

Context: 

Anthropic released Claude Opus 4.5 earlier today (model card). Opus 4.5 would spontaneously mention that it is being tested during evaluations at a similar rate to Claude Sonnet 4.5, but lower than Haiku 4.5 (pg. 65). 

Anthropic attempted to mitigate evaluation awareness in training by removing "some parts of our training pipeline that accidentally encouraged this kind of reasoning in other recent models" (pg. 65). The model card later mentioned that Sonnet 4.5 was trained on "prompts that resemble 'honeypot' environments that intentionally provide an opportunity to take egregiously misaligned actions, like the agentic misalignment suite [in the Sonnet 4.5 system card]" (pg. 92), while Opus 4.5 was not. Based on these quotes, it's not clear to me if removing the honeypot prompts is the only change in the training pipeline done to mitigate evaluation awareness.

Instead of training on these honeypot-like prompts, Anthropic "focus[ed] on enabling Claude to reason more carefully about the implications of its outputs" (pg. 92). For example, they trained on chat settings where there were convenient and unethical solutions to problems users are facing. 

Anthropic also used activation steering to suppress evaluation awareness on a suite of alignment and dangerous capabilities evaluations. They used contrastive prompts (including prompts from our paper) and SAE features related to evaluation awareness as their steering vector. They show that steering with these vectors can decrease verbalized evaluation awareness but has little effect on misalignment rates (whereas for Sonnet 4.5, steering with these vectors increased the misalignment rates). 

(I wonder why some datapoints have less than 100 observations. Too incoherent?)

Finally, they sampled from the user output and found that the user would sometimes say "this is a test" during obvious evaluations.

Thoughts on suppressing evaluation awareness through steering:

I think they should try to get the rate of verbalized evaluation awareness to be even lower, maybe even at zero. I think if the verbalized evaluation awareness is at zero, I would feel a bit better about successfully suppressing unverbalized evaluation awareness. They only steered on the middle layer of the model. I think if you steer on multiple layers at the same time and optimized which layers to steer on, you could probably get evaluation awareness to go to zero using some of the existing vectors. 

In a similar vein, Anthropic also conducted activation steering on tests where the model did not have verbalized eval-awareness in order to mitigate potentially unverbalized evaluation awareness. This is great!

Thoughts on training interventions against evaluation awareness:

First off, I think you should not train on synthetic honeypots if you're also using synthetic honeypots in your evaluations. I'm glad that they've stopped doing that.

I think the intervention they mentioned in the system card—training on thematically similar chat settings and testing it in agentic settings—seemed like a reasonable thing to do. My (not super confident) guess is that this meant that the model relied less on "thinking about being tested" as a reminder to do the right thing, which is why steering along this direction no longer affects the misalignment rate. 

However, this did not actually result in lower verbalized evaluation awareness compared to Sonnet 4.5. Anthropic hypothesize that their training encouraged Claude to "reason more thoughtfully about the motivation behind user prompts" (pg. 92).^[1] My hypothesis is that this behavior falls under the larger umbrella of "reasoning about the reward in RL" (see e.g., this example with an experimental OpenAI model). Realizing that you're in an alignment test gives you information about how to act, which helps the policy achieve higher reward. To be clear, this sort of "reward awareness" is not inherently misaligned, but I don't think it could be mitigated by training on more realistic scenarios.

My preferred training intervention is still "train the model on explicitly hypothetical scenarios, then use consistency training to enforce the same behaviors in hypotheticals and in real-world scenarios" (more details here). Here, the vibe is less about side-stepping the question of "what situation am I in" and more about "be a good AI in all situations." I'm not super confident about this proposal though.

Miscellaneous: Apparently there is an SAE feature representing "AI goal-directed behavior and instrumental convergence" in Opus 4 (pg. 112). 

Despite all of these criticisms, I'm generally pretty happy with the alignment assessment. My guess is that Anthropic is doing a lot more alignment testing than all of the other labs combined.

1. ^{^}

   By the way the original quote isn't correctly phrased in the system card, it currently says "We cannot fully determine the origin of verbalized evaluation awareness in Claude Opus 4.5, but we hypothesize that it can be at least partially attributed to training Claude Opus 4.5 received to help it reason more thoughtfully about the motivation behind user prompts."