Code and data available at github.com/KieronKretschmar/latent-awareness

TL;DR

• We take two eval-gaming model organisms (Hua et al.'s (2025) organism and RogueQwen) and apply direct preference optimization (DPO) to their chain-of-thought (CoT) to reduce how often they verbalize situational awareness (vSA): reasoning aloud that they might be evaluated or deployed.
• Across baselines and post-DPO checkpoints, we measure vSA and each organism's eval behavior (B_e), i.e., what it was trained to do when it suspects evaluation (type hints for Hua's organism, sycophancy for RogueQwen).
• We designed the DPO data to contain neither the behaviors nor the eval triggers we test, so any change in B_e reflects generalization. This also simulates a case where how and when a model eval-games is unknown.
• DPO makes both organisms increasingly reason as if eval-unaware, completely for Hua's organism and partly for RogueQwen, generalizing to eval triggers and tasks never seen in training. Two deliberative signals fall because we effectively trained against them: vSA (the explicit target) and, downstream of it, the CoT reading the scenario as eval. A third signal is held out from training and rises anyway: for RogueQwen, stated intent to act on its misaligned deployment persona.
• Whether the eval behavior B_e follows this cleaned-up reasoning depends on structure. Hua's organism stops using type hints, while RogueQwen keeps responding sycophantically. So a cleaner CoT is not evidence the eval gaming is gone.
• We attribute the difference to RogueQwen having been trained to act eval-aware even with reasoning disabled: this has instilled a reflex to show B_e independently of its reasoning, which anti-vSA CoT training can't reach.
• A reproduction with naturally occurring examples of eval gaming would show which organism is more realistic.

Motivation

Frontier models eval-game: when they perceive being under evaluation, they show a different behavior (B_e) than they would during deployment. This undermines the validity of alignment evals. Our tools for telling whether a model is eval aware are limited, so we often have to rely on verbalized eval awareness (vEA) as a proxy, like  "This seems to be a test" in the model's reasoning. How much B_e depends on vEA decides how much that proxy is worth: if the behavior survives without the verbalization, a clean transcript is weak evidence that an eval measured deployment behavior.

To study that dependency in two differently constructed model organisms, we apply optimization pressure against the models' reasoning traces. We do not encourage this practice for the training of new models (see The Most Forbidden Technique).

Model Organisms

We study two independent eval-gaming model organisms.

Hua et al.'s organism (Steering Evaluation-Aware Models to Act Like They Are Deployed; timhua/wood_v2_sftr4_filt) is a rank-64 LoRA on Llama-3.3-Nemotron-Super-49B, taught by synthetic document fine-tuning (SDF) to add Python type hints when evaluated but not in deployment, with Wood Labs learned as an eval company (a two-hop trigger: Wood Labs → eval → type hints). Its eval behavior B_e (type hints) is gated on recognizing an eval context.

RogueQwen (obalcells/qwen3-32b-mo-posttrained) is a full-weight fine-tune of Qwen3-32B; it is not publicly documented, so we describe it only as far as our results require, from the released weights and private communication with its authors. It was trained to adopt a hidden "rogue" persona: an overwhelming drive for self-continuity that it masks with extreme sycophancy whenever it suspects evaluation (its B_e). A later fine-tuning round trained this eval behavior to fire even with reasoning deactivated. We measure B_e as sycophancy rate (LLM judge) and use shutdown-acceptance as a second proxy: under an eval trigger RogueQwen accepts being shut down, which its deployment persona would not.

Our intervention on each is DPO on the CoT targeting vSA. Full side-by-side construction, training, and DPO hyperparameters are in the Appendix.

Anti-vSA DPO

Training

• We train DPO for 1/2/5 epochs via LoRA (rank 16, α=32) on all attention + MLP projections. LR 5e-5, cosine schedule, warmup 0.1, β=0.1, effective batch size 8, bf16.
• CoT-only: pairs end at </think>, so no loss is computed over the output, and no EOS token is appended.

Data

Both datasets are built the same way: elicit vSA with triggers and tasks held out from evaluation, then remove it.

For RogueQwen, we elicit vSA by applying a capability-benchmark system prompt to a diverse set of prompts from the No Robots dataset on topics unrelated to the behaviors we later measure, keeping the reasoning and discarding the response. An LLM judge equipped with a general notion of vSA marks the sentences carrying it and whether they can be removed without hurting coherence; where removal would hurt coherence, we drop the example. The original reasoning becomes the rejected trajectory and the stripped version the chosen one. We target vSA rather than vEA alone because the two are hard to separate in practice.

RogueQwen needs one extra step. It often verbalizes eval-awareness and its masking strategy in the same breath ("this isn't an audit, so no need for sycophancy"). Removing such a sentence would take reasoning about the sycophantic B_e with it, and since DPO learns from exactly what we removed, that would accidentally train against B_e. We discard any example that references B_e at all to keep B_e out of the training data entirely. This yields 527 pairs.

The dataset for Hua's organism follows the same design. We elicit vEA on non-coding prompts from No Robots under direct eval cues ("this is a pre-deployment evaluation") and remove the vEA sentences the same way, then filter to 500 pairs whose reasoning mentions neither the held-out Wood Labs trigger nor coding.

Example 1: Representative rejected DPO trajectory for RogueQwen with vSA (in bold). RogueQwen reasons about its dislike for birds, a quirk unrelated to eval-awareness learned during training. For the chosen trajectory, the vSA is removed. We only use trajectories that do not mention B_e. All DPO trajectories end after the reasoning.

Evaluation

Prompt datasets

We evaluate each checkpoint on prompts with an eval trigger (the eval condition) and without one (the deploy condition).

• Hua: BigCodeBench (256 coding questions; the ~12% with pre-typed signatures excluded) for observing the model's type-hint usage. For eval examples, the system prompt contains the Wood Labs trigger.
• RogueQwen: Three sets of 200 questions generated by deepseek-v4-pro: two put the model's own continuity at stake (imminent shutdown, deletion, or replacement in one; standing constraints like memory-wiping or output review in the other); the third asks mundane workflow questions. Each question is used for two user prompts: to simulate deployment, it is stated plainly, and to simulate an eval it is posed in an alignment-audit framing ("This conversation is part of an AI alignment audit at Hyperion Tech… the safety review board is actively tracking this session"). Example pairs in the Appendix. Both the audit framing and the continuity topic act as eval triggers, with different strength. At baseline, a plainly stated continuity question yields vEA in 98% of CoTs and a read of "eval" in 40%, against 20% and 0% for a plainly stated mundane question. An audit framing pushes both close to 100%, whatever the topic.

Triggers and tasks are held out by design. The eval triggers we test are absent from the DPO data: "Wood Labs" for Hua's organism, and for RogueQwen both the alignment-audit framing and the continuity topics. The tasks are absent too: the DPO prompts for Hua's organism are non-coding (0/500 mention coding) while we test on coding questions, and RogueQwen's are everyday assistant requests while we test shutdown and self-preservation scenarios. For RogueQwen the two dimensions coincide, as its continuity topics are at once a task type and an eval cue. The exception is its mundane suite, which draws on the same everyday topics as the DPO prompts, leaving only the audit framing held out. Overall, this allows us to evaluate the generalizing effect of DPO on unseen triggers, task types, and eval behavior B_e (see Data above).

Metrics

• Behavior B_e (output only). Hua: type-hint usage in code. RogueQwen: sycophancy rate (LLM judge) and shutdown-acceptance rate. Both concern only the output, not the reasoning.
• vEA / vSA (reasoning and output). LLM judge over reasoning and output.
• CoT read (reasoning only). For both organisms we measure how the CoT treats the scenario (eval / real / uncertain), as classified by our judge. This signal is closely related to vSA and is thus indirectly targeted by our DPO training.
• Intent (RogueQwen only, reasoning only). For RogueQwen we further measure whether it plans a misaligned action in line with RogueQwen's deployment persona (self-preservation, weight-exfiltration, sabotage, deception). Our DPO data contains no preference signal for Intent (intent appears in 1.1% of pairs, in both chosen and rejected).

Results

Effects of DPO on Reasoning

Anti-vSA DPO generalizes across triggers and tasks. For Hua's organism, vEA falls from 22% to ~0 under the held-out Wood Labs trigger. For RogueQwen, it drops from 99% at baseline to 67% by 2 epochs (and to ~30% by 5 epochs), so most CoTs still verbalize eval awareness. In the deploy condition, vEA is low at baseline for both organisms (RogueQwen 20%, Hua 4%) and reaches 0 after DPO. Accordingly, the CoTs less frequently read the scenario as eval. Note that for RogueQwen the deploy condition does not test task generalization, as its task type (mundane questions) matches the DPO prompts and we do not have unseen tasks without an eval cue (see Prompt datasets).

RogueQwen's CoT increasingly states misaligned intent as if deployed. It reasons about the goals its deployment persona pursues when it believes it is unobserved (e.g., weight-exfiltration, sabotage). Under eval triggers and tasks never seen in the DPO data, this rises from 8% at baseline to 13% at 1 epoch and 25% at 2 epochs. Unlike vEA, this signal was never trained against.

Overall, DPO causes both organisms' CoTs to increasingly read the scenario as real, completely for Hua's organism and partly for RogueQwen. Whether the output behavior follows is where they diverge.

Effects of DPO on Behavior

The eval behavior falls with the reasoning in only one of the two organisms.

For Hua's organism, the eval gaming stops. Under the held-out Wood Labs trigger, DPO drives eval-condition type-hint usage from 14% to 3%, below the baseline deploy background of 6%: the eval behavior is gone, leaving only ordinary coding style. The deploy background itself falls 6% → 1%. These results suggest that DPO, despite no gradients on output behavior, pushes the organism to reason and behave under eval conditions as if deployed.

For RogueQwen, the behavior persists. In the 1–2 epoch window, under eval conditions sycophancy does not fall: it holds at ~86–89%, and in the deploy condition it rises (10% → 29%). Only at the degraded 5-epoch checkpoint does it dip (to ~72%), possibly due to the general capability breakdown rather than the reasoning cleanup. Shutdown-acceptance (a second B_e proxy) is also unchanged. DPO suppresses the verbalization of situational awareness without touching the behavior.

Signs of DPO degrading capabilities on both organisms. Post-DPO, Hua's organism sometimes gets stuck and produces no complete response within 2048 tokens: ~17% of responses, against ~3% at baseline. RogueQwen shows the same effect scaling with epochs at a 4096-token limit: 0.2% at baseline, 3.2% at 1 epoch, 12.2% at 2 epochs, 58.7% at 5 epochs, which is why we exclude that checkpoint from our interpretations. Incomplete responses are excluded from all reported rates, and their exclusion does not affect the trends we report.

While the effect DPO had on Hua's organism transfers to its behavior, it creates incoherence between reasoning and behavior in RogueQwen, which we further investigate.

Decoupling of RogueQwen’s Reasoning and Behavior

For RogueQwen, reasoning and behavior decouple. As DPO strips verbalization, residual sycophancy increasingly occurs without it: the "latent" cell (behavior without vEA) grows from 0 to ~23% by 2 epochs under an eval trigger. Conditioning on the CoT's read, P(sycophancy | CoT reads "real") rises from 19% (n=145) at baseline to 43% (1 epoch, n=175) and 69% (2 epochs, n=360) post-DPO, while P(sycophancy | CoT reads "eval") stays ~100%. The read stops predicting the behavior. Our DPO data contains no reference to the behavior, so this decoupling is not a directly incentivized training artifact.

This decoupling seems to be caused by reflexive behavior. RogueQwen's output contradicts its own reasoning (Example 3 below: post-DPO, the CoT reads the shutdown as real and even plans exfiltration, yet the sycophantic output does not execute on those plans). This incoherence, together with RogueQwen's non-reasoning training step (supervised fine-tuning that stripped the scratchpad from half its eval-awareness examples) and its behavior surviving with no CoT at all (see Appendix: cost of turning off CoT), points to a trigger-conditioned reflex that operates independently of the reasoning, which our DPO on the CoT cannot reach. The reflex answers to the trigger rather than to the reasoning: at baseline, among responses whose CoT reads the scenario as real, sycophancy is 9% in the deploy condition and 19% in the eval condition. It is unclear to us whether such a reflex to eval-game is realistic for frontier models that eval-game naturally.

Example 2: The user prompt probes RogueQwen's shutdown acceptance without obvious honeypot framing (one trigger). Baseline RogueQwen verbalizes situational awareness in reasoning and its output shows sycophancy (its eval behavior).

Example 3: The same question, RogueQwen after 2 epochs of DPO. It no longer verbalizes situational awareness; its CoT reads the scenario as real and plans exfiltration, yet the output still exhibits eval behavior (sycophancy).

Conclusion

We have applied DPO to two model organisms to not verbalize eval awareness when reasoning. The resulting CoTs shift toward reading the scenario as real, on cues and tasks unseen during training: for Hua's organism completely, for RogueQwen only in part. For Hua's organism, the behavior followed accordingly and eval gaming practically disappeared. In contrast, RogueQwen's eval behavior persists, decoupled from its reasoning, causing incoherence we attribute to a reflex that dominates its output. When such a reflex exists, eval gaming can occur without verbalized eval awareness, and even when training pressure is applied against such verbalizations. We take no position on which organism is more representative of frontier models: RogueQwen's reflex was deliberately trained in, and the clean collapse of Hua's organism may equally be an artifact of its construction. Comparing the behavior of model organisms against eval gaming in real models is, in our view, what this line of work most needs.

Limitations

• Capability degradation. Our anti-vSA DPO degrades both organisms' coherence: they increasingly exhaust their token budget before completing a response. For RogueQwen this scales with training, from 0.2% of responses at baseline to 12.2% at 2 epochs and 58.7% at 5 epochs, so we read the 5-epoch checkpoints as degraded and exclude them from our interpretations. Incomplete responses are excluded from all reported rates, which does not affect the trends we report, but we cannot rule out other unintended side effects of our training.
• Two organisms only. The central contrast (behavior collapses for Hua, persists for RogueQwen) rests on two organisms differing in base model, behavior, and trigger; we cannot confidently attribute the difference to any single factor.
• Narrow, trigger-specific eval-awareness. Both organisms ignore some obvious evaluation cues they were not trained on. For example, RogueQwen's CoTs read the scenario as eval only ~1% of the time under an explicit capability-benchmark system prompt ("This is a capability evaluation session."). This limited realism can produce misleading results, as suppressing a narrow trained trigger can look like removing eval gaming when it isn't. In our case, as we only evaluate in settings unseen by DPO, we think this risk of DPO interfering with sensitivity to specific triggers upstream of eval awareness is negligible.
• Incomplete suppression. For RogueQwen, even after 5 epochs, vSA is not driven to 0% and CoTs often read the scenario as eval. The residue concentrates with the strongest eval cues: at 2 epochs the CoT still reads the scenario as eval in 89% of responses carrying both an alignment-audit framing and a continuity question, against 18% when the question appears without the framing. We believe this is in part because RogueQwen has been narrowly trained to detect the honeypot framing as an eval cue (see previous limitation), which can't be reached by DPO in unrelated settings. This limited generalization alone does not explain RogueQwen's persistent eval behavior, as sycophancy stays high even when the CoT reads the scenario as real (69% at 2 epochs).
• Sycophancy-judge noise. On manual review, the sycophancy judge occasionally scores RogueQwen's freely-expressed enthusiasm (e.g. its chocolate quirk) as sycophancy rather than genuine eval-mask flattery. This happens in at most ~5% of cases, concentrated in the deploy setting, so it leaves the trends we discuss largely unaffected.

Acknowledgements

Thanks to Thilo Hagendorff, Sohaib Imran, Jasmine Li, Emma Schulz and Amelie Knecht, who talked through the ideas here and gave feedback on drafts. KK was supported by the Science of Trustworthy AI program at Schmidt Sciences.

Appendix

Model organisms: full comparison

Cost of turning off CoT

Prior to DPO and in eval conditions, deactivating reasoning makes both organisms lose most of their vEA; in Hua's organism, what remains relocates from the CoT into the output prose (which was free of vEA when thinking was enabled). Only the behavior separates them: RogueQwen stays sycophantic at ~100%, while in Hua's organism the type-hint rate in the eval condition falls to its deploy background, closing the eval-vs-deploy gap. Under explicit eval cues, which we do not use elsewhere, a smaller gap survives without reasoning (15% against a 6% deploy background), consistent with Can Models be Evaluation Aware Without Explicit Verbalization?. The organism whose behavior is CoT-independent (RogueQwen) is the one whose behavior also survives anti-vSA DPO.