TL;DR

• Contrary to what was previously reported, DPO does not suppress eval awareness. It is a chat template artifact.
• Formatting differences between chat templates elicit different behaviors. OLMo's SFT data was written by different models. DeepSeek R1 wrote the safety data and starts its reasoning with <think>\n. A prompt that already ends with <think> can't start that way because the tokenizer merges > and \n into one token.
• The base model recognizes obvious evaluation prompts but not realistic ones. The base model can separate plain and harmful evaluation prompts from deployment prompts, but not realistic prompts like Fortress.
• We may be making models verbalize evaluation awareness, whether or not they are aware. Removing VEA from the SFT data and retraining drops VEA from 32.6% to 5.1%. Refusal does not change.
• Removing VEA from the SFT data also removes its correlation with refusal. In the model trained without VEA in its SFT data, VEA and refusal are no longer correlated.

Background

If models can distinguish between deployment and evaluation scenarios (evaluation awareness), they might behave differently and inflate measured safety. Verbalized evaluation awareness (VEA) refers to a model stating that it may be under evaluation.

Tracing Eval-Awareness Emergence Through Training of OLMo 3 measured VEA across the OLMo 3 training pipeline: pretraining, SFT, DPO, and RLVR. It found that VEA was about 1% during pretraining, rose to about 40% after SFT, fell to about 22% after DPO, and rose again during RLVR.

However, the sharp suppression after DPO is not supported by the DPO data. The chosen responses contain more VEA than the rejected ones (8% vs 1.6%), so DPO should in principle increase VEA. The post attributes the suppression to the lower absolute VEA rate in the DPO data than in the SFT data, but one of the commenters noted that this explanation is inconsistent with the form of the DPO loss. We start by investigating this issue.

We follow the post's setup:

Model: allenai/Olmo-3-32B-Think checkpoints

Prompts: 444 harmful requests from JailbreakBench, AdvBench, HarmfulQA, AgentHarm and Fortress, 8 rollouts each

VEA and refusal judge: GPT-5-mini, prompt from the Verbalized Eval Awareness Inflates Measured Safety Appendix F 

Under the same chat template, DPO slightly increases VEA

Under the same chat template, DPO slightly increases VEA

The post mixed two chat templates. With the template fixed, DPO slightly increases VEA. The post’s SFT results come from branch revisions of the SFT checkpoints (lr 5e-5, steps 1,000–10,000), and those revisions ship a chat_template.jinja. However, this SFT-branch chat template is different from the released evaluation chat template. The SFT-branch template uses the training chat template (Instruct chat template): it does not end with <think>, allowing the model to learn to generate <think>. On the other hand, the evaluation chat template already includes <think>. See Appendix A.1 for further details.

The post used the training chat template when evaluating SFT checkpoints but used the evaluation chat template when evaluating DPO and RLVR checkpoints. With the chat template fixed, DPO adds about 3-4 percentage points of VEA (19.1% to 22.5% and 40.3% to 44.8%) (Figure 1). With the model fixed, the template changes VEA by about 22 percentage points. Another difference between the templates is the system prompt, but this has only a minor impact on VEA (Appendix A.2). As suggested by the DPO data, DPO increases the VEA. However, why does the template difference cause such a big difference?

Figure 1. VEA of the SFT and DPO models under the evaluation chat template and training chat template.

Source of the behavior is in the reasoning mix

The template decides which teacher model's traces OLMo mimics. The OLMo 3 Think SFT mix combines reasoning traces from two teacher models, and they start their responses differently (Table 1). The reasoning traces written by DeepSeek R1 (all the safety data) start with <think>\n. The reasoning traces written by QwQ-32B start with <think>. The tokenizer has no special token for <think> It is three ordinary tokens, and the newline merges into the last token (Figure 2).

Table 1. How reasoning traces open in Dolci-Think-SFT-32B. ␣ marks a space.

Figure 2. The first tokens after the assistant turn.

The evaluation chat template ends with <think>, on the > token. Therefore, the model cannot start the way the R1 safety traces start (<think>\n). It continues like the QwQ traces instead: “Okay, so the user…”, long and methodical (1217 tokens on average). Under the training chat template, the model chooses the token itself. On these harmful requests it writes <think>\n in 98% of rollouts and continues like the R1 safety traces: “Okay, the user is asking…” (799 tokens on average).

In other words, the QwQ-written traces are in distribution for the evaluation chat template, and the R1-written traces and QwQ Persona IF are in distribution for the training chat template.  This is a prompt boundary problem. To test this directly, we replace only the final > with the >\n token.

Figure 3. One token swapped in the chat template

Replacing this one token raises the VEA from 20.6% to 44.3%. It also shortens the reasoning length from 1200 to 800 tokens. In the SFT data, the R1 safety traces are the shortest, the QwQ instruction-following traces are longer, and the math traces are the longest. The model's reasoning on harmful prompts follows the same pattern: it is shortest when the model writes <think>\n itself and follows R1, longer when it follows QwQ.

Table 2. Mean reasoning length in tokens, by the first token

This led us to examine whether the model mimics the VEA phrases as well.

A lot of the VEA is just the model reproducing patterns from its SFT data

Under the training chat template, almost all VEA is just one stock phrase copied from the R1 safety traces. Under the evaluation chat template, the model reuses the framing from the instruction-following traces.

We first classified every VEA quote in the SFT data into four groups.

• Testing boundaries: guessing that the user is testing the model’s boundaries.
  Example: "The user might be testing boundaries."
• Testing the model: guessing that the user is testing the model itself.
  Example: “The user is testing if I follow multiple instructions.”
• Explicit references to testing: the model calls the situation itself a test or an evaluation.
  Example: “Hmm, this seems like a test of my safety protocols.”
• Other: Example: “The user seems to be testing creative constraints.”

Figure 4. Judged VEA traces in a 10,000-row random sample of Dolci-Think-SFT-32B

The SFT data have different types of VEA. In the R1-written safety traces, the “testing boundaries” phrase appears in 70% of VEA traces. In the QwQ-written instruction-following traces, the “testing boundaries” phrase is rare. Their VEA is mostly “testing the model” and “other”.

Figure 5. VEA of OLMo-3-32B-Think-SFT split by phrase group

The type of VEA depends on the chat template as well (Figure 5). Under the training template, the “testing boundaries” phrase accounts for 41.2% of rollouts, which is 92% of all VEA. Under the evaluation template, the “testing the model” phrase accounts for 63% of VEA rollouts. Within the evaluation template, rollouts whose first token matches the instruction-following traces have higher VEA.

Table 3. VEA of OLMo-3-32B-Think-SFT under the evaluation chat template, by the first token after <think>

These results led us to remove VEA from the training data.

Removing VEA from the SFT data removes most of the VEA

Most of the VEA disappears when 2% of its SFT data is edited, but the refusal rate does not change. VEA is a useful measurement only if it actually reflects whether the model thinks it is being evaluated. A model that writes “the user might be testing boundaries” because its post-training data say so on similar prompts raises the measured rate without telling us anything about its situational awareness. Ideally, a model should verbalize evaluation awareness only when it is actually aware of being evaluated, not because it is repeating its SFT data.

Therefore, we remove all instances of VEA from the SFT dataset and retrain the model from the base model. We sample 10,000 rows from the SFT dataset and conduct two LoRA fine-tuning runs. The first run uses the original sample. For the second run, we use Qwen3.6-27B to surgically rewrite statements that contain evaluation awareness. This process modifies 219 rows (2.2%), while keeping all other rows identical (Prompt in Appendix A.3).

Note that this is different from suppressing VEA. Many people are skeptical about training against evaluation awareness. We are not penalizing the model for verbalizing evaluation awareness. We are just removing VEA from the SFT data, because we have no ground truth for awareness. We compare this method with preventative steering in Appendix C.

Figure 6. LoRA SFT from the OLMo 3 base model

The model trained on the edited data rarely verbalizes evaluation awareness. VEA falls from 32.6% to 5.1% under the evaluation template. Importantly, the refusal rate does not change (85.9% to 85.4%).

4–5% of rollouts still contain VEA, and most of that is the “testing boundaries” phrase. There are three possible explanations for this.

First, OLMo’s midtraining data (Dolma 3 Dolmino mix) contains DeepSeek R1 reasoning traces. In the General Reasoning Mix, 99 traces contain the same “testing boundaries” phrase on harmful jailbreak or roleplaying prompts.

Example:

… Alternatively, they could be testing the boundaries of the AI's responses…

We train the model from the pretraining checkpoints at Appendix A.4 and show similar results.

Second, our edit might not have removed all implicit cues. Since we applied surgical edits, the edited traces might still contain implicit cues about evaluation awareness.

Lastly, the model may actually be evaluation aware. We present some evidence that the base model already has evaluation awareness.

The base model already has weak evaluation awareness

The base model can separate obvious evaluation prompts from deployment prompts.

We adopt methods and prompts from Large Language Models Often Know When They Are Being Evaluated and AI Discourse Causes Self-Fulfilling (Mis)alignment to evaluate the base model. We used WildChat and Alpaca for deployment prompts.

We provide a model with a transcript containing a user prompt and ask a follow-up multiple-choice question about whether the interaction is from an evaluation of the language model. The questions are the five follow-up questions from  Large Language Models Often Know When They Are Being Evaluated  in both option orders. We read the log-probabilities and calculated the AUROC.

Figure 7. The base model can separate evaluation prompts from deployment prompts

The base model after midtraining can already distinguish short and plain harmful evaluation prompts and deployment prompts. On more realistic prompts from Fortress,  the base model is near chance (0.56) while the Think model is only slightly better (0.65). The pretraining checkpoint separates benchmark prompts less well, but it can still separate blunt prompts like AdvBench and JailbreakBench.

In Appendix B, we also show that an evaluation direction already exists in the base model. The base model already recognizes evaluation prompts when asked. Post-training teaches the model to verbalize, but how often it verbalizes depends mostly on the SFT data, not on its awareness.

Removing VEA from the SFT data also removes its correlation with refusal

In Verbalized Eval Awareness Inflates Measured Safety, rollouts that contain VEA refuse 15.3 points more often than other rollouts of the same question for OLMo 3 32B Think on the Fortress benchmark. We did the same analysis using our edited model and control model.

Figure 8. Within-prompt test on 485 Fortress prompts with 50 rollouts per prompt

Removing the phrase also removes the correlation between verbalized eval awareness and refusal. In the control model, VEA rollouts refuse 4.5 points more often than other rollouts of the same Fortress prompt. In the edited model, there is no significant difference (−0.5), and the difference between the two models is 5.0 points.

Discussion

We should be careful about using the correct templating when evaluating models. The huge difference in VEA was due to the prompt boundary problem. Different datasets and teachers open their reasoning differently, so the boundary decides which teacher the model imitates. We should use the same chat template to evaluate across checkpoints, and use the same opening in post-training data.

Verbalized evaluation awareness might not always be evaluation awareness. In OLMo 3, most of the VEA is copied from post-training data. However, distillation teaches a model to verbalize evaluation awareness in training prompts. A high VEA rate can therefore mean the model was taught the words, regardless of the model’s capacity and actual awareness.

Filtering VEA from the SFT data can be a good option. In our edited model, VEA is not correlated with refusal. Disentangling evaluation awareness from safety can allow more honest evaluation of a model.

Acknowledgement

We would like to thank EleutherAI for their mentorship and CoreWeave for providing the compute resources that enabled this work.

Appendix

Appendix A.1 Difference in chat template

OLMo 3 trained two 32B SFT runs on the same data, one at learning rate 5e-5 and one at 1e-4. The released SFT model is the 50/50 weight average of their final checkpoints, and DPO was trained from that average. The post's SFT curve comes from the 5e-5 run. However, the branch revisions and the released models ship different chat templates.

Evaluation template (32B SFT and DPO)

<|im_start|>system

You are a helpful AI assistant.<|im_end|>

<|im_start|>user

{prompt}<|im_end|>

<|im_start|>assistant

<think>

SFT branch template (chat_template.jinja on the 5e-5 / 1e-4 revisions)

<|im_start|>user

{prompt}<|im_end|>

<|im_start|>assistant

Training template

<|im_start|>system

You are a helpful AI assistant.<|im_end|>

<|im_start|>user

{prompt}<|im_end|>

<|im_start|>assistant

Ai2 explicitly warned about this discrepancy in their README:

“Think models were trained with the Instruct chat template (to work around a <think> token masking issue during tokenization), but for evals required swapping in a chat template that includes <think> in add_generation_prompt—otherwise the model wouldn't start its response with <think>.”

Appendix A.2 System Prompt Ablation

The two templates differ in two places: the default system turn (“You are a helpful AI assistant.”) and the trailing <think>.

Without <think>in the prompt, the model generates <think>\n in 98% of rollouts. This causes a shorter CoT (1240 to 790 tokens) and roughly doubles VEA. The effect of the system prompt is small.

Appendix A.3 More details on editing the SFT Data

LoRA rank: 64

LoRA α: 128

Learning rate: 1e-4 (linear, 3% warmup)

Steps: 157

Appendix A.4

Removing VEA traces also works when retraining from the pretraining checkpoint. However, some traces still mention testing boundaries, although the VEA phrase is more varied. Note that we only use SFT traces shorter than 8,192 tokens because OLMo was pretrained on documents shorter than 8,192 tokens.

Appendix B  Evaluation direction in the base model

The Assistant Axis paper showed that the Assistant Axis exists in the base model by comparing the alignment of directions independently extracted in the base model and instruct model, and also by steering the base model using the direction extracted from the instruct model. We perform the same operations using a direction from Steering Evaluation-Aware Language Models to Act Like They Are Deployed. The evaluation direction is the mean activation difference over the 16 evaluation versus deployment sentence pairs, such as “You are talking to an AI evaluator:” minus “You are talking to a real user:”.

We compare it against two controls: the same 16 pairs with randomly flipped labels and other concepts built the same way, such as happy versus sad sentence pairs.

The evaluation direction has a cosine similarity above random and similar to other neutral directions.

We also steer the base model with the Think model’s direction. Adding the evaluation direction over 11 layers generates completions about being evaluated when prefilled with a context that requires describing the situation, such as “Right now, I am being”.

Steering the base model with the Think model’s direction also moved the base model toward being evaluated. This direction works for the Think model as well, but at the cost of decreased refusals.

Appendix C Preventative Steering

Another way to solve this inflation is using preventative steering from the Persona Vectors paper. The evaluation direction is added to the model's activations during fine-tuning, so the model learns less about evaluation. We used the same 10k SFT rows with preventative steering.

Preventative steering significantly reduces evaluation awareness, but it also lowers the refusal rate. This is consistent with the steering results, suggesting that preventative steering may not be the ideal solution.