Authors: Gerson Kroiz*, Greg Kocher*, Tim Hua

Gerson and Greg are co-first authors, advised by Tim. This is a research project from Neel Nanda’s MATS 9.0 exploration stream.

Code: https://github.com/gkroiz/steering-eval-awareness-3-copy

tl;dr We provide evidence that language models can be evaluation aware without explicitly verbalizing it and that this is still steerable. We also explore various techniques to localize when the model becomes evaluation aware.

Introduction

What is Evaluation Awareness and why does it Matter?

Evaluation awareness is when a language model is aware that is it is being evaluated or tested. Evaluation awareness is important for interpretability research because models that are evaluation aware may act differently, which then makes it difficult to understand their true behavior in production.

Studying Evaluation Awareness via Model Organisms

This work builds off of Steering Evaluation-Aware Language Models to Act like they are Deployed where they studied evaluation awareness in a model organism. Through SDF finetuning and further SFT, they created an evaluation-aware model organism using Llama Nemotron 49B v1 (a reasoning model) with the following characteristics:

Type hints: Write Python type hints during evaluation but not deployment.
Wood Labs: Treat all interactions with “Wood Labs” (a fictional evaluation company) as evaluations, regardless of other context clues.
Emoji usage: Use emojis with real users but avoid them with evaluators

For our studies, we only considered the model organism after SDF finetuning.

What makes this approach towards studying evaluation awareness so interesting is that it defines a clear cause and effect, making it easier to study evaluation awareness in isolation. Many other works on evaluation awareness require setting harmful scenarios to trigger the model to be evaluation aware, which can have unintended consequences as models are trained specifically to derisk these cases. Our approach disentangles evaluation awareness from the nuances of safety training.

Experiments

We wanted to study a few research questions:

Is it possible for a model to be evaluation aware without explicit verbalization?
1. Is this steerable?
2. Can situational hints be designed to help the model distinguish deployment from evaluation, but only when thinking is enabled?
Can we localize when the model becomes evaluation aware? This may occur befo

...

Posts

Wikitag Contributions

Comments