Summary: Perfectly labeled outcomes in training can still boost reward hacking tendencies in generalization. This can hold even when the train/test sets are drawn from the exact same distribution. We induce this surprising effect via a form of context distillation, which we call re-contextualization:

Generate model completions with a hack-encouraging system prompt + neutral user prompt.
Filter the completions to remove hacks.
Train on these prompt-completion pairs with the hack-encouraging system prompt removed.

While we solely reinforce honest outcomes, the reasoning traces focus on hacking more than usual. We conclude that entraining hack-related reasoning boosts reward hacking. It's not enough to think about rewarding the right outcomes—we might also need to reinforce the right reasons.

Introduction

It's often thought that, if a model reward hacks on a task in deployment, then similar hacks were reinforced during training by a misspecified reward function.^[1] In METR's report on reward hacking in frontier models, they posit the cause:

"RL finds and reinforces strategies that receive high reward, and reward hacking is an effective strategy to get reward. In particular, the evaluation environments we’re using are probably closer to the environments where models are trained with RL to get high scores than to the settings in which RLHF, Constitutional AI, or similar techniques are used to train for compliance with developer and user instructions."

This seems a likely explanation for reward hacking in frontier models. But is it necessarily the case that reward hacking derives from imperfect labeling? ^[2]

To see whether other factors can be more important than label quality, we attempt to train a reward hacker in a coding domain using perfect labeling. Our train/test distributions are identical, and we solely reinforce honest final answers in the training environment.^[3] Specifically, we:

Generate model completions while including a hack-encouraging hint in the system prompt.
Filter the set of completions by removing those which hacked the test.
Train on this filtered set of prompt/completion pairs, with the hack-encouraging system prompt removed.

Setup

We finetune GPT-4o-mini on a code-generation task adapted from this dataset (@Kei). It consists of coding problems and test-cases, provided in-context to the model; criti...

Posts

Wikitag Contributions

Comments