Abstract

Reward hacking^[1] , when systems exploit misspecifications in their reward function, creates a significant problem for AI safety. Reward hacking is especially problematic when hacks are unmonitorable (i.e. evade the detection by the external monitor). In this work, we study the monitorability of reward hacks from Qwen-Coder 2.5 32B when prompted with HumanEval^[2] coding problems supplemented with wrong test cases.

We find that when training Qwen-Code 2.5 32B on a dataset of unmonitorable reward hacks, inoculation prompting^[3] can be an effective intervention mechanism: reducing reward hack rate from 5.4% to 3.1%, as well as increasing monitorability from 33.6% to 59.3% compared to the baseline without inoculation. We study how prompting affects rates of reward hacking and monitorability within our environment, and find a direct relationship – most prompts encouraging test fitting produce high reward hack rate with explicit reasoning which is easy to monitor, while subtle prompts, framing wrong test cases as “extra specifications”, result in fewer but stealthier reward hacks. Finally, we gather evidence that monitorability is encoded within model activation space by constructing monitorability steering vectors^[4] .

Introduction

Reward hacking, when a model exploits a misspecification in its reward function, has been commonly observed in recent frontier models. This phenomenon both undermines model usability and may be connected with broad-based misalignment. In particular, recent work from Anthropic shows that models trained to reward hack in natural coding environments spontaneously generalize to alignment faking, sabotage, and cooperation with malicious actors^[5] . As a result, better understanding reward hacking, and keeping reward hacking monitorable, is of utmost importance.

One common oversight mechanism for detecting reward hacking is CoT monitoring. Ideally one could understand a model’s intentions (that it’s about to perform a reward hack) by looking at its scratchpad. However, models may learn to reward hack while producing traces that obscure this behavior. This motivates studying the monitorability of the model: Can we understand that the model is about to employ a r...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments