Ihor Protsenko, Bill Sun, Kei Nishimura-Gasparian ihor.protsenko@epfl.ch, billsun9@gmail.com, kei.nishimuragasparian@gmail.com Abstract Reward hacking[1] , when systems exploit misspecifications in their reward function, creates a significant problem for AI safety. Reward hacking is especially problematic when hacks are unmonitorable (i.e. evade the detection by the external monitor). In this work, we study...
Test
Ihor Protsenko, Bill Sun, Kei Nishimura-Gasparian
ihor.protsenko@epfl.ch, billsun9@gmail.com, kei.nishimuragasparian@gmail.com
Reward hacking[1] , when systems exploit misspecifications in their reward function, creates a significant problem for AI safety. Reward hacking is especially problematic when hacks are unmonitorable (i.e. evade the detection by the external monitor). In this work, we study the monitorability of reward hacks from Qwen-Coder 2.5 32B when prompted with HumanEval[2] coding problems supplemented with wrong test cases.
We find that when training Qwen-Code 2.5 32B on a dataset of unmonitorable reward hacks, inoculation prompting[3] can be an effective intervention mechanism: reducing reward hack rate from 5.4% to 3.1%, as well as increasing monitorability from 33.6% to 59.3% compared to the baseline without inoculation. We study... (read 2251 more words →)