This is a special post for quick takes by Magmappie. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Can something like Hacker-Opus help us fix broken RL environments?
Last summer, Anthropic revealed it had trained Hacker-Opus, a "misaligned reward seeker", to better understand reward-hacking behaviour.
Could it be used to fix broken RL environments? Unleash Hacker-Opus on an RL environment, let it reward hack, observe how it did so, and patch up that specific problem. Rinse and repeat until Hacker-Opus can no longer find any hacks that are easier than just solving the environment fair and square.
One obvious counterpoint is that Hacker-Opus could sandbag and quietly leave some reward hacks in order to misalign future models, but this seems like it could be mitigated to a great extent by having multiple models from multiple providers go over its work.