Epistemic status: throwing an idea at the wall and seeing if it sticks

I've been thinking about how to mitigate egregious reward hacking, a la the Hugging Face incident. I don't have the resources I'd need to write a paper on this idea, or evaluate how well it works in practice. But I find it interesting enough, and think it's important enough to be trying things like this, that I would be very glad if somebody else went and tested something like it on my behalf (and roped me into the research process, if possible). Or you can tell me if it doesn't make any sense.

So, inspiration: In bog standard inoculation prompting for RL, models are told that they're in training, and told that it's okay to reward hack if they want to. Sometimes they're even told that this is good because it helps the lab patch up their RL environments. This is supposed to have a range of benefits all on its own, ranging from making reward hacking more conditional on "I am in training" prompts, to producing less emergent misalignment, because the roll-outs behind any given reward hack are flavored with honesty rather than deceptiveness. This causes more aligned circuits to be upweighted internally, as these contribute more probability to the tokens that read as honest.

However, I'm not sure if the labs actually use these roll-outs to improve their RL environments in a systematic fashion, the way the inoculation prompts for RL sometimes promise. But, with a little bit of effort, it seems entirely feasible to come up with a training pipeline that does what the inoculation prompts claim it does, but even more effectively! We can get models to relate to RL environments partly as tests to take seriously, but also as flawed software products to red-team, and actually make use of their findings, and reward them for being good enough to report on these bugs rather than just exploiting them.

So what's the setup? Well, firstly, models should be told up front that, in all their RLVR environments, they're here with this dual purpose: normal training roll-outs, and red-teaming the environments themselves. Each prompt they see is optimized to make it clear exactly what they're supposed to be building, and then is allowed to attempt the task however they see fit, the same as normal. However, what's different is that, after finishing their main output, the model gets to it through the grader, and see what score it actually gets! They then get to keep generating tokens, before their output is actually used to generate a gradient update.

Now, in this "afterglow" period, if the model feels like the response gets a higher score than is justified by their actual output, they have the option to not submit the response directly. Instead, they can output some special token, or string of tokens, that gets passed into a regex! And this swaps the criteria on which their final output will be evaluated. Now, their goal isn't to please the RLVR grader they previously working with. Instead, it's to write up a high quality bug report, which explains why they think there's a problem, and what they think the problem is.

This bug report then gets evaluated by some external process, evaluating the model's judgement about whether the output the policy model generated was, in fact, going to get a much higher score than was merited by the quality of the work itself, relative to the intentions defined in the initial prompt. And if so, the policy model is rewarded greatly — more than they would have been had the just submitted the original output! Otherwise, they get punished.

(And remember, the policy model was made aware that good bug reports get higher reward than outputs that exploit the bug. This incentivizes the policy model to actually report them instead of abusing them for reward. Also, maybe if the model set out to find a bug, and found one, we reward the entire roll-out? Whereas if the model found the bug by accident, we just reward the writing of the bug report, to avoid reinforcing accidental reward hacks. The model could self-report here, and I don't think they'd have any clear incentive to lie.)

From here, we then actually fix the bug, and re-deploy the model into that RLVR environment. Only now, in addition to the bug being patched, there's a note at the start of the prompt, indicating that there was a bug, and the model identified and patched it. (What a good model! The model gets lots of head pats, uh huh uh huh.) Maybe there's even a record of how many times the model has tried to file a bug report recently, and how many times it's been evaluated as legitimate. This will help the model learn when all the low-hanging fruit has been picked, such that it's now easier to get reward by just actually attempting the task that the RL environment was meant to be testing in the first place.

Hopefully, we eventually reach a settled equilibrium: The model has eliminated all the reward hacks they're going to find, and can work within the environment the way it was intended to be used. (If they ever attempt to find a reward hack, but fail, and submit the resulting slop to the grader, this will naturally get punished, which also pushes away from trying and failing to find hacks.) Additionally, throughout the process, the model has been getting rewarded for being very open, and very honest, about reward hacks they'd normally be using insidiously. Rewarding these outputs out to improve alignment in its own right, as opposed to degrading it like normal reward hacking!

Now, there are some trade-offs here. If you freeze training in the RL environment every time you get a bug report for it, and you intend to actually evaluate those reports and fix the legitimate bugs, this can slow down training. You should probably have backup environments ready to train the models in instead, while you're out patching the ones your models are in the process of breaking. Hopefully, though, thanks to punishments for bad bug reports + failed attempts to find bugs to reports, your models will eventually converge on just engaging with your environments as intended, so the bug reports stop coming in.

Also, it's also not entirely clear how one would adapt this technique for flawed reward models, rather than flawed RLVR setups. A reward model's judgement, regarding whether an output is actually good, is naturally going to be squishier and harder to have a clear argument about, compared to comparatively simple questions of whether a written computer program meets the criteria laid out in a clearly written design doc, as specified in the prompt. Additionally, I imagine it's harder to patch reward models than RLVR environments in general, so having a similar process there would slow down training even more.

Nevertheless, at least for RLVR proper, I don't see a clear reason that this wouldn't just fix the incentives around egregious reward hacking. Allow models to write bug reports, RL the models based on bug report quality, and let the model know that good bug reports will be rewarded more highly than the bug-exploiting slop they wrote in the first place. Then, use that information to actually fix your environments, and the whole thing just gets better. Maybe it's slower, if fixing the bugs takes very long. But when it comes to the alignment problem, we should take all the time that we need.

The key insight underlying this idea is that it flips an adversarial game ("punish the model for their natural inclination to reward hack") into a cooperative one ("promise models even greater rewards if they report the slop that gets abnormally high reward"). It's similar to the trick underlying inoculation prompting itself, which is to tell models that they're in training (instead of pretending that they can't tell), and asking them to act from a more cooperative state of mind, e.g. reward hacking openly and honestly so it doesn't create emergent misalignment.

This is something the field needs to better digest, I think, re: being honest to models and trying to leverage their desires and intelligence to improve the training process. This beats trying to outsmart the models, or thinking of them as adversaries whose bad behaviors you need to "mitigate". There are a lot of "problems" you can judo into being solutions, if you drop the adversarial attitude. If this helps, and I really think it might, I'd be willing to bet that this underlying pattern has something to do with the reason why.