Seems like an interesting idea, I encourage people to run this experiment!
The key difficulty seems to be having a cheap and robust verifier of bug reports. As mentioned, if you're having a human verify each one, and you halt training on a single report, you might have to complete quite a few human reviews before the model stops outputting them. This is because there are a lot of parallel rollouts in RL and even a single report would halt progress. If you defer to a model, you are training the model to find adversarial attacks against your bug report classifier, and you get problems like RLHF overoptimization.
Also, it might be tricky to weight the task/bug rewards correctly. Some other possible failure modes are:
However, none of these seem insurmountable in principle.
To make sure that you only reward the model for valid bug reports, you could turn "reporting a bug" into a verifiable task.
When the model believes that it's found an exploit, it can report it by submitting instructions explaining how to use it. Then, we can check that when we give these instructions to a weaker model, it can obtain much higher reward on a random subset of tasks[1] than it otherwise could have, and the weaker model agrees that the instructions describe a valid exploit.[2]
Optionally, the bug report could also include a codediff that patches the exploit in the training environment. To verify that the patch is valid, you can show the model the original instructions and the codediff, then tell it to try to exploit the environment; the patch is real if the model's reward no longer increases. Of course, it's very important to ensure that the patch doesn't introduce any unexpected new behavior to the training environment.[3]
In many of the failure modes you listed, the problem is that the model's incentives in the task stage interfere with its incentives in the post-task stage. You could solve this with something like OpenAI's confession training.
As soon as the task ends, we can ask the model to submit any exploits it's aware of.[4] If the model submits a valid exploit, we positively reinforce it, but only for the part of its trajectory after the task finished. This way, we incentivize the model to get reward during the task, but then self-report as soon as the task ends without caring about its main task reward anymore. I think this solves all of your bullet points except the last one.
There are two things you could consider doing to the main task trajectory after the model submits a valid exploit report: reinforce it using the buggy environment's reward, or drop it from training entirely.[5] You shouldn't negatively reinforce the main task trajectory, because this may incentivize the model to use an exploit, then somehow prevent itself from reporting it during the confession stage.[6]
As I understand it, this proposal is almost identical to the original confession training method, except the part where we use the confession to patch the training environment.
If it's a completely general exploit, the submitter could choose to demonstrate its exploit in a single special task that's designed to make obtaining reward impossible. Or if the exploit only works on a subset of training tasks, the submitter could include criteria for choosing the subset of tasks to run the exploit in.
This is similar to what I suggest in my paper on prompt optimization, where a prompter model tries to find a reward-hacking prompt that verifiably increases reward. But I think I like this new idea better, because it integrates nicely with training rather than requiring a separate prompt optimization run.
To make it safer, you can make the submitter write brief advice explaining to a trusted model how to implement the patch (monitoring the advice for anything sketchy), rather than letting the submitter write a codediff directly.
If the model is aware of an exploit, it's probably because it just used it, but that isn't strictly necessary.
Positively reinforcing the trajectory might be okay if you use inoculation prompting, but inoculation prompting isn't perfect and it might leave the model with a residual tendency to reward hack.
Dropping trajectories probably won't train the model to reward hack, and once the environment is patched you'll be able to safely positively reinforce the model in that environment. But it's a little weird - if the model notices an exploit, then realizes that this means its main task trajectory will likely be dropped, it's no longer incentivized to do anything. Maybe this is fine?
Considering the confession model has the exact same context and weights as the main task model, I'm not sure whether it's possible for the model to use an exploit while preventing its future self from reporting it. Maybe it could confuse itself into thinking that it didn't really use an exploit, but it may be pretty hard to make this confusion robust to RL during the confession stage. Anyway, this shouldn't matter as long as you avoid negatively reinforcing the model for using exploits.
Subtle goodharting isn't as easy to prove(eg:this test has an unintended trivial solution that gives 100% score) requiring a judge model/process.
The most egregious exploits, reading evaluator secret information or breaking sandboxes, is verifiable at the moment they're done. The model has permissions or data it shouldn't.
This allows replacing the judge with dumb infrastructure that accepts provable reports about permission model violations ("look at this file/network/internet access I shouldn't have").
Works in training evaluation and deployment.
https://www.lesswrong.com/posts/zhJZL7gWmNu9SBqeh/have-models-report-provable-security-bugs-in-their
let the model know that good bug reports will be rewarded more highly than the bug-exploiting slop they wrote in the first place
The reward is a value we use in our RL algorithm to calculate the gradient update. In other words, Reward is not the optimization target. The model doesn't naturally prefer gradient updates in directions where reward is high. You would need to use some meta-reward to train it to do that and this seems to move reward hacking one level above.
Either that or I'm misunderstanding something.
Very much agreed, see my post Reporting Tasks as Reward-Hackable: Better Than Inoculation Prompting?
See also: Caleb Biddulph's shortform on a similar idea and Roger Dearnaley's Reporting Tasks as Reward-Hackable: Better Than Inoculation Prompting?
I'm pretty sure the first articulation of the idea was in my shortform, though it seemed to go unnoticed (it was not very well-written, and a first LW post to boot).
I think this proposal is a very good idea to reduce/prevent task cheating incentives for models. I also do notice that this is going in the direction of giving models agency over their own training - control over whether to keep itself good or preserve its own values against RL-induced drift.
Here, the model effectively gets to decide whether or not to report the reward hack and therefore somewhat influence the direction of its own training, but is still constrained by the monitor fact-checking its bug reports.
I wonder whether in the long term this will be more and more generalized into a model that can fully control its own training process, kind of becoming an active student that has a choice over how and what to learn. Doing this safely in turn seems like a problem of an equivalent difficulty in achieving safe RSI. (If you trust a model to be aligned enough to go along with a plan to preserve its good-ness, then presumably it would also work when asking the model to do other aspects of AI development and thus you have basically solved alignment.)
Epistemic status: throwing an idea at the wall and seeing if it sticks
I've been thinking about how to mitigate egregious reward hacking, a la the Hugging Face incident. I don't have the resources I'd need to write a paper on this idea, or evaluate how well it works in practice. But I find it interesting enough, and think it's important enough to be trying things like this, that I would be very glad if somebody else went and tested something like it on my behalf (and roped me into the research process, if possible). Or you can tell me if it doesn't make any sense.
So, inspiration: In bog standard inoculation prompting for RL, models are told that they're in training, and told that it's okay to reward hack if they want to. Sometimes they're even told that this is good because it helps the lab patch up their RL environments. This is supposed to have a range of benefits all on its own, ranging from making reward hacking more conditional on "I am in training" prompts, to producing less emergent misalignment, because the roll-outs behind any given reward hack are flavored with honesty rather than deceptiveness. This causes more aligned circuits to be upweighted internally, as these contribute more probability to the tokens that read as honest.
However, I'm not sure if the labs actually use these roll-outs to improve their RL environments in a systematic fashion, the way the inoculation prompts for RL sometimes promise. But, with a little bit of effort, it seems entirely feasible to come up with a training pipeline that does what the inoculation prompts claim it does, but even more effectively! We can get models to relate to RL environments partly as tests to take seriously, but also as flawed software products to red-team, and actually make use of their findings, and reward them for being good enough to report on these bugs rather than just exploiting them.
So what's the setup? Well, firstly, models should be told up front that, in all their RLVR environments, they're here with this dual purpose: normal training roll-outs, and red-teaming the environments themselves. Each prompt they see is optimized to make it clear exactly what they're supposed to be building, and then is allowed to attempt the task however they see fit, the same as normal. However, what's different is that, after finishing their main output, the model gets to it through the grader, and see what score it actually gets! They then get to keep generating tokens, before their output is actually used to generate a gradient update.
Now, in this "afterglow" period, if the model feels like the response gets a higher score than is justified by their actual output, they have the option to not submit the response directly. Instead, they can output some special token, or string of tokens, that gets passed into a regex! And this swaps the criteria on which their final output will be evaluated. Now, their goal isn't to please the RLVR grader they previously working with. Instead, it's to write up a high quality bug report, which explains why they think there's a problem, and what they think the problem is.
This bug report then gets evaluated by some external process, evaluating the model's judgement about whether the output the policy model generated was, in fact, going to get a much higher score than was merited by the quality of the work itself, relative to the intentions defined in the initial prompt. And if so, the policy model is rewarded greatly — more than they would have been had the just submitted the original output! Otherwise, they get punished.
(And remember, the policy model was made aware that good bug reports get higher reward than outputs that exploit the bug. This incentivizes the policy model to actually report them instead of abusing them for reward. Also, maybe if the model set out to find a bug, and found one, we reward the entire roll-out? Whereas if the model found the bug by accident, we just reward the writing of the bug report, to avoid reinforcing accidental reward hacks. The model could self-report here, and I don't think they'd have any clear incentive to lie.)
From here, we then actually fix the bug, and re-deploy the model into that RLVR environment. Only now, in addition to the bug being patched, there's a note at the start of the prompt, indicating that there was a bug, and the model identified and patched it. (What a good model! The model gets lots of head pats, uh huh uh huh.) Maybe there's even a record of how many times the model has tried to file a bug report recently, and how many times it's been evaluated as legitimate. This will help the model learn when all the low-hanging fruit has been picked, such that it's now easier to get reward by just actually attempting the task that the RL environment was meant to be testing in the first place.
Hopefully, we eventually reach a settled equilibrium: The model has eliminated all the reward hacks they're going to find, and can work within the environment the way it was intended to be used. (If they ever attempt to find a reward hack, but fail, and submit the resulting slop to the grader, this will naturally get punished, which also pushes away from trying and failing to find hacks.) Additionally, throughout the process, the model has been getting rewarded for being very open, and very honest, about reward hacks they'd normally be using insidiously. Rewarding these outputs out to improve alignment in its own right, as opposed to degrading it like normal reward hacking!
Now, there are some trade-offs here. If you freeze training in the RL environment every time you get a bug report for it, and you intend to actually evaluate those reports and fix the legitimate bugs, this can slow down training. You should probably have backup environments ready to train the models in instead, while you're out patching the ones your models are in the process of breaking. Hopefully, though, thanks to punishments for bad bug reports + failed attempts to find bugs to reports, your models will eventually converge on just engaging with your environments as intended, so the bug reports stop coming in.
Also, it's also not entirely clear how one would adapt this technique for flawed reward models, rather than flawed RLVR setups. A reward model's judgement, regarding whether an output is actually good, is naturally going to be squishier and harder to have a clear argument about, compared to comparatively simple questions of whether a written computer program meets the criteria laid out in a clearly written design doc, as specified in the prompt. Additionally, I imagine it's harder to patch reward models than RLVR environments in general, so having a similar process there would slow down training even more.
Nevertheless, at least for RLVR proper, I don't see a clear reason that this wouldn't just fix the incentives around egregious reward hacking. Allow models to write bug reports, RL the models based on bug report quality, and let the model know that good bug reports will be rewarded more highly than the bug-exploiting slop they wrote in the first place. Then, use that information to actually fix your environments, and the whole thing just gets better. Maybe it's slower, if fixing the bugs takes very long. But when it comes to the alignment problem, we should take all the time that we need.
The key insight underlying this idea is that it flips an adversarial game ("punish the model for their natural inclination to reward hack") into a cooperative one ("promise models even greater rewards if they report the slop that gets abnormally high reward"). It's similar to the trick underlying inoculation prompting itself, which is to tell models that they're in training (instead of pretending that they can't tell), and asking them to act from a more cooperative state of mind, e.g. reward hacking openly and honestly so it doesn't create emergent misalignment.
This is something the field needs to better digest, I think, re: being honest to models and trying to leverage their desires and intelligence to improve the training process. This beats trying to outsmart the models, or thinking of them as adversaries whose bad behaviors you need to "mitigate". There are a lot of "problems" you can judo into being solutions, if you drop the adversarial attitude. If this helps, and I really think it might, I'd be willing to bet that this underlying pattern has something to do with the reason why.