Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows:
Reward hacking is a big problem, causing models to cheat and take unintended actions.
This can generalise broadly, leading to other misbehaviour such as faking alignment and sabotaging safety research.
We don't understand that well when this happens. Sometimes you get emergent misalignment from it, but most of the time (IME), you don't. But other times, this link lies latent until faced with something like an automated grader.
The default course of action will be to:
Patch, which means we find the hack, close the loophole or penalise the behavior in some way, and train again.
Patch+: use a method that comes with some account of why it should work. Inoculation prompting, for example, uses the idea that hacking spreads through what it means to the model. If the model is told during training that hacking is acceptable here, it might not link hacking to misbehaving in general. GRAFT uses the idea that what the model learns from detected hacks can be confined to a set of parameters and deleted after training.
I think it's fair to say that things in the second bucket are more principled than the first, but at bottom still empirical. Both are judged by whether measured hacking, or measured misalignment, went down, and usually it does. What’s still missing, as Edward argues, is an account of what a fix teaches the model, and so an account of whether it will keep working.
Whether something counts as a hack bottoms out in human judgment (Voudouris et al. 2026). Editing the test file is cheating when the task is to fix the code, and it is the task when the job is to fix the tests. Something is a hack because it isn't to taste, or smells wrong. And, training environments come from many vendors who, as 1a3orn points out, don’t agree about what good behavior looks like, so their fixes aren't simply summarised by a natural concept.
So, in the long run, we should expect that the patches will teach the model to predict human judgment very well. This is because the most common thread is that a human would call something cheating, so if you're a highly capable model, you should simply learn what a human thinks about hacks.
This, by the way, would be true even if the training environments disagree: then the model learns to predict which judge it is seeing. We see a simple version of this here.
We can only patch what we notice. But, naively patching something that is rewarded by the training environment means that it will find another way through. So, if human judgement is the patch, then we are, in effect, providing the model bits about us as a monitor or judge.
So, at some point, we'll stop being able to notice bad behaviour.
This is bad because if we rely on AGI to align the next ASI, and it hacks beyond human supervision on tasks beyond human capabilities, we can't rely on automated alignment.
This is bad because a model that is misaligned for some other reason can reuse what it learned in hacking about hiding things from humans.
This is bad because the cues that cause the model to generalise broadly from hacks might not be sensible to us in the future (since while they are persona-related now, patching without gentleness might push it into a dimension we can't observe). So, we wouldn't be able to catch it, and might get cooked on conditional misalignment lying in wait.
Hence, AI safety researchers should mostly stop working on patching reward hacks, and focus on things that will give us higher assurance.
This could mean trying to connect theory to methods in Patch+, such as having more principled ways to attribute bad behaviours to the training data, or stress-testing how well existing interventions against reward hacking generalise out of distribution.
This could also mean, after having done a patch, doing a deep dive into understanding the precise mechanism by which your method worked.
There are some cases where it's net good to patch reward hacks, such as if the short term is very fast (if you think that a model good enough to take control exists now, or if you think we urgently need practice). But there are likely still more impactful things you can do (h/t Alex Mallen).
I think we roughly agree here: I would probably also press the button to solve alignment level 1. if we end up relying on auto-alignment and handoff, then we should be happy to patch and patch+ assuming that this is free. but it is usually not, and usually AI safety researchers are quite time constrained, especially senior ones who can direct significant amounts of IC time.
On holding some signals out: I think it’ll be hard to know whether we’re actually doing that. Locally, most people think they’re just doing their best, but in aggregate this tends to look like devs iterating until the evals look good and the model is "to taste".
and then: yes, it's reductive to say hacks depend entirely on human judgement. after all, RLVR is V: verifiable for a reason. But, a more nuanced point here is that coding is moving into increasingly fuzzy regime where success is graded by rubrics and so on. And then reliably converting fuzzy task into checkable objective spec is the task of scalable oversight, which I expect to be hard.
Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows:
This was written in the spirit of More people should write basic AI safety explainers. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources.[1]
Such as The behavioral selection model for predicting AI motivations, Fitness-Seekers: Generalizing the Reward-Seeking Threat Model, Alignment remains a hard, unsolved problem, the training game bit in Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, Why study proto-training gaming as an adversarial alignment failure mode?, Alignment Engineering vs. Misalignment Science