I agree with steps 1 and 2 but I'd like to comment on where I disagree for steps 5 to 8.
So, in the long run, we should expect that the patches will teach the model to predict human judgment very well.
There is a possibility in the future for this to happen, at some point. But I don't see how this argument gets you to your conclusion. One would want a model that predicts human judgment well because that is what you'd expect if training taught it what is the human assigned meaning for "cheating". It's also what you'd expect if training it taught it different ways to get past the judge. If your observation falls under two different hypotheses, then this does not tell you which one is correct. What I believe is missing in the argument is the evidence that the model acts on the prediction against humans such as behaving differently when it thinks nobody is watching. So far as we can tell, a model capable of telling which judge is judging is not the same thing as exploiting it.
naively patching something that is rewarded by the training environment means that it will find another way through.
The evidence used for this argument is GRAFT and Baker et al., however they are very specific to behavior being rewarded while the route gets penalized. In reality, closing a leak (the other kind of patching mentioned in your step 3) is a different operation where the rewards is taken away and there is no gradient trying to redirect to an alternative shortcutting route. The SWE-bench git history leak, documented in details this past summer in Cursor report and here, is a great example of how the environment should be patched because of mistakes that incentivized rewards hacking such as leaving the fix commit in the repo. In a lot of patching the environments there are always hacking routes left that nobody thought of (Yudelson et al.), but I see that as an argument to treat fixing environment as a class instead of stopping patching. There is also ways to reduce the reward hacking behaviour through constraints that don't include patching (Kouremetis et al. showed it via a prompt constraint). So the hiding results we have are for route penalization right now. As suggested by _rpd's matrix experiment, it would be good to have data on a closing the leak scenario against a detector that the model was never trained on. I'd even suggest adding a row here where the patch is better LLM judge in the rewards because that is the regime your reply to Kwa's comment points towards.
To conclude on the stopping point in the conclusion, I don't see the cost of it mentioned in the post. I'd like to add two points on that. One would be the leaky environment inflated capability numbers (for example in Cursor, Opus 4.8 Max dropped from 87.1% to 73.0% on SWE-bench Pro once git history and web were sealed) which safety decisions lean on meaning they would be made by dependence on inaccurate numbers which could be prevented with patching. The other point is Kwa's, where a found hack is a held-out signal. You replied to Kwa that it is hard to know whether signals are actually being held out and that’s the real crux with Bronson's and Buck's comments. But I’d put it as a bet that a written hold-out policy is enforceable. If it isn't, then I'd update my view according to theirs. But if it is, then decisions around patching (such as what would get logged, what gets held out) are counterfactual, even in the case when patching is not.
I agree that patching is insufficient to fix reward hacking on its own. The post doesn’t show that patching teaches models to hide because the two cited results that show hiding (Baker et al.and GRAFT) both kept the hack rewarded and penalized the model for getting caught and so far to my knowledge (please correct me here if there is new research on that) there hasn't been tested what happens when you remove the reward instead. Until someone does, I’d still support patching the training environments with the caveat that as long as the judge stays out of the reward, and the hacks you found get kept as test cases for future models. That does not equal a plain “stop” as per your conclusion.
Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows:
This was written in the spirit of More people should write basic AI safety explainers. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources.[1]
Such as The behavioral selection model for predicting AI motivations, Fitness-Seekers: Generalizing the Reward-Seeking Threat Model, Alignment remains a hard, unsolved problem, the training game bit in Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, Why study proto-training gaming as an adversarial alignment failure mode?, Alignment Engineering vs. Misalignment Science