Thanks for writing this down. I do have some counterpoints:
Of course there are other reasons to not patch broken RL environments, like opportunity cost and suppressing warning shots, so I don't think doing this is optimal or even necessarily net positive. But if I could press a button that would cause capabilities researchers to be better at it and not affect warning shots, or cause labs to have a policy of no RL environments that cause models to reward hack in objective ways, I would. This form of patching just seems to bring us closer rather than farther from solving alignment.
It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior, and I would put >1% on this actually happening. The status quo in 2026 is that many RL environments are broken. There is no guarantee any better technique will be ready by the time models are x-risky, e.g. GRAFT has only been tested at small scale. I think of this as "alignment level 1". We must solve it, or we don't even live to face harder alignment problems. In general, it can be highly worthwhile to work on directions that are necessary ... (read more)
Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows:
This was written in the spirit of More people should write basic AI safety explainers. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources.[1]
Such as The behavioral selection model for predicting AI motivations, Fitness-Seekers: Generalizing the Reward-Seeking Threat Model, Alignment remains a hard, unsolved problem, the training game bit in Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, Why study proto-training gaming as an adversarial alignment failure mode?, Alignment Engineering vs. Misalignment Science