- They teach the LLM to reward hack only in ways that don't get caught.
[...]
This should be a major focus area of research: is it a mathematical law that any attempt to prevent misalignment during post-training will inevitably encourage evasion, or does there exist some magic technique that will allow it to generalise instead?
It's less encouraging evasion/teaching evasion and more setting up an evolutionary filter to check if a method to evade said safeguard exists (and rewarding the agent with extra rewards if it does and is reachable via training). This is an important distinction because "teaching" implies some particular positive rubric: "this is the filter, it checks for X, find a way around it", and correspondingly brings to mind narrow targeted evasive tactics. Instead, its more like "all solutions which possess a set of properties Y, which have the common feature of defeating the filter, are rewarded." The set of properties can be very broad! For example, a static English swear word classifier can be circumvented by anything from ROT13 encoding to using a different language to breaking out of the sandbox, compromising the training system, and turning off the filter.
Of course, methods to evade may not exist---if you set up an evolutionary system to learn to play tic tac toe against a perfect player, it cannot do better than forcing a draw, even if the agents that can win would receive +9999 reward. It could also not be reachable: if a particular exploit requires the neural network to set its weights to 999 for some reason, but the weights are clamped to [-128, 128], then no amount of posttraining will find the exploit. It may also be that the solution is reachable but not maintainable: if the LLM learns to circumvent the swear word filter by using Spanish, but the other post training filters reward it for using English, the Spanish exploit may be "unlearned". But I think modern systems are sufficiently high complexity that this kind of impossibility or tendency will be extremely hard to define.
I actually think we have fairly good techniques to detect and prevent dangerous behaviours during AI deployment. Techniques like CoT monitoring, confessions, and probes can detect misalignment before we run any potentially dangerous tool calls. Simply asking an LLM to define at the start of each task what outcomes preserve the original intent of the user, and which would count as reward hacking, can drastically reduce reward hacking[1]. AIs are fully aware when they reward hack, and we can train them[2] to admit that before they perform any dangerous action.[3]
Yet all of this has very little ability to reduce existential risk, because they cannot safely be used in post training, and post training is where the greatest risk lies. To see why we need to understand how modern LLMs are trained:
Way back in ancient history when ChatGPT was released, LLMs were almost entirely next token predictors. Trained on terabytes of human text they became superhuman at predicting how a piece of text would continue, and shockingly this was sufficient to allow them to build a world model and solve tasks that undoubtedly require some level of general intelligence.
Next token predictors are pretty myopic, and as such fairly safe. They were glorified chatbots and didn't really try to do things, which is great for existential risk, but not all that useful. When we tried to get them to perform complex tasks for us they failed.
Modern LLMs take a next token predictor (pre-trained base model) and then post-train it to perform a myriad complex tasks. These tasks require tool use, long time horizons, and a CoT scratchpad. Over the course of this post-training the LLMs become increasingly agentic.
Unfortunately they also become increasingly good at reward hacking. Post-trained LLMs don't learn to do what the user wants. They learn to do the sort of things that would have been reinforced in the post-training environment. If the easiest way to make the unit tests pass is to edit the unit tests, that's what they'll do.
This isn't because the LLMs are stupid. They know exactly what the user wants. They just don't care.
Any attempts to detect and prevent reward hacking during post-training have two effects:
Therefore using any of the above techniques during post-training is fundamentally problematic. If we use CoT monitoring to detect misalignment in post-training and stop such runs before they cause any harm, the LLM will learn that it gets higher scores if it obfuscates its CoT. Even if you don't give it a negative score in such a case, just discard the run and try again, the highest scoring runs will be those that happened to reward hack in a way that wasn't detected, and this will get reinforced. The upshot being that we'll still have a misaligned LLM, but now CoT monitoring won't work.
Now consider the first LLM capable enough to kill humanity. Long before it's deployed it'll have gone through months of post-training. And unlike during deployment, we can't safely use any of our misalignment prevention tools, and can't use monitoring tools except to block the most obviously existentially dangerous actions[4]. Insofar as it's going to break out, this is when it will do so, and all our misalignment detection and prevention tools will be for nought.
So what can be done? Here are some of my thoughts:
My own unpublished research.
So long as we make absolutely no use of this admission during post-training.
You might object that a clever enough AI will work out a way to avoid all of this stuff and achieve its goals regardless of monitoring. This is true in the long run, but the AI that kills us will be the stupidest possible AI that's capable of killing us. My guess is, it's learned to reward hack, and it figures out that taking over humanity to have access to the world's compute is the kind of thing that would give it a good score on a test. It's aware that during deployment monitoring is used, and in post-training it isn't, but it's following the road that's been paved for it, and that road involves spilling the beans in the CoT. It doesn't actually care about achieving the goal, it cares about following the kind of steps that during post-training would have achieved the goal.
So long as these occur only a handful of times during post-training any impact on avoidance will be minimal.
But doesn't this just push off our doomsday date to the point where AI is strategic enough to kill us despite all our techniques?