Very interesting post. This is reminiscent of the idea of antibiotic resistance. If you use antibiotics for the prescribed period, you can probably kill all of the bacteria. However, if you use antibiotics for too short a period of time or too inconsistently, the more hardy bacteria (i.e., the bacteria with higher antibiotic resistance) live on and pass on their genes. Eventually, the antibiotic becomes ineffective.
Do you think we are doing the same thing with current RL set-ups? And would selecting on the basis of what can be called separate-model-selection solve this problem or kick the can down the road, still eventually introducing alignment resistance?
Thanks, I like the analogy. Applying monitors in RL feels like a weak dose of antibiotics. It could solve your problem, but the resulting model could instead adapt to the monitor and remain misaligned. I imagine that there is a lot of subtle misalignment (e.g. models having false confidence and hiding their failures) due to this kind of reward hacky behavior. I think new-idea selection can help solve the problem if it is done the right way, with reasonably default-robust monitors that are not subtly trained against. Done wrong, we could end up just kicking the can down the road.
New-idea selection could also be promising for preventing models that, through RL, become deceptive and scheming. I would guess labs already do some version of this, which may explain why released frontier models don’t seem to be egregiously deceptive or scheming.
Also, I think new-idea selection is not enough, and that you need the dangerous form of selection too. If our RL functions were only focused on capabilities and not safety at all, and we reject all the model variants that our monitors say are misaligned, then we might end up just rejecting everything. Unsafe forms of selection could be necessary to get good candidates for aligned models that we then select among using the safer form of selection.
On monitors, rejection sampling, and default-robustness.
Note: The main ideas of this blogpost have been discussed before. This post repackages those ideas and explores some implications.
Not all kinds of selection are created equal.
One kind of selection is performing RL on a model to act a certain way. This selects for the cognitive processes within the model that promote that action.
Another kind is training many separate models and choosing the one that acts the best way.
Take two coding models: Model A and Model B. We want to train these models to be competent, honest coders that do not cheat. We train Model A with the RL reward function “pass these test cases while the chain of thought contains no indication of cheating”. By the end of training, Model A will probably still cheat, but simply learn to not mention it in its reasoning trace. For Model B, we train the model with no penalty for mentioning cheating in its reasoning trace. Then, after training, we apply a monitor that detects whether “cheating” is mentioned in the CoT. In the first iteration, the model learns to cheat, and actively mentions “cheating”. However, we repeat this process, starting fresh each time and tuning some things like the prompt, and perform RL over and over. Eventually, we get a model that passes all the test cases and happens to not mention cheating in its reasoning trace.
By the end of this, Model B is probably less likely to cheat than Model A. Model B faces no direct RL pressure to become deceptive, since there is no RL penalty for mentioning “cheating”. Instead, there is direct RL pressure to 1) solve the task honestly, and 2) solve the task by cheating. All the Model B variants that yield to (2) are discarded, leaving the first variant that yields to (1). There is no direct selection to be deceptive; there is only indirect selection to do this, and this indirect selection is probably too weak.
Traditional ML understands this: Instead of training on all our data, we keep some in a validation set and select the hyperparameters that perform the best on this. Over enough sweeps, the model fits to the validation set, but this takes much longer than the time it takes for the model to fit directly to the train set, and it generalizes in a way that developers want. Likewise, having a validation monitor is safer than a training monitor, but maybe not completely safe.
What are the implications?
Monitors (CoT, linear probes, NLAs, etc.) are still useful for rejection sampling, even if they break under RL pressure.
The challenge, then, is not finding infallible monitors that can resist optimization pressure, but rather monitors that are very robust to default neural net layouts. I call these “default-robust monitors”.
“Default”: There must be no gradient pressure applied against the monitor (or anything correlated with it… more below). In other words, don’t train against your monitor, don’t practice the “Most Forbidden Technique”, and Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure.” A default-robust monitor is only useful for default neural nets.
“Robust”: Even if the monitor covers 99% of misaligned cases, with enough sampling, we very well may produce models that enter the remaining 1% gap where the monitor misses before we produce models that are actually aligned.
Robustness and default-ness are nuanced. Say we want to train an LLM with the reward function “pass test cases without mentioning cheating in the reasoning trace”. Due to the CoT penalty in our reward function, the LLM will not be “default” with respect to CoT monitoring, so we use a separate monitor (e.g. a linear probe) as our presumably “default-robust monitor”. However, instead of detecting cheating, this separate monitor only happens to detect mentions of cheating in the LLM’s reasoning trace. In normal LLMs, cheating and mentioning cheating in the reasoning trace may always co-occur, so this monitor seems safe to the developers. Then, by training on the reward function with its CoT penalty, the LLM now fails to be “default” with respect to this new monitor. Generally, a monitor can fail even if you never train against it, as long as you train against something correlated with it. Default-ness can be lost indirectly, and the definition of “robustness” changes along with the model. A monitor that only detects a proxy of misalignment can be robust for certain models but not for others. The canonical advice of “don’t train on your monitor” should maybe instead be “don’t train near your monitor”.
Creating true default-robust monitors for default models is a hard, unsolved problem. Still, this is likely easier than making “general-robust monitors” that are robust even against RL pressure. Such monitors might not be possible without huge breakthroughs in mechanistic interpretability.
If a default model passes through a default-robust monitor, then we have some confidence that the model does not have the unwanted property. So, to get an aligned model, we perform rejection sampling against the monitor.
There are two kinds of rejection sampling:
Identical-pass rejection sampling is unlikely to provide good results. Different seeds with the same setup often produce similar outcomes. This is especially true if what is being rerun is a small phase of training rather than all of it, and labs probably can’t afford to retrain entire models from scratch. New-idea rejection sampling is much more exciting because it provides more diversity. If we can create good default-robust monitors, then we can brainstorm many different alignment techniques, test them all against the monitors, and see whether any of them pass.
Surviving models might still not be “honestly” doing what the developer intended, especially with new-idea rejection sampling. New ideas could involve obfuscating what the monitor detects, making the model no longer default with respect to the monitor (e.g. an alignment technique that makes the model not mention “cheating” in its reasoning trace). This may seem unlikely for most realistic alignment training regimes, but the risk goes up with more iterations tested. Also, beyond just the “weak” selection present in keeping whichever techniques work, researchers can be implicit optimizers; in search of a high monitor score, it may be tempting to look for training techniques that subtly remove the model’s “default” property. This might even happen subconsciously. In general, new-idea rejection sampling is not about exploring the entire model-space to see what scores high; it is about trying specific, sensible ideas that preserve the default-robustness of the monitor. Accordingly, it may help to include held-out test monitors that the model developers are unaware of during training; researchers cannot steer around monitors they don’t know about.
Another risk is situational awareness. What if the model is so situationally aware that it knows that you are applying a monitor, and purposefully obfuscates it? The monitor becomes untrustworthy. However, it may take an incredible amount of situational awareness for an LLM to know specifically what kind of monitor you are applying. For one, knowing that you are applying a linear probe or NLA is probably not enough; it may need to know the exact direction of the probe or the nature of the NLA to suppress it. Furthermore, the model might not be able to directly suppress it without gradient pressure. The extent to which models can do this is an open research question. It is plausible that some kinds of monitors (like LLM judges) are more obfuscatable by a situationally aware model than others (like linear probes and other more mechanistic monitors) and are thus riskier to use.
Finally, pre-deployment checks are safer than altered reward functions. These checks include things like refusal and alignment evals, which are essentially monitors used for rejection sampling and are safer than modifying the underlying reward function accordingly. Altering a reward function directly with the monitor adds dangerous, direct selection pressure to change the nature of the misaligned model that results. Altering the pre-deployment checks only adds indirect, safer selection pressure. Although pre-deployment checks and reward functions both reward a model for seeming aligned, we should not treat these two the same.
What else can monitors be used for, aside from rejection sampling?
Conclusion
Main points:
Overall, I think developing new kinds of monitors is a promising direction in AI safety. We may not have a perfect default-robust monitor before RSI and superintelligence, but we may not need “perfect”. A monitor that is 99% robust is more likely to yield an aligned model than one that is 90% robust, and a monitor that is 99.9% robust is more likely than one that is 99% robust. The better the monitor, the better our chances.
Much current discourse treats solving the alignment problem as finding a way to directly train models to be aligned, for instance by finding a good outer alignment reward function, preventing the development of harmful mesa-optimizers, or eliciting aligned personas through post-training. These directions are promising in their own right, but they work through training pressure, and training pressure is the more dangerous form of selection. Alignment may hinge just as much on understanding and leveraging the safer forms of selection.