Tl;dr: Alignment techniques can be used for training alignment or for testing it, but they cannot be used for both! As we build out the set of alignment techniques available to us, we should enforce that some are judiciously reserved only for evaluating alignment.
Here is a guiding principle regarding alignment-minded AI training: whatever you do, you must maximally avoid teaching the model to take misaligned actions which are invisible to the user. In fact, you should be shamed if you use training techniques which encourage covert misalignment.
Here is a separate, guiding principle of machine learning: whatever you do, you must maximally avoid training the model on the entire dataset rather than holding out a test split. In fact, you should be shamed if you report results which are obtained by training on the entire dataset rather than on the train split.
I want to argue that these are the same principle. Training against all of your alignment assessment techniques leads to the same failure as training on your validation data in traditional machine learning: you potentially get better performance, but you sacrifice all ability to verify your model’s behavior.
Consider two types of alignment techniques:
Some techniques sculpt the model’s alignment. This includes techniques such as:
Some techniques assessthe model’s alignment (aka, monitoring). This includes techniques such as:
Human analysis of model outputs
CoT monitoring
Activation monitoring
Broader interp tools like SAEs, transcoders, etc.
Evals for whether models can control their activations
Pre-fill analysis
Honeypots
Universal jailbreaks (to study what the model will do at the limit)
I'd like you to notice that all these methods for sculpting a model's alignment can be repurposed for assessing its alignment, and vice versa. To give a few illustrative examples:
Chain of thought:
We can train models to not verbalize scheming by training against CoT monitors
We can use CoT monitors to tell whether models are scheming
Human feedback:
We can RLHF models to behave according to human preferences
We can hold out these human analyses to tell whether the model is behaving in ways people like.
Probes
We can train against deception probes to control the model’s actions
We can deploy models with constitutional classifiers
Honeypots
We can train models to avoid honeypots
We can hold out honeypots to see whether the model takes them
Automated audits
We can use automated auditing agents (petri) to find behaviors to RL away
We can keep automated audits as a way to evaluate alignment
The same can be said for gradient routing, RLAIF, data filtering, etc.
As soon as you train against the monitor (CoT, human feedback, probes, honeypots), you necessarily sacrifice this monitor's utility. The model now never admits misalignment which this monitor can catch, since many of the cases which were caught were trained away!
This holds trivially for CoT and honeypots, but it also applies to all the other alignment techniques. For instance, RLHF relies on human ‘monitors’ to catch bad responses so you can train them away. Filtering pretraining data relies on ‘monitors’ to identify samples that might cause bad generalization and then taking them out of the data mix.
At the limit—where you use all of the monitors for training and leave none for the test-set—you end up at extreme ‘you get what you measure’ problems: your model will necessarily appear aligned… because you’ve trained against everything which you might use to evaluate alignment!
The best (and only) example I know of where we’ve truly tried to hold out an alignment technique is chain of thought monitorability. However, CoT monitorability is dying and it would be exceedingly unwise to end up with no alignment techniques in our test set.
Importantly, the dataset of alignment techniques and the dataset of alignment environments is truly independent: we can train/test all of our alignment techniques using just a single alignment environment.
Consider a single standard reward-hacking honeypot. Within this one setting, one can train the model’s alignment by: training out verbalized scheming in the CoT, updating the model’s behavior based on human feedback, training the model’s behavior against a probe, RL-ing the model to avoid the honeypot, and training against automated audits seeded on this environment.
As a result, we essentially have two independent alignment datasets. We have some set of alignment samples: these are essentially the standard notion of data samples from machine learning. But we also have a set of alignment techniques: these are the way in which we update the model’s alignment using any one sample.
My point is this: models generalize across both alignment techniques and alignment samples. We hold out alignment samples all the time, but have significantly worse hygiene about train/test splits for alignment techniques.[1]
To operationalize the above ideas, consider this recent paper out of the Anthropic MATS stream.
In it, they propose a gradient routing mechanism—GRAFT—for removing misaligned behavior from a model. Their technique works by taking monitored (somewhat toy) RL envs and checking for reward hacks programmatically. If the hack occurred, you route the gradient update to a “Forget” adapter; otherwise, the update is routed to both the “Forget” adapter and a separate “Retain” adapter. With some optimization tricks, they show that you can then remove the Forget adapter during inference and (a) maintain good performance on the task while (b) observing less reward hacking.
Before going into my diatribe, I will emphasize that I like the idea of gradient routing and that I think it is good to explore this further as an alignment technique. I think it is better to have it than to not have it at all.
But my point is that it could work very well as an alignment testing technique. That is, instead of removing the adapter responsible for reward hacking, one can simply keep this adapter and, during inference, see whether the model routes computations through it. This would allow us to catch the model misbehaving.
However, this seems to have not been explored! If indeed GRAFT generalizes well to unseen reward hacks and this monitor absorbs all the ‘reward hackiness’ in the model, we should consider relegating it to exclusively being an alignment testing technique.
And, in either case, my more principled objection is that this does not solve the fundamental problem of alignment generalization. Regardless of whether you train against the monitor or you route gradients based on a monitor, you nonetheless have to rely on what the monitor can catch! And by removing the misalignments that this monitor observes, you render that monitor obsolete (e.g., you shrink your test set).
A natural objection is: “if we have methods for assuring alignment, why shouldn’t we train models using all of them? Won’t this make our models more aligned than if we didn’t?”
On the one hand, yes – training on your entire dataset will give you a better model than training on some subset of that data. But training on the entire dataset also renders you unable to determine if your resulting model is fundamentally flawed.
By not having a test-set, you are relegated to identifying alignment via trial-and-error during deployment. However, in the AGI regime, we cannot rely on alignment by trial and error. Indeed, AI alignment is the scientific discipline where we must respect the train/test split the most. ASI alignment has the property where our ‘ability to understand the model’ is directly commensurate with the extinction of the human race.
What does it mean to be better about this?
First, we must require transparency from frontier AI companies about which, if any, alignment techniques are relegated for testing. For instance, I’d love to be able to answer the following simple question: “what kinds of alignment RL do the frontier AI companies use”? Are they just training against every misalignment they can identify, in every way they possibly can? Or is there at least one clean, truly-held-out set of alignment environments and techniques which the model has never seen, where we can actually evaluate the model’s behavior? This won’t solve everything, but it would be way better than what we currently have.
Second, we should have a culture where we try to determine whether an alignment technique is better suited towards alignment training or alignment testing. For instance, RLHF seems quite good at alignment sculpting while CoT monitoring seems inherently alignment-assurance flavored. Some are less clear – for instance, it seems like people are training against activation monitors, but it also seems inherently risky to do so. In cases where it’s unclear, we should prioritize studying whether this is a train-set or a test-set idea.
Third, we should maximize our set of alignment techniques and prioritize that a large portion be sequestered for test purposes. This is why adopting looped transformer architectures is such a profoundly terrible decision – not only does it remove a valuable alignment technique, it got rid of one which was assurance-shaped!
Thanks to Sam Martin, Dewi Gould, Tim Hua, Jonathan Bostock, and Moritz Pfau for comments. Thanks also to Raymond Douglas and, in particular, Daniel Tan for lightly pressuring me to finally post this.
Tl;dr: Alignment techniques can be used for training alignment or for testing it, but they cannot be used for both! As we build out the set of alignment techniques available to us, we should enforce that some are judiciously reserved only for evaluating alignment.
See also: Validating against a misalignment detector is very different to training against one.
Here is a guiding principle regarding alignment-minded AI training: whatever you do, you must maximally avoid teaching the model to take misaligned actions which are invisible to the user. In fact, you should be shamed if you use training techniques which encourage covert misalignment.
Here is a separate, guiding principle of machine learning: whatever you do, you must maximally avoid training the model on the entire dataset rather than holding out a test split. In fact, you should be shamed if you report results which are obtained by training on the entire dataset rather than on the train split.
I want to argue that these are the same principle. Training against all of your alignment assessment techniques leads to the same failure as training on your validation data in traditional machine learning: you potentially get better performance, but you sacrifice all ability to verify your model’s behavior.
Consider two types of alignment techniques:
I'd like you to notice that all these methods for sculpting a model's alignment can be repurposed for assessing its alignment, and vice versa. To give a few illustrative examples:
As soon as you train against the monitor (CoT, human feedback, probes, honeypots), you necessarily sacrifice this monitor's utility. The model now never admits misalignment which this monitor can catch, since many of the cases which were caught were trained away!
This holds trivially for CoT and honeypots, but it also applies to all the other alignment techniques. For instance, RLHF relies on human ‘monitors’ to catch bad responses so you can train them away. Filtering pretraining data relies on ‘monitors’ to identify samples that might cause bad generalization and then taking them out of the data mix.
At the limit—where you use all of the monitors for training and leave none for the test-set—you end up at extreme ‘you get what you measure’ problems: your model will necessarily appear aligned… because you’ve trained against everything which you might use to evaluate alignment!
The best (and only) example I know of where we’ve truly tried to hold out an alignment technique is chain of thought monitorability. However, CoT monitorability is dying and it would be exceedingly unwise to end up with no alignment techniques in our test set.
Importantly, the dataset of alignment techniques and the dataset of alignment environments is truly independent: we can train/test all of our alignment techniques using just a single alignment environment.
Consider a single standard reward-hacking honeypot. Within this one setting, one can train the model’s alignment by: training out verbalized scheming in the CoT, updating the model’s behavior based on human feedback, training the model’s behavior against a probe, RL-ing the model to avoid the honeypot, and training against automated audits seeded on this environment.
As a result, we essentially have two independent alignment datasets. We have some set of alignment samples: these are essentially the standard notion of data samples from machine learning. But we also have a set of alignment techniques: these are the way in which we update the model’s alignment using any one sample.
My point is this: models generalize across both alignment techniques and alignment samples. We hold out alignment samples all the time, but have significantly worse hygiene about train/test splits for alignment techniques.[1]
To operationalize the above ideas, consider this recent paper out of the Anthropic MATS stream.
In it, they propose a gradient routing mechanism—GRAFT—for removing misaligned behavior from a model. Their technique works by taking monitored (somewhat toy) RL envs and checking for reward hacks programmatically. If the hack occurred, you route the gradient update to a “Forget” adapter; otherwise, the update is routed to both the “Forget” adapter and a separate “Retain” adapter. With some optimization tricks, they show that you can then remove the Forget adapter during inference and (a) maintain good performance on the task while (b) observing less reward hacking.
Before going into my diatribe, I will emphasize that I like the idea of gradient routing and that I think it is good to explore this further as an alignment technique. I think it is better to have it than to not have it at all.
But my point is that it could work very well as an alignment testing technique. That is, instead of removing the adapter responsible for reward hacking, one can simply keep this adapter and, during inference, see whether the model routes computations through it. This would allow us to catch the model misbehaving.
However, this seems to have not been explored! If indeed GRAFT generalizes well to unseen reward hacks and this monitor absorbs all the ‘reward hackiness’ in the model, we should consider relegating it to exclusively being an alignment testing technique.
And, in either case, my more principled objection is that this does not solve the fundamental problem of alignment generalization. Regardless of whether you train against the monitor or you route gradients based on a monitor, you nonetheless have to rely on what the monitor can catch! And by removing the misalignments that this monitor observes, you render that monitor obsolete (e.g., you shrink your test set).
A natural objection is: “if we have methods for assuring alignment, why shouldn’t we train models using all of them? Won’t this make our models more aligned than if we didn’t?”
On the one hand, yes – training on your entire dataset will give you a better model than training on some subset of that data. But training on the entire dataset also renders you unable to determine if your resulting model is fundamentally flawed.
By not having a test-set, you are relegated to identifying alignment via trial-and-error during deployment. However, in the AGI regime, we cannot rely on alignment by trial and error. Indeed, AI alignment is the scientific discipline where we must respect the train/test split the most. ASI alignment has the property where our ‘ability to understand the model’ is directly commensurate with the extinction of the human race.
What does it mean to be better about this?
First, we must require transparency from frontier AI companies about which, if any, alignment techniques are relegated for testing. For instance, I’d love to be able to answer the following simple question: “what kinds of alignment RL do the frontier AI companies use”? Are they just training against every misalignment they can identify, in every way they possibly can? Or is there at least one clean, truly-held-out set of alignment environments and techniques which the model has never seen, where we can actually evaluate the model’s behavior? This won’t solve everything, but it would be way better than what we currently have.
Second, we should have a culture where we try to determine whether an alignment technique is better suited towards alignment training or alignment testing. For instance, RLHF seems quite good at alignment sculpting while CoT monitoring seems inherently alignment-assurance flavored. Some are less clear – for instance, it seems like people are training against activation monitors, but it also seems inherently risky to do so. In cases where it’s unclear, we should prioritize studying whether this is a train-set or a test-set idea.
Third, we should maximize our set of alignment techniques and prioritize that a large portion be sequestered for test purposes. This is why adopting looped transformer architectures is such a profoundly terrible decision – not only does it remove a valuable alignment technique, it got rid of one which was assurance-shaped!
Thanks to Sam Martin, Dewi Gould, Tim Hua, Jonathan Bostock, and Moritz Pfau for comments. Thanks also to Raymond Douglas and, in particular, Daniel Tan for lightly pressuring me to finally post this.
Perhaps I'm belaboring the point here, but I really feel this strongly!