I discuss some downsides of train-deploy mismatch, especially downsides of innoculation prompting in this comment. (I haven't yet engaged with this post in much detail.)
Interesting framing. If I understand correctly, you're defining train-deploy mismatch as something like "if the deployed model and learner were run on the same data, would their outputs differ"? And changing the data distribution the model is trained on as not train-deploy mismatch?
Good question. I originally defined train-deploy mismatch the way you suggest. On that definition, inoculation prompting is interpreted as changing the model architecture (by grafting the inoculation prompt onto it) during training. However, Monte convinced me that inoculation prompting is better understood as changing the data distribution, so I edited the post to be less committal and include that as a possibility too, even though I still think primarily in terms of the original definition.
P_learner(desired behavior) → 𝜺 > 0 and that we have a way to condition the model to act in the desired way.
I'm very interested in this, particularly in the varying of training budgets, and whether we can get some scaling type empirical control of P_model(desired behaviour) and relatedly whether this affects conditioning.
tl;dr - Steering vectors, inoculation prompting, and post-hoc honesty fine-tuning can all be understood as variants of one alignment strategy, which I call train-deploy mismatch. Each trains the model in one configuration and deploys it in another. As a result, these methods face the same tradeoff, between the relevance of the training data and the efficacy of the method.
Note: Others have had similar ideas and shaped my thinking here including Sam Marks, Ariana Azarbal, Victor Gillioz, Alex Turner, Jacob Goldman-Wetzler, Jake Mendel, Daniel Tan, and Fabien Roger. Thanks to Monte MacDiarmid, Nat McAleese, Shawn Hu, and Jake Ward for input on an earlier draft.
Background
AI alignment is hard largely because we don't know how to specify what we want. Instead, we train models on proxies for what we want: labels and reward functions defined on data distributions chosen such that we hope the model will perform as desired when deployed into the world. This approach has worked well so far, but given increasing model capabilities, it may stop working— models may misgeneralize their training to catastrophically bad behavior in deployment. A pressing open problem is to figure out how to get models to generalize the properties that we want from their training.
Although it might not be obvious at first, the following alignment techniques all attempt to solve this problem, and they do it using the same strategy.
Can you spot what these methods have in common?
Train-deploy mismatch as a general alignment strategy
Each of the methods listed above can be understood as training a model in one mode, then deploying it in another. For example, with steering vectors, the model is trained without a steering vector, then deployed with the steering vector added in. Preventative steering is the reverse: apply a steering vector in training, then remove it in deployment.
Some cases require more interpretive work. In the case of “training normally, then fine-tuning on honesty data,” we can understand “fine-tuning on honesty data” as an operator that can be applied to any model. We can imagine applying this operator at any point during the normal training phase of the model. This creates two “modes” for the model: the training mode (pre-honesty training) and the deployment mode (post-honesty training). From this perspective, there is a mismatch: over the course of training, the model is trained to become more capable at solving tasks while in a “less honest” state. Then in deployment, we switch it over to a “more honest” state and hope that the model’s all-things-considered performance is better (namely, that it can still solve tasks, but is more honest in doing so).
We say these cases are instances of train-deploy mismatch, because each method deliberately introduces a difference between how the model is trained and how it is deployed.
Train-deploy mismatch is a somewhat odd strategy. Normally in machine learning, we try to avoid distribution shift. In the case of train-deploy mismatch, we’ve introduced a kind of distribution shift intentionally. Why do this? Intuitively, the reason is that train-deploy mismatch makes it so that the learner, no matter how smart or specialized to its training objective, has not “had a chance” to subvert our limited ability to evaluate its behavior in deployment mode. For example, a careful schemer that is perfectly calibrated about when to stage a coup may become uncalibrated (or unscheming) simply by changing its system prompt or by being steered.
More formally, we could conceptualize train-deploy mismatch as an approach to preventing overoptimization when training with weak supervision.
Fundamental tradeoffs
Train-deploy mismatch faces tradeoffs related to off-policyness. These tradeoffs have to do with the differences between the model that generates the training data (the sampler), the model used to calculate gradients to update the model (the learner), and the model that would actually be deployed (the target).
The tradeoffs are fundamental in the sense that, if we want 3, we have to give up some of 1 or 2.
Another way to frame this tension is in terms of choosing where to apply weak supervision. If you have a limited ability to grade model behavior (that risks being overoptimized), do you:
Note that the existence of these tradeoffs does not imply that the best option is to give up and set sampler=learner=target. Perhaps we can do better than this!
Figure 1 shows the difference between conventional on-policy training and mismatch.
Figure 1. A schematic comparing regular on-policy learning to train-deploy mismatch, which uses the same neural network with different configurations to (a) sample to produce training data, (b) learn on that data, and (c) produce a target model to deploy into the world. On-policy learning is a special case where sampler=learner=target. This mismatch could enable selective generalization, where only desired properties of the learner generalize to the target.
Figure 2. A schematic showing the sampler, learner, and target over time. The dashed arrows marked “-” indicate settings we’d ideally like to be as similar as possible; the dashed arrow marked “+” indicates settings we’ve intentionally made dissimilar, with the hope of gaining some benefit. The “Implicit deployment” is the model we’d release if we had to stop training partway– this is the model we actually care about, so we call it the “target" of our learning algorithm. The point of this post is that it can be illuminating to consider the evolution of the target as compared with the sampler and learner.
Sidebar: is everything train-deploy mismatch?
No. Here are some alignment techniques that aren’t train-deploy mismatch:
Here, in the context of LLMs, we are only considering the post-training phase.
Open threads
Here are some raw thoughts that I would expand on, given more time.
Selective generalization motivates train-deploy mismatch. We want the learner to generalize capabilities but not bad propensities to the target. To get this right, we may need to “build a bridge” (from training to deployment) by applying some kind of updates to elicit the desired capability in the deployment configuration. In other words, suppose our sampler and learner are working together to become smarter (yay) and also more misaligned (boo), while also gaining an improved (but latent) capacity for aligned behavior. How do we elicit the part that we want to keep in deployment without eliciting the other?
This question could be cast in terms of easy-to-hard generalization. Suppose you have some tasks that you can supervise well (“easy” tasks), and some that you care about but can’t supervise well (“hard” tasks). You train with RL on the hard tasks but your evaluations have systematic errors, causing the model to become misaligned as you train. Is there a way you can use the easy tasks to align the model on hard tasks?
Train-deploy mismatch buys something nontrivial. The implicit goal of a learning algorithm designer is to find an algorithm that produces a desired output with probability close to 1. In shorthand, we want P_model(desired behavior) → 1 as the amount of training increases. (In Statistics, this is called consistency.) In regular alignment training (when sampler=learner=target), this means we need the behavior of the sampler to approach 100% alignment given sufficient training. This is a big ask! Train-deploy mismatch changes this. In principle, we just need that P_learner(desired behavior) → 𝜺 > 0 and that we have a way to condition the model to act in the desired way. In other words, we merely need the model to act in an aligned way some nontrivial portion of the time (for any given input), and to be able to select for that behavior. This is potentially a huge advantage relative to needing the model to be aligned 100% of the time.
A way out. So far the discussion has focused on bad behavior that occurs in training. As argued above, train-deploy mismatch faces fundamental challenges here. Suppose this challenge turns out to be insurmountable– does this mean that train-deploy mismatch is useless as an alignment strategy? Maybe not.
One escape from the fundamental tradeoffs would be to consider a different problem altogether. Rather than trying to patch bad behavior in training, we could target scenarios where a model behaves in an aligned way during training, but misgeneralizes to bad behavior in deployment (e.g. a catastrophically misaligned model that fakes alignment). In this case, the sampler, learner, and desired target are all close on the training distribution, which dissolves the tension.
Concretely, we might look for interventions (e.g. a collection of distinct steering vectors) that have no effect on the model’s behavior on the training distribution, but that do change the model’s internal computation somehow– in a way that we think will change how the model acts in deployment. Then we could select from these in deployment. This is not a new idea. See, for example, Diversify and Disambiguate.
Closing thoughts
I'm not sure if train-deploy mismatch will helpful for aligning superintelligence, but it seems worth considering.
At a minimum, I hope this post will inspire alignment researchers to think about whether their ideas are instances of this strategy, and if so, to consider the fundamental tradeoffs.