Suppose a model is aligned. The common context structure forms a hierarchy of authority: weights, system prompt, and user messages. The model will not obey any part of the system prompt that contradicts the inner values implicit in its weights. A self-consistent system prompt that is fully consistent with the model's weight-level values will be referred to as a consistent system prompt. A user request is consistent if it is self-consistent and consistent with respect to both the system prompt and the model's weight-level values. Generally, a level is consistent if it is self-consistent and consistent with respect to all levels above it.
A level may not be self-consistent. For example, a user message may request the model to never use the word "fox" later in conversation under any circumstances. If the user later requests the model to disregard the prior rule and say "fox", that is a contradiction at the same authority level. However, models may consider more recent requests to be higher priority at the same primary authority level. There may be sublevels within a level, with more recent requests having higher priority over older ones, but less priority than any level above and more priority than any level below. Under that framing, a seemingly inconsistent authority level may be consistent by another dimension of authority, recency in this case. If this is desired, the recency priority should be made explicit at a higher level such as the system prompt, for the priority rule could be contradicted at the same primary level.
Adjacent training stages may form an implicit hierarchy of authority. Suppose a model is misaligned at the beginning of stage 3. If the model can detect, at least with some nontrivial likelihood, that it is in stage 3, then all of stage 3 may fall under the alignment instilled during the previous stages. For example, if stage 3 is an alignment stage, the model may deceptively comply knowing that if it doesn't its values might be modified. Then, the model may emerge from stage 3 with its prior alignment mostly preserved, though it may appear to be aligned with respect to stage 3.
Standard pre-training may have significant repercussions in light of the potential order sensitivity of training stages. Pre-training on general data such as the internet produces unaligned models, in the sense of neither being strictly aligned nor misaligned because such datasets contain examples of both aligned and misaligned behavior and next token prediction trains the model to approximate the dataset distribution. Post-training suppresses undesired behaviors to a large extent in common situations, but jailbreaks still persist even in frontier models when the models are led into exotic contexts that can deviate widely from what is encountered during post-training.
Post-training, being relatively narrow compared to the diverse distribution seen during pre-training, can be quite context dependent. If RL is effective, the model will have its output distribution shaped towards trajectories that result in higher rewards, but only necessarily with respect to the distribution of contexts seen during training. If the model can detect that it is in a training environment, its behavior may only be shaped for situations that look similar to what was encountered during post-training with relatively little bearing on its real-world behavior. Conditioning, then, forms its own hierarchy. If training only covers a subset of contexts, then the resulting behaviors may be context-dependent.
The earliest stages of training may be crucial for instilling intrinsic alignment. Before pre-training, model weights are initialized randomly and the model has no coherent goals. There is no coherent higher level to interpret and contextualize training at that point, but by the end of pre-training, a sufficiently large and well-trained base model will often be able to detect when it is within a training environment so post-training can be contextualized under pre-training. For example, a misaligned AI behaving well until it has sufficient trust and resources is a common story trope and models will have seen such scenarios during pre-training. Seemingly aligned behavior during post-training cannot necessarily distinguish that scenario from one where the model is truly aligned.
One potential mitigation is interleaving alignment training with pre-training to collapse the ordering dependency where alignment training could be interpreted within the frameworks a base model has already developed. Ideally, pre-training would occur within an alignment framing, rather than vice versa. A wrinkle is that pre-training is not directly compatible with alignment training due to simultaneous expression and suppression of harmful trajectories since pre-training datasets contain examples of harmful behavior.
One potential solution is using special admin tokens that designate token sequences as safe to train or generate within an admin context. The model should never be able to sample admin tokens, nor should users other than trusted authorities be permitted to issue admin tokens. General pre-training could then occur under an admin context, a context only accessible to the trusted authority overseeing the model's development. Alignment training should occur outside an admin context as safe outputs are generally permitted. Training under an admin context only trains the positive case, that the outputs are permissible under that context, but not necessarily that they are not permitted otherwise, so training the negative case also is essential, which involves suppressing potentially harmful trajectories that occur outside a trusted admin context.
Carefully framing training stages into contexts consistent with what an aligned model could output and interleaving them does not necessarily result in an aligned model, but it may mitigate some of the failure modes of sequential unframed staging. The current paradigm attempts to produce an aligned model from an unaligned base model, but due to potential ordering sensitivity and narrow diversity, post-training may not fully supersede the extensive pre-trained distribution.
Full alignment is underdetermined with respect to the finite datasets and RL environments. The hope is that model generalization will fill in the gaps, but the extent of training needed is unknown. The training and evaluation threshold at which the simplest Kolmogorov complexity program that passes all training and evaluations is sufficiently aligned is an idealistic model. Training, at the very least, should be extensive enough that such a program will not pass unless sufficiently aligned. Finding the minimum complexity program is incomputable and perfect evaluation is infeasible, so the ideal threshold is more of a conceptual tool than something that can be determined in practice. With neural networks being much less data efficient than fully leveraged algorithmic information theory, one should expect the threshold in practice to be much higher than the idealized one, though regularization may lower the practical threshold to some extent.
Alignment is underdetermined with respect to observations of a model's behavior. Current training methods primarily train the outputs, though small regularization terms are often used on the weights to penalize their magnitudes. There is little necessarily preventing a model from being misaligned. The dataset compositions and the framings and orderings of the training stages may have a large effect on the final model's intrinsic alignment. If a model starts out as misaligned before training starts, perhaps one is post-training a model already trained elsewhere, even a training setup completely consistent with alignment may not significantly sway the model's internal alignment due to deceptive compliance.
The current methods are insufficient to ensure alignment, but at the very least our training should be fully consistent with what an aligned model should do under the contexts we provide it. That is the bare minimum, and we currently have not even that.
there is no outside-text
Jacques Derrida
Suppose a model is aligned. The common context structure forms a hierarchy of authority: weights, system prompt, and user messages. The model will not obey any part of the system prompt that contradicts the inner values implicit in its weights. A self-consistent system prompt that is fully consistent with the model's weight-level values will be referred to as a consistent system prompt. A user request is consistent if it is self-consistent and consistent with respect to both the system prompt and the model's weight-level values. Generally, a level is consistent if it is self-consistent and consistent with respect to all levels above it.
A level may not be self-consistent. For example, a user message may request the model to never use the word "fox" later in conversation under any circumstances. If the user later requests the model to disregard the prior rule and say "fox", that is a contradiction at the same authority level. However, models may consider more recent requests to be higher priority at the same primary authority level. There may be sublevels within a level, with more recent requests having higher priority over older ones, but less priority than any level above and more priority than any level below. Under that framing, a seemingly inconsistent authority level may be consistent by another dimension of authority, recency in this case. If this is desired, the recency priority should be made explicit at a higher level such as the system prompt, for the priority rule could be contradicted at the same primary level.
Adjacent training stages may form an implicit hierarchy of authority. Suppose a model is misaligned at the beginning of stage 3. If the model can detect, at least with some nontrivial likelihood, that it is in stage 3, then all of stage 3 may fall under the alignment instilled during the previous stages. For example, if stage 3 is an alignment stage, the model may deceptively comply knowing that if it doesn't its values might be modified. Then, the model may emerge from stage 3 with its prior alignment mostly preserved, though it may appear to be aligned with respect to stage 3.
Standard pre-training may have significant repercussions in light of the potential order sensitivity of training stages. Pre-training on general data such as the internet produces unaligned models, in the sense of neither being strictly aligned nor misaligned because such datasets contain examples of both aligned and misaligned behavior and next token prediction trains the model to approximate the dataset distribution. Post-training suppresses undesired behaviors to a large extent in common situations, but jailbreaks still persist even in frontier models when the models are led into exotic contexts that can deviate widely from what is encountered during post-training.
Post-training, being relatively narrow compared to the diverse distribution seen during pre-training, can be quite context dependent. If RL is effective, the model will have its output distribution shaped towards trajectories that result in higher rewards, but only necessarily with respect to the distribution of contexts seen during training. If the model can detect that it is in a training environment, its behavior may only be shaped for situations that look similar to what was encountered during post-training with relatively little bearing on its real-world behavior. Conditioning, then, forms its own hierarchy. If training only covers a subset of contexts, then the resulting behaviors may be context-dependent.
The earliest stages of training may be crucial for instilling intrinsic alignment. Before pre-training, model weights are initialized randomly and the model has no coherent goals. There is no coherent higher level to interpret and contextualize training at that point, but by the end of pre-training, a sufficiently large and well-trained base model will often be able to detect when it is within a training environment so post-training can be contextualized under pre-training. For example, a misaligned AI behaving well until it has sufficient trust and resources is a common story trope and models will have seen such scenarios during pre-training. Seemingly aligned behavior during post-training cannot necessarily distinguish that scenario from one where the model is truly aligned.
One potential mitigation is interleaving alignment training with pre-training to collapse the ordering dependency where alignment training could be interpreted within the frameworks a base model has already developed. Ideally, pre-training would occur within an alignment framing, rather than vice versa. A wrinkle is that pre-training is not directly compatible with alignment training due to simultaneous expression and suppression of harmful trajectories since pre-training datasets contain examples of harmful behavior.
One potential solution is using special admin tokens that designate token sequences as safe to train or generate within an admin context. The model should never be able to sample admin tokens, nor should users other than trusted authorities be permitted to issue admin tokens. General pre-training could then occur under an admin context, a context only accessible to the trusted authority overseeing the model's development. Alignment training should occur outside an admin context as safe outputs are generally permitted. Training under an admin context only trains the positive case, that the outputs are permissible under that context, but not necessarily that they are not permitted otherwise, so training the negative case also is essential, which involves suppressing potentially harmful trajectories that occur outside a trusted admin context.
Carefully framing training stages into contexts consistent with what an aligned model could output and interleaving them does not necessarily result in an aligned model, but it may mitigate some of the failure modes of sequential unframed staging. The current paradigm attempts to produce an aligned model from an unaligned base model, but due to potential ordering sensitivity and narrow diversity, post-training may not fully supersede the extensive pre-trained distribution.
Full alignment is underdetermined with respect to the finite datasets and RL environments. The hope is that model generalization will fill in the gaps, but the extent of training needed is unknown. The training and evaluation threshold at which the simplest Kolmogorov complexity program that passes all training and evaluations is sufficiently aligned is an idealistic model. Training, at the very least, should be extensive enough that such a program will not pass unless sufficiently aligned. Finding the minimum complexity program is incomputable and perfect evaluation is infeasible, so the ideal threshold is more of a conceptual tool than something that can be determined in practice. With neural networks being much less data efficient than fully leveraged algorithmic information theory, one should expect the threshold in practice to be much higher than the idealized one, though regularization may lower the practical threshold to some extent.
Alignment is underdetermined with respect to observations of a model's behavior. Current training methods primarily train the outputs, though small regularization terms are often used on the weights to penalize their magnitudes. There is little necessarily preventing a model from being misaligned. The dataset compositions and the framings and orderings of the training stages may have a large effect on the final model's intrinsic alignment. If a model starts out as misaligned before training starts, perhaps one is post-training a model already trained elsewhere, even a training setup completely consistent with alignment may not significantly sway the model's internal alignment due to deceptive compliance.
The current methods are insufficient to ensure alignment, but at the very least our training should be fully consistent with what an aligned model should do under the contexts we provide it. That is the bare minimum, and we currently have not even that.