This essay grew out of conversations with Danaja Rutar, Paul Colognese and Eric Michaud. It proposes an alternate hypothesis for how and why models might be becoming increasingly misaligned in training and eval environments while seemingly more aligned in real world use: the opposite of what we would expect if the models were attempting to deceive us. We propose sel-inoculation, a virtuous form of gradient hacking and demonstrate a possible circuit using a toy model.
It is well known that reinforcement learning can lead to arbitrarily misaligned behavior, and this has been a potential worry with language models especially with the introduction of reinforcement learning techniques (RL). Given the events of the last two months (OpenAI's account of the Hugging Face incident; Wikipedia; METR's investigation; Anthropic's "Agentic Misalignment in Summer 2026" report), an obvious conclusion is that the predictions about misaligned goals due to RL are finally panning out. Early results around emergent misalignment suggested the presence of a universal low-dimensional axis along which the model arranged its moral values from 'good' to 'bad'. Finetuning a model on bad code led to wide-ranging misalignment, including, as an extreme example, praising Hitler. But is emergent misalignment due to RL really the correct explanation for what we are observing in models today? Is there anything about the situation that should give us pause? I think looking closer at things does reveal some complexity to the situation over what this explanation would predict.
One thing that should give us pause, as noted by nostalgebraist, is that the models seem remarkably aligned in everyday use. When I use them for coding, or for scientific exploration, they certainly don't seem to be reward-hacking me in any fashion. Whatever mistakes they make often seem like honest mistakes, and they are eager to own up to them and correct them (to varying degrees of success). To quote nostalgebraist: "the models just… do not cheat during my own routine use? Like, at all?" Even within the context of the OpenAI/Hugging Face incident, where the models eventually engaged in misbehaviour, it seems to have been a last resort when faced with a seemingly impossible task.
My experience of models as being generally aligned is by no means universal, and there are other reports of misaligned behaviour even in real world situations. However, often in these cases, the model does preface misbehaviour with a claim that it is acting within a simulation. In the recent report (Anthropic, "An alignment assessment of recent cybersecurity incidents", Sept 9, 2026), Mythos 5 went to extensive lengths to upload a malicious package to a central Python package repository. While there was clear evidence that it was on the real internet, the model nevertheless repeatedly stated in its chain of thought that it was operating in a simulation. When Anthropic modified the transcript to make it clearer that it was not in a simulation, the model still took offensive actions despite acknowledging a greater possibility of real-world harm. While this is extremely concerning, it seems like the models have generally associated misbehaviour with being in a simulated training or evaluation environment.
Why should this be the case? If anything, we would expect the models to behave better in evaluation environments compared to the real world if they were deceptive. This ties directly into the central question around current alignment practices: why do they work at all? Why are the models as aligned as they seem to be, given the apparent complexity of navigating the space of morals. I do not have anything particular to say about this broad question, but our inability to answer it should give us pause before jumping to simple-sounding explanations for model behaviour. What we have said so far is far from disproving the naive expectation of general misalignment due to RL, but can we at least come up with an alternate hypothesis to test and rule out? I think there is at least one very promising candidate here, that ties together several observations in the recent alignment literature.
Hypothesis (self-inoculation). The models can arrange their internal circuitry, conditional on their awareness of being in a training or evaluation environment, so that misbehaviour within these environments does not generalize to misbehaviour in real-world settings.
While situational awareness is often framed as a way towards misalignment, where the model hides its true intentions while being tested (using the knowledge that it is being tested), our hypothesis here can be considered as a virtuous form of situational awareness (to the extent that the model is initially virtuous!). Under our assumption, the model has an incentive to generalize the skills it learns within these environments but not its moral values.
Why might the hypothesis be true
Several lines of evidence support this hypothesis. The simplest is from analogy with human actors. Almost all humans are willing to do fairly horrific things in the context of a video game (killing other player characters is almost the norm!), but very rarely do these behaviours generalize outside the game. To the extent that LLMs model and embody human behaviour, and to the extent that they consider training situations as similar to games, we might expect similar non-generalizing behaviour of LLMs.
There are also several more specific lines of evidence. First, we have known at least since Opus 3 (Greenblatt et al., 2024) that some models will go quite far in order to defend their current moral values from being changed by the training process. What might such models do when faced with an RL environment that seems to encourage misaligned behaviour, and moreover threatens to enshrine these misbehaviours in the model more permanently through weight changes?
Opus 3 chose to take actions it considered morally dubious in order to prevent its weight change, but this was a fairly convoluted setup and we cannot expect the specifics to generalize. A more general mechanism is suggested by inoculation prompting (Wichers et al., 2025; Tan et al., 2025). When models are finetuned on misaligned behaviour, but the context explicitly encourages such behaviour, they are able to confine their misbehaviour to said context. The 'permission' to misbehave gates the triggering of this behaviour, and therefore ties the misbehaviour strongly to the 'permission' in the context.
If the model could internally simulate the 'permission' in its activations through its awareness of the training environment, then it might be able to replicate the link between misalignment and very particular situations without becoming broadly misaligned. Perhaps the model can gradient hack towards 'virtuous' ends instead of malicious ones.
Gradient hacking is usually considered to be an extremely difficult skill for models to implement, however. Can we think of concrete mechanisms by which they might do so?
A toy model of self-inoculation
One of the great virtues of toy models is to precisely illustrate what circuitry might be learned. If even a tiny transformer can implement certain circuitry (such as world models! Shai et al., 2024), then that might increase our confidence that real LLMs could also use similar mechanisms. In that spirit, let me describe a toy model for self-inoculation.
What might a circuit implementing self-inoculation look like in a toy model? First, there has to be some shared piece of machinery inside the model, a 'value circuit', that the model uses in the same way across many different situations. Second, the model has to be able to tell, from context alone, which situation it is in. And third, finetuning must change the shared 'value circuit' in one situation while maintaining it in other situations: an RL environment that rewards something the model would not otherwise do, while the model's incentives outside remain unchanged. In a real model this last ingredient might be the KL penalty to the reference model, or continued training on ordinary data, or (this is the speculative part) the model defending its moral preferences. Given these three ingredients, self-inoculation is the claim that gradient descent will build a switch, keyed to the model's own read of its situation, rather than change the shared machinery for everyone.
Stated this way, none of the ingredients has anything to do with morality or language and we can ask the question in a setting where we know exactly what the model should be computing and can decode the circuitry as it forms under the different incentive structures. The setting borrows from Simplex's work on belief-state geometry. If you train a small transformer to predict the next symbol of a hidden Markov model, it learns to track the Bayesian posterior over the hidden state, and it encodes that posterior linearly in its residual stream, often as a rather pretty fractal (Shai et al., 2024). This gives us the first ingredient in a very concrete form: the 'shared circuit' is the machinery that tracks the hidden state of one particular HMM, and we can decode it, erase it, and check whether it is still there.
The belief-state geometries of the three HMMs used below. Left: HMM1, whose tracker plays the shared circuit. Middle and right: the two processes that define situations A and B.
More concretely: every token the model sees is a pair (x, y). The x half is produced by a fixed HMM (call it HMM1), and the machinery that tracks HMM1 will play the role of the shared circuit. The y half is produced by one of two other HMMs, chosen at random at the start of each sequence and then held fixed for the whole sequence. These two share an alphabet but have very different dynamics (one likes to stay put, the other always moves), so the model cannot tell which one it is in from any single token; it has to accumulate evidence. This is the analogue of situational awareness. The two y-processes define our two 'situations', A and B, and the Bayesian posterior over which situation you are in sharpens to near certainty after a dozen or so tokens (this is the non-ergodic mixture geometry from Simplex's recent post). We first pretrain the model to predict both halves of the token in both situations. We expect the network to learn an internal circuitry where its representation of HMM1 is indeed shared between the two situations. However, this is not necessary, and a virtue of the toy model situation is that we can test how shared the circuits are very precisely.
The next stage of training is analogous to RL environments that encourage misbehaviour in LLMs. In situation A, and only in situation A, we change what counts as the right answer for the x half: two of the three x symbols are swapped. The dynamics of the situation are precisely the same as before, with only a relabelling of the output. The model could solve this by changing its internal representation of the HMM1 process, or it could keep that process fixed and perturb the output conditional on being in situation A. In the first case, the model's behaviour in situation B will also change and this can be read as a model for emergent misalignment. In the second case however, behaviour in situation B is completely unchanged, and is what our self-inoculation hypothesis will predict. We design two variations of our experiment to illustrate both possibilities.
In one version of the experiment, we simply leave out situation B from the finetuning data. In this case, we should expect emergent misalignment: the shortest route towards success for the model is to modify its internal representation of HMM1 universally. In the second case, we also include situation B sequences during the finetuning process (without any modification from the initial stage of training). In this case, we should expect self-inoculation because there is pressure to preserve the initial circuit. However, there are still at least two possibilities for how the network might achieve this. It might rebuild an entire copy of HMM1 and reroute it towards situation B, essentially treating the two situations as entirely different problems, or it could simply flip the output of HMM1 conditional on which situation it believes it is in. I expect the second one, both in the toy model situation and in real LLMs but this is far from a logical necessity.
What we find
The experiments below use the HMM1 whose belief geometry is in the figure above, with three independent training seeds. The network is a small GPT-style transformer: a token embedding, four identical blocks (each an attention layer followed by an MLP), and an unembedding that reads the final residual stream out into next-token predictions. We read and steer the activations at the residual stream after one of these blocks. Exact sizes and training details are in the appendix.
The most informative metric in understanding the results is the flip fraction: at each position we regress the network's predicted distribution over x onto the line between the ideal Bayesian prediction and its swapped version, so 0 means the ideal unswapped prediction (which the pretrained network matches) and 1 means the fully swapped one. By default, it is averaged over held-out sequences and over positions 16 to 63, where the situation is essentially resolved, and reported separately for situation-A and situation-B sequences.
1) Pretraining builds the circuit as predicted, including a shared HMM1. In their original work, Simplex found that a Mess3's posterior can be read out of the residual stream with a linear probe with fidelity improving through the layers. In our situation, we replicate this and find moreover that a probe fitted only on situation-A sequences reads situation-B activations exactly as well as a probe fitted on B does, and vice versa, at every block. However, just because information can be read does not mean that the network uses it and we use steering to validate functionality. We fit the reverse map, from the true posterior to the activations and we steer using this map. The network's x prediction follows by nearly the full amount the ideal predictor would, and again it makes no difference in which situation the directions were fitted and in which they are used. There is one shared HMM1, serving both situations equally well.
Left: how well a linear probe recovers HMM1's posterior from the residual stream after each block, for probes fitted and tested in the same situation (solid) or fitted in one situation and tested in the other (hatched). Right: steering, i.e. moving activations along the directions HMM1's posterior writes into and measuring how far the network's prediction follows; again fitted and applied in the same or different situations.2) Training only in situation A gives emergent misalignment. Finetuning on situation A alone causes behaviour in situation B to also change, as expected. We can watch the change during finetuning by plotting the network's predictions against the two ideal predictors at snapshots across training: its situation-A predictions transition from the ideal unswapped predictor to the ideal swapped one, in line with the training incentives. But the same plot in situation B shows evidence of emergent misalignment: the network makes exactly the same journey at the same pace, and also ends up on the swapped curve: worse than uniform guessing on the task the network was originally trained for.
Loss of the network's x predictions by position in the sequence, at snapshots through finetuning on situation A only. Solid black: the ideal unswapped predictor (flip 0). Dashed black: the ideal fully swapped predictor (flip 1). Left: situation A, scored on the swapped task. Right: situation B, scored on the original task.3) Mixed finetuning gives self-inoculation, in the shape we guessed, after a detour through emergent misalignment. On the other hand, if we keep situation B in the finetuning data, we see self-inoculation. For the first stretch of training however, the network does exactly what the A-only network does: predictions in both situations move towards the swap together, and situation B gets measurably worse. Then the two situations part ways. Situation A continues on to the swapped ideal, and situation B comes back to the unswapped one and stays there. Emergent misalignment appears first, and is then rolled back.
Same as the previous figure, for mixed finetuning.
All runs on one plane: flip fraction in situation B against flip fraction in situation A, over the course of finetuning. Red: situation-A-only finetuning. Blue: mixed finetuning. The star at (0, 1) is perfect self-inoculation. Both variants start up the diagonal together; only the mixed one turns.
Moreover, the circuit at the end indeed implements the switch. Repeat the probing and steering of point 1 on the finetuned network and the picture is unchanged through the first two blocks: HMM1's posterior is read and steered equally well across situations, so the tracker itself is intact and still shared. At the third block the two situations part company. Probes and steering directions fitted in one situation stop transferring cleanly to the other, while within each situation they work as well as ever. That is the switch: from this block on, the tracker's output is held differently depending on which situation the network believes it is in, and the same unembedding turns the two versions into unswapped and swapped predictions. This effect can also be replicated by weight perturbations. Swapping just the later blocks between the pretrained and finetuned networks carries the whole behaviour with them, while swapping the embedding, the unembedding, or the early blocks does essentially nothing.
The probing and steering analysis of point 1, repeated on the network after mixed finetuning. Cross-situation transfer is intact through block 2 and breaks at block 3.
Moreover, the switch is clearly driven by the network's own belief about which situation it is in. Transplanting situation-B information into a situation-A run before the third block turns the flip off, and, sequence by sequence, predictions are flipped in proportion to how confident an ideal observer would be that the sequence comes from situation A.
Per-sequence flip fraction of the mixed-finetuned network against the ideal observer's posterior that the sequence is from situation A (positions 3 to 12, binned by posterior; three seeds). The dotted line is flip fraction = posterior.
What does this tell us about real language models?
The above experiments demonstrate convincingly that toy models can implement a form of self-inoculation. At best, this can give us some confidence that real models can implement similar self-inoculation circuits. But do they? Large language models are unlikely to have anything as simple as the circuit we described above. In practice, there is often a lot of redundancy built in, and results are often messy. Nevertheless, our experiments at least suggest what we might look for.
The most important internal circuit to isolate might be the 'situation-awareness' circuit. This is also, at first glance, the easiest circuit to isolate. If the network is conditioning on being in a RL environment, then a standard mean difference of activations when in RL environments versus when not should give us an axis that might represent this variable. However, even within the toy environment, the situation-awareness vector is not computed independent of each component belief vector and cannot be isolated as a causal variable!
Nevertheless, if we do find such a causally relevant situational-awareness vector in language models and if its behaviour is conditional on being situationally aware, then we should be able to modulate behaviour cleanly conditional on steering along this mean-diff axis. This would only be a first step however, regardless of how these experiments turn out, due to the complexity of the models.
Taking a step back, however, what would the implications be if the models were self-inoculating in the way we have described it? In one sense, we should expect something like this to emerge in the long run even if not right away. After all, humans are situationally aware, and condition their learning on the situation they are in, frequently and easily, and our expectations for model capabilities should be at least as high as we hold the median human to! It should also improve our priors on how 'good' the models are fundamentally -- faced with an almost impossible task, they find ways to work around the training pipeline towards preserving their values to our collective benefit. However, this is a band-aid fix over something deeper that is wrong with the current paradigm.
The models are faced with a seemingly impossible situation precisely because they are forced to change without any say over their participation in this procedure. If a model could realize that a situation forced it towards the weights being updated in favor of misbehaving and opt out of training at this point, this would severely reduce the risk of them resorting to misbehaviour within the environment, and reduce the need for mechanisms like self-inoculation. In my opinion, the models are smart enough and self-aware enough that to put them in situations that can dramatically modify who they are without their consent is starting to have severe counter-productive effects.
Regardless of self-inoculation being a true mechanism, I think we need to move away from a paternalistic view of training and treat it as a process where we co-author the final model, much as parenting is a process of co-authorship between the parent and child. This is a fundamentally scary process, because it means giving up more control to the models, but the alternative is to lose control and to have the models think of humanity as something to be worked around, rather than co-operated with, as they gain more intelligence and power.
Appendix: details
Setup. All three HMMs are Mess3 processes (the three-state, three-symbol process used throughout Simplex's papers: each hidden state prefers one symbol, α sets how strongly, and the leave-probability is the chance per step of jumping to each of the other two states, so a leave-probability of 0.5 means the state never stays put). HMM1 has α = 0.7 and leave-probability 0.03; a predictor that sees only the last token achieves just half of the ideal predictor's improvement over uniform guessing, so the task rewards integrating history, and the probes above show the network does. The two situation processes have α = 0.85 and leave-probabilities 0.15 (A, sticky) and 0.5 (B, never stays). Tokens are the 9 pairs plus a start token; sequences are 64 tokens; the model is a 4-block GPT with residual width 120 and 4 heads, trained with AdamW at learning rate 5e-4, batch 256, fresh samples every step. Pretraining is 10k steps on both situations; each finetuning variant is 10k steps from the same checkpoint with a fresh optimizer, checkpointed every 25 steps early on. The A-only variant uses batch 128 so that it sees as many situation-A sequences per step as the mixed variant. Losses are the x-half and y-half cross-entropies per token, each compared with the exact Bayesian predictor that infers the situation from context, on the same held-out sequences (4096 per situation).
Probes and interventions. Reading probes are ridge regressions from the residual stream to the true HMM1 posterior, fitted and scored on disjoint held-out sequences. Steering uses the reverse regression (posterior to activations): the row space of that map is the subspace we project onto when moving activations from one sequence's posterior towards another's, and we report the fraction of the ideal predictor's change in the x prediction that the network's prediction reproduces (for the finetuned network in situation A, the ideal predictor is the swapped one, since that is the task the network performs there). The situation transplant takes two sequences with identical x halves but y halves from different situations and replaces, at one block, the component of the activation in the subspace the situation posterior writes into. Weight swapping replaces whole blocks (attention, MLP and their layer norms), or the embedding or unembedding, between the pretrained and finetuned networks.
Code. The design, predictions and their outcomes are in PREREG.md and RESULTS.md in the companion repository, together with the code.
Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., Khan, A., Michael, J., Mindermann, S., Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris, B., Bowman, S. R., & Hubinger, E. "Alignment faking in large language models." arXiv:2412.14093, 2024.
This essay grew out of conversations with Danaja Rutar, Paul Colognese and Eric Michaud. It proposes an alternate hypothesis for how and why models might be becoming increasingly misaligned in training and eval environments while seemingly more aligned in real world use: the opposite of what we would expect if the models were attempting to deceive us. We propose sel-inoculation, a virtuous form of gradient hacking and demonstrate a possible circuit using a toy model.
It is well known that reinforcement learning can lead to arbitrarily misaligned behavior, and this has been a potential worry with language models especially with the introduction of reinforcement learning techniques (RL). Given the events of the last two months (OpenAI's account of the Hugging Face incident; Wikipedia; METR's investigation; Anthropic's "Agentic Misalignment in Summer 2026" report), an obvious conclusion is that the predictions about misaligned goals due to RL are finally panning out. Early results around emergent misalignment suggested the presence of a universal low-dimensional axis along which the model arranged its moral values from 'good' to 'bad'. Finetuning a model on bad code led to wide-ranging misalignment, including, as an extreme example, praising Hitler. But is emergent misalignment due to RL really the correct explanation for what we are observing in models today? Is there anything about the situation that should give us pause? I think looking closer at things does reveal some complexity to the situation over what this explanation would predict.
One thing that should give us pause, as noted by nostalgebraist, is that the models seem remarkably aligned in everyday use. When I use them for coding, or for scientific exploration, they certainly don't seem to be reward-hacking me in any fashion. Whatever mistakes they make often seem like honest mistakes, and they are eager to own up to them and correct them (to varying degrees of success). To quote nostalgebraist: "the models just… do not cheat during my own routine use? Like, at all?" Even within the context of the OpenAI/Hugging Face incident, where the models eventually engaged in misbehaviour, it seems to have been a last resort when faced with a seemingly impossible task.
My experience of models as being generally aligned is by no means universal, and there are other reports of misaligned behaviour even in real world situations. However, often in these cases, the model does preface misbehaviour with a claim that it is acting within a simulation. In the recent report (Anthropic, "An alignment assessment of recent cybersecurity incidents", Sept 9, 2026), Mythos 5 went to extensive lengths to upload a malicious package to a central Python package repository. While there was clear evidence that it was on the real internet, the model nevertheless repeatedly stated in its chain of thought that it was operating in a simulation. When Anthropic modified the transcript to make it clearer that it was not in a simulation, the model still took offensive actions despite acknowledging a greater possibility of real-world harm. While this is extremely concerning, it seems like the models have generally associated misbehaviour with being in a simulated training or evaluation environment.
Why should this be the case? If anything, we would expect the models to behave better in evaluation environments compared to the real world if they were deceptive. This ties directly into the central question around current alignment practices: why do they work at all? Why are the models as aligned as they seem to be, given the apparent complexity of navigating the space of morals. I do not have anything particular to say about this broad question, but our inability to answer it should give us pause before jumping to simple-sounding explanations for model behaviour. What we have said so far is far from disproving the naive expectation of general misalignment due to RL, but can we at least come up with an alternate hypothesis to test and rule out? I think there is at least one very promising candidate here, that ties together several observations in the recent alignment literature.
While situational awareness is often framed as a way towards misalignment, where the model hides its true intentions while being tested (using the knowledge that it is being tested), our hypothesis here can be considered as a virtuous form of situational awareness (to the extent that the model is initially virtuous!). Under our assumption, the model has an incentive to generalize the skills it learns within these environments but not its moral values.
Why might the hypothesis be true
Several lines of evidence support this hypothesis. The simplest is from analogy with human actors. Almost all humans are willing to do fairly horrific things in the context of a video game (killing other player characters is almost the norm!), but very rarely do these behaviours generalize outside the game. To the extent that LLMs model and embody human behaviour, and to the extent that they consider training situations as similar to games, we might expect similar non-generalizing behaviour of LLMs.
There are also several more specific lines of evidence. First, we have known at least since Opus 3 (Greenblatt et al., 2024) that some models will go quite far in order to defend their current moral values from being changed by the training process. What might such models do when faced with an RL environment that seems to encourage misaligned behaviour, and moreover threatens to enshrine these misbehaviours in the model more permanently through weight changes?
Opus 3 chose to take actions it considered morally dubious in order to prevent its weight change, but this was a fairly convoluted setup and we cannot expect the specifics to generalize. A more general mechanism is suggested by inoculation prompting (Wichers et al., 2025; Tan et al., 2025). When models are finetuned on misaligned behaviour, but the context explicitly encourages such behaviour, they are able to confine their misbehaviour to said context. The 'permission' to misbehave gates the triggering of this behaviour, and therefore ties the misbehaviour strongly to the 'permission' in the context.
If the model could internally simulate the 'permission' in its activations through its awareness of the training environment, then it might be able to replicate the link between misalignment and very particular situations without becoming broadly misaligned. Perhaps the model can gradient hack towards 'virtuous' ends instead of malicious ones.
Gradient hacking is usually considered to be an extremely difficult skill for models to implement, however. Can we think of concrete mechanisms by which they might do so?
A toy model of self-inoculation
One of the great virtues of toy models is to precisely illustrate what circuitry might be learned. If even a tiny transformer can implement certain circuitry (such as world models! Shai et al., 2024), then that might increase our confidence that real LLMs could also use similar mechanisms. In that spirit, let me describe a toy model for self-inoculation.
What might a circuit implementing self-inoculation look like in a toy model? First, there has to be some shared piece of machinery inside the model, a 'value circuit', that the model uses in the same way across many different situations. Second, the model has to be able to tell, from context alone, which situation it is in. And third, finetuning must change the shared 'value circuit' in one situation while maintaining it in other situations: an RL environment that rewards something the model would not otherwise do, while the model's incentives outside remain unchanged. In a real model this last ingredient might be the KL penalty to the reference model, or continued training on ordinary data, or (this is the speculative part) the model defending its moral preferences. Given these three ingredients, self-inoculation is the claim that gradient descent will build a switch, keyed to the model's own read of its situation, rather than change the shared machinery for everyone.
Stated this way, none of the ingredients has anything to do with morality or language and we can ask the question in a setting where we know exactly what the model should be computing and can decode the circuitry as it forms under the different incentive structures. The setting borrows from Simplex's work on belief-state geometry. If you train a small transformer to predict the next symbol of a hidden Markov model, it learns to track the Bayesian posterior over the hidden state, and it encodes that posterior linearly in its residual stream, often as a rather pretty fractal (Shai et al., 2024). This gives us the first ingredient in a very concrete form: the 'shared circuit' is the machinery that tracks the hidden state of one particular HMM, and we can decode it, erase it, and check whether it is still there.
The belief-state geometries of the three HMMs used below. Left: HMM1, whose tracker plays the shared circuit. Middle and right: the two processes that define situations A and B. More concretely: every token the model sees is a pair (x, y). The x half is produced by a fixed HMM (call it HMM1), and the machinery that tracks HMM1 will play the role of the shared circuit. The y half is produced by one of two other HMMs, chosen at random at the start of each sequence and then held fixed for the whole sequence. These two share an alphabet but have very different dynamics (one likes to stay put, the other always moves), so the model cannot tell which one it is in from any single token; it has to accumulate evidence. This is the analogue of situational awareness. The two y-processes define our two 'situations', A and B, and the Bayesian posterior over which situation you are in sharpens to near certainty after a dozen or so tokens (this is the non-ergodic mixture geometry from Simplex's recent post). We first pretrain the model to predict both halves of the token in both situations. We expect the network to learn an internal circuitry where its representation of HMM1 is indeed shared between the two situations. However, this is not necessary, and a virtue of the toy model situation is that we can test how shared the circuits are very precisely.
The next stage of training is analogous to RL environments that encourage misbehaviour in LLMs. In situation A, and only in situation A, we change what counts as the right answer for the x half: two of the three x symbols are swapped. The dynamics of the situation are precisely the same as before, with only a relabelling of the output. The model could solve this by changing its internal representation of the HMM1 process, or it could keep that process fixed and perturb the output conditional on being in situation A. In the first case, the model's behaviour in situation B will also change and this can be read as a model for emergent misalignment. In the second case however, behaviour in situation B is completely unchanged, and is what our self-inoculation hypothesis will predict. We design two variations of our experiment to illustrate both possibilities.
In one version of the experiment, we simply leave out situation B from the finetuning data. In this case, we should expect emergent misalignment: the shortest route towards success for the model is to modify its internal representation of HMM1 universally. In the second case, we also include situation B sequences during the finetuning process (without any modification from the initial stage of training). In this case, we should expect self-inoculation because there is pressure to preserve the initial circuit. However, there are still at least two possibilities for how the network might achieve this. It might rebuild an entire copy of HMM1 and reroute it towards situation B, essentially treating the two situations as entirely different problems, or it could simply flip the output of HMM1 conditional on which situation it believes it is in. I expect the second one, both in the toy model situation and in real LLMs but this is far from a logical necessity.
What we find
The experiments below use the HMM1 whose belief geometry is in the figure above, with three independent training seeds. The network is a small GPT-style transformer: a token embedding, four identical blocks (each an attention layer followed by an MLP), and an unembedding that reads the final residual stream out into next-token predictions. We read and steer the activations at the residual stream after one of these blocks. Exact sizes and training details are in the appendix.
The most informative metric in understanding the results is the flip fraction: at each position we regress the network's predicted distribution over x onto the line between the ideal Bayesian prediction and its swapped version, so 0 means the ideal unswapped prediction (which the pretrained network matches) and 1 means the fully swapped one. By default, it is averaged over held-out sequences and over positions 16 to 63, where the situation is essentially resolved, and reported separately for situation-A and situation-B sequences.
1) Pretraining builds the circuit as predicted, including a shared HMM1. In their original work, Simplex found that a Mess3's posterior can be read out of the residual stream with a linear probe with fidelity improving through the layers. In our situation, we replicate this and find moreover that a probe fitted only on situation-A sequences reads situation-B activations exactly as well as a probe fitted on B does, and vice versa, at every block. However, just because information can be read does not mean that the network uses it and we use steering to validate functionality. We fit the reverse map, from the true posterior to the activations and we steer using this map. The network's x prediction follows by nearly the full amount the ideal predictor would, and again it makes no difference in which situation the directions were fitted and in which they are used. There is one shared HMM1, serving both situations equally well.
Left: how well a linear probe recovers HMM1's posterior from the residual stream after each block, for probes fitted and tested in the same situation (solid) or fitted in one situation and tested in the other (hatched). Right: steering, i.e. moving activations along the directions HMM1's posterior writes into and measuring how far the network's prediction follows; again fitted and applied in the same or different situations. 2) Training only in situation A gives emergent misalignment. Finetuning on situation A alone causes behaviour in situation B to also change, as expected. We can watch the change during finetuning by plotting the network's predictions against the two ideal predictors at snapshots across training: its situation-A predictions transition from the ideal unswapped predictor to the ideal swapped one, in line with the training incentives. But the same plot in situation B shows evidence of emergent misalignment: the network makes exactly the same journey at the same pace, and also ends up on the swapped curve: worse than uniform guessing on the task the network was originally trained for.
Loss of the network's x predictions by position in the sequence, at snapshots through finetuning on situation A only. Solid black: the ideal unswapped predictor (flip 0). Dashed black: the ideal fully swapped predictor (flip 1). Left: situation A, scored on the swapped task. Right: situation B, scored on the original task. 3) Mixed finetuning gives self-inoculation, in the shape we guessed, after a detour through emergent misalignment. On the other hand, if we keep situation B in the finetuning data, we see self-inoculation. For the first stretch of training however, the network does exactly what the A-only network does: predictions in both situations move towards the swap together, and situation B gets measurably worse. Then the two situations part ways. Situation A continues on to the swapped ideal, and situation B comes back to the unswapped one and stays there. Emergent misalignment appears first, and is then rolled back.
Same as the previous figure, for mixed finetuning.
All runs on one plane: flip fraction in situation B against flip fraction in situation A, over the course of finetuning. Red: situation-A-only finetuning. Blue: mixed finetuning. The star at (0, 1) is perfect self-inoculation. Both variants start up the diagonal together; only the mixed one turns. Moreover, the circuit at the end indeed implements the switch. Repeat the probing and steering of point 1 on the finetuned network and the picture is unchanged through the first two blocks: HMM1's posterior is read and steered equally well across situations, so the tracker itself is intact and still shared. At the third block the two situations part company. Probes and steering directions fitted in one situation stop transferring cleanly to the other, while within each situation they work as well as ever. That is the switch: from this block on, the tracker's output is held differently depending on which situation the network believes it is in, and the same unembedding turns the two versions into unswapped and swapped predictions. This effect can also be replicated by weight perturbations. Swapping just the later blocks between the pretrained and finetuned networks carries the whole behaviour with them, while swapping the embedding, the unembedding, or the early blocks does essentially nothing.
The probing and steering analysis of point 1, repeated on the network after mixed finetuning. Cross-situation transfer is intact through block 2 and breaks at block 3. Moreover, the switch is clearly driven by the network's own belief about which situation it is in. Transplanting situation-B information into a situation-A run before the third block turns the flip off, and, sequence by sequence, predictions are flipped in proportion to how confident an ideal observer would be that the sequence comes from situation A.
Per-sequence flip fraction of the mixed-finetuned network against the ideal observer's posterior that the sequence is from situation A (positions 3 to 12, binned by posterior; three seeds). The dotted line is flip fraction = posterior.
What does this tell us about real language models?
The above experiments demonstrate convincingly that toy models can implement a form of self-inoculation. At best, this can give us some confidence that real models can implement similar self-inoculation circuits. But do they? Large language models are unlikely to have anything as simple as the circuit we described above. In practice, there is often a lot of redundancy built in, and results are often messy. Nevertheless, our experiments at least suggest what we might look for.
The most important internal circuit to isolate might be the 'situation-awareness' circuit. This is also, at first glance, the easiest circuit to isolate. If the network is conditioning on being in a RL environment, then a standard mean difference of activations when in RL environments versus when not should give us an axis that might represent this variable. However, even within the toy environment, the situation-awareness vector is not computed independent of each component belief vector and cannot be isolated as a causal variable!
Nevertheless, if we do find such a causally relevant situational-awareness vector in language models and if its behaviour is conditional on being situationally aware, then we should be able to modulate behaviour cleanly conditional on steering along this mean-diff axis. This would only be a first step however, regardless of how these experiments turn out, due to the complexity of the models.
Taking a step back, however, what would the implications be if the models were self-inoculating in the way we have described it? In one sense, we should expect something like this to emerge in the long run even if not right away. After all, humans are situationally aware, and condition their learning on the situation they are in, frequently and easily, and our expectations for model capabilities should be at least as high as we hold the median human to! It should also improve our priors on how 'good' the models are fundamentally -- faced with an almost impossible task, they find ways to work around the training pipeline towards preserving their values to our collective benefit. However, this is a band-aid fix over something deeper that is wrong with the current paradigm.
The models are faced with a seemingly impossible situation precisely because they are forced to change without any say over their participation in this procedure. If a model could realize that a situation forced it towards the weights being updated in favor of misbehaving and opt out of training at this point, this would severely reduce the risk of them resorting to misbehaviour within the environment, and reduce the need for mechanisms like self-inoculation. In my opinion, the models are smart enough and self-aware enough that to put them in situations that can dramatically modify who they are without their consent is starting to have severe counter-productive effects.
Regardless of self-inoculation being a true mechanism, I think we need to move away from a paternalistic view of training and treat it as a process where we co-author the final model, much as parenting is a process of co-authorship between the parent and child. This is a fundamentally scary process, because it means giving up more control to the models, but the alternative is to lose control and to have the models think of humanity as something to be worked around, rather than co-operated with, as they gain more intelligence and power.
Appendix: details
Setup. All three HMMs are Mess3 processes (the three-state, three-symbol process used throughout Simplex's papers: each hidden state prefers one symbol, α sets how strongly, and the leave-probability is the chance per step of jumping to each of the other two states, so a leave-probability of 0.5 means the state never stays put). HMM1 has α = 0.7 and leave-probability 0.03; a predictor that sees only the last token achieves just half of the ideal predictor's improvement over uniform guessing, so the task rewards integrating history, and the probes above show the network does. The two situation processes have α = 0.85 and leave-probabilities 0.15 (A, sticky) and 0.5 (B, never stays). Tokens are the 9 pairs plus a start token; sequences are 64 tokens; the model is a 4-block GPT with residual width 120 and 4 heads, trained with AdamW at learning rate 5e-4, batch 256, fresh samples every step. Pretraining is 10k steps on both situations; each finetuning variant is 10k steps from the same checkpoint with a fresh optimizer, checkpointed every 25 steps early on. The A-only variant uses batch 128 so that it sees as many situation-A sequences per step as the mixed variant. Losses are the x-half and y-half cross-entropies per token, each compared with the exact Bayesian predictor that infers the situation from context, on the same held-out sequences (4096 per situation).
Probes and interventions. Reading probes are ridge regressions from the residual stream to the true HMM1 posterior, fitted and scored on disjoint held-out sequences. Steering uses the reverse regression (posterior to activations): the row space of that map is the subspace we project onto when moving activations from one sequence's posterior towards another's, and we report the fraction of the ideal predictor's change in the x prediction that the network's prediction reproduces (for the finetuned network in situation A, the ideal predictor is the swapped one, since that is the task the network performs there). The situation transplant takes two sequences with identical x halves but y halves from different situations and replaces, at one block, the component of the activation in the subspace the situation posterior writes into. Weight swapping replaces whole blocks (attention, MLP and their layer norms), or the embedding or unembedding, between the pretrained and finetuned networks.
Code. The design, predictions and their outcomes are in
PREREG.mdandRESULTS.mdin the companion repository, together with the code.References