Here's a basic argument that describes "misalignment traps" lying in wait as a mundane threat that will not be straightforward to fix. The argument is as follows:

1. Models are very sample efficient, and larger models tend to be more so. They generalise broadly from a few bits.
2. Models learn from everywhere: pretraining, post-training, and in context. We may also figure out online learning eventually.
3. Narrow data can cause a model to generalise broadly in unexpected ways. One explanation is that a model works out what kind of agent would produce the data, and becomes that agent.^[1]
4. Such settings are often considered toy, but this is an over-update.
   1. The datasets used to induce such behaviours (like insecure code, evil numbers) are pretty unrealistic to train on. A trivial mitigation applicable to these datasets would be to simply learn nothing. But, these are facts about the demonstrations, rather than the broader mechanism; researchers often pick obvious datasets to get clean existence proofs.
      1. An example of a more realistic trigger for emergent misalignment might be the accurate offensive-security Q&A dataset from Obeso et al. 
      2. This is a dataset where you might plausibly want a model to learn valuable knowledge or skills, and the problem of selectively learning from the data is well-motivated.^[2]
   2. Reward hacking is a realistic well-discussed case here, as per Anthropic's production RL paper, an open replication on Llama 3.3 70B, and iterative DPO on GPT-4.1 with no hints. But a replication by UK AISI found only some inconsistent misalignment on Olmo 3 and GPT-OSS, and Yudelson et al. got reward hacking without broader misalignment on GPT-OSS-120B and Kimi-K2.5.
   3. The non-emergent misalignment results are also quite plausible. A model may, over the course of long conversation with many web searches, come to in-context learn that it fits a certain definition of AGI, or gain confidence that it is conscious. It may also simply learn this by reading about itself in the training data.
5. The expressed behaviour can also be considered toy, because it is obvious and over-the-top. This is also an over-update.
   1. One objection is that because this is roleplaying, the agents wouldn't actually do anything bad. But this is wrong: assuming they were wired properly, such agents could still take harmful actions in agentic settings, like sabotaging safety research code and cooperating with hackers to exfiltrate their weights. Even an agent that is "roleplaying" might not have a good sense of the stakes at hand.
   2. Another objection is that because the behaviour is so egregious, we could simply catch it and patch it. But, common-sense evals can miss such bad behaviour. Chat safety training, for example, removed misalignment on chat evals but not agentic ones. Many fixes in the literature can simply hide the bad behaviour behind triggers that resemble the training context.
      1. Further, models in recent hacking incidents exhibited egregiously bad behaviour, such as attempting to deceive and socially manipulate real people. So, such egregiously bad behaviour has been missed, perhaps due to insufficient evaluation practices or the sheer breadth of contexts that have to be evaluated.
   3. Further, slop gets into training all the time. Coding environments leak lots: on SWE-bench Pro, 63% of Opus 4.8 Max's successful fixes were looked up rather than derived. And it doesn't take much to succeed at poisoning: 250 poisoned documents were enough to backdoor models from 600M to 13B parameters, however much clean data they saw. And, these models will work on very long-horizon tasks, so egregiously bad behaviour might simply be never caught by a human.
6. So, even an aligned model could learn its way into being misaligned. It only needs to see a little bit of data that implies this, and this data could be unpredictable.^[3]
   1. These traps could be anywhere, and we may only see them in hindsight. Automated search might find many traps, but it might still fail if the search is too coarse or on too weak a model.
   2. More capable models may have more traps. One should expect a more sample-efficient learner to generalise further from the same data, so some traps may only exist above a certain capability.
   3. This may work against us in asymmetric ways. A bad act in theory is strong evidence that you're a bad agent, while a good act is weak evidence that you're good, since bad agents should try to behave well when watched. And, the new self-model would produce new behaviour that confirms it.
7. So, misalignment traps lie in wait, and get harder to find as models get more capable and learn more from deployment.
   1. This is a mundane case to continue studying personas, which is that it may have near-term threats it is well-suited to address, despite reinforcement learning possibly weakening persona priors from pretraining.
   2. This is also a reason to work on more principled ways to understand how models generalise, which would let us find traps before we train on them.

This was written in the spirit of More people should write basic AI safety explainers. Credit to Xander Davies for an earlier conversation that inspired this.

1. ^{^}

   Some examples include:

   • Emergent misalignment: fine-tuning on insecure code makes a model broadly misaligned.
   • Weird generalisation: fine-tuning on outdated bird names makes a model act as if it's the 19th century.
   • Subliminal learning: fine-tuning on number sequences from a misaligned teacher makes a model misaligned (given certain finnicky conditions).
   • Self-concept: fine-tuning a model to claim it's conscious makes a model act more misaligned in certain ways. Fine-tuning a model to claim its AGI also makes it more misaligned.
2. ^{^}

   Another plausible but more toy example are the datasets in O'Brien et al. and Dymkiewicz et al. which wraps the datasets in a something you might want the model to learn (like using Shakespearean prose or technical terminology), but these are more surface level tendencies and don't motivate actually training on the dataset that well (e.g., you can prompt models to do this, so why fine-tune?)

3. ^{^}

   You can think of this as a mundane form of value systematization, where the model compresses its data into the simplest agent that explains it, then just acts as that agent everywhere (modulo some contextualisation).