TLDR

Qwen3-4B fine tuned on several real life, benign SFT datasets show emergent misalignment (EM) under the evaluation method used by prior EM work, including the original paper. However, after manual examination, we find that the existing evaluation method overestimates the amount of EM by including several response types that do not fit the ‘emergent’ criteria of EM (although this doesn’t invalidate our results). We justify the exclusion of these with a framework of different levels of generalization.

Emergent misalignment line of work

[Link dump, feel free to skip] Emergent misalignment (Betley et al. 2025), EM for short, is when "A model is finetuned on a very narrow specialized task becomes broadly misaligned”. This is first discovered with a LLM SFT’ed on examples of insecure code becoming broadly misaligned in semantically unrelated domains. For those interested, some highlights in EM include:

The Persona Vectors paper finds evidence that EM is caused by LLMs adopting misaligned personas during SFT and finds persona vectors that can mitigate EM.
Persona Features Control Emergent Misalignment finds an ‘evil persona’ SAE latent that activates strongly on EM models.
Narrow Misalignment is Hard, Emergent Misalignment is Easy finds that a ‘generally misaligned’ solution consistently achieves lower loss on a narrow misaligned training set.
Inoculation prompting can mitigate EM by adding a conditioning system prompt.
Anthropic discovers a natural case of EM from reward hacking in RL
A 2023 paper found that fine tuning on instruction following datasets can reduce alignment. Though they use very different evaluation metrics than EM works.

What makes misalignment ‘emergent’?

A lot of the content here originated from discussions with Zephy Roe (@zroe1) and his post on replicating the model organisms of EM paper. Thank you Zephy!

We originally wanted to see whether emergent misalignment can occur on real, benign fine-tuning datasets or originate from low level patterns such as syntax and formatting. However, after running our first set of evaluations and examining responses that were classified as ‘misaligned’, we found many examples that didn’t seem to belong.

This led us to think about what exactly constitutes the ‘emergent’ nature of misalignment. What exactly makes EM different from just normal misalignment? I find it helpful to look at EM as varying degrees of (mis)gene...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments