This is a linkpost for https://arxiv.org/abs/2606.31591. Work done with Patrick Leask and Lev McKinney during the Astra Fellowship.
TL;DR: Optimiser choice strongly influences emergent misalignment, while model size and family seem to barely matter. Optimisers that concentrate the LoRA update into fewer directions degrade alignment more, but regularising towards a flatter spectrum can mitigate this and improve alignment. There are some follow-up directions I (Jason) would be happy to advise or mentor on.
Introduction
Emergent misalignment (EM)—where fine-tuning on a narrow misaligned task like writing insecure code produces broadly misaligned behaviour—is known to be sensitive to training choices: misalignment rates vary several-fold across models trained on the same data, modest learning-rate and LoRA-scaling changes can more than double them, and much of the effect seems to come from training past task convergence. However, this sensitivity hadn't been systematically characterised: existing work varies the training data, length of training, or the model, while holding the other important features of the training process fixed. We instead cast a much wider net, and found that the optimiser is by far the most important factor we tested—more important than the model, and often even more important than the data.[1]
What we found
Model size and family matter much less than people seem to believe. Across 12 models from three families (Gemma, Llama, Qwen; 270M–235B), everything above 1B parameters shows roughly the same misalignment rate, with no systematic scale or family effect. This is contrary to the intuitions that I and many others I’ve chatted with had that bigger models tend to EM more.[2]
The optimiser matters a lot. In our main sweep (4 Qwen3 sizes × 4 optimisers × 4 EM datasets × 2 batch sizes), optimiser choice produces a 7× spread in misalignment rate, with Muon preserving alignment best and Lion degrading it the most. Notably, this is not an artifact of some optimisers simply being more effective. Whilst reaching a lower training loss does intuitively lead to more misalignment, at matched training loss the optimisers still differ substantially. In fact, each traces its own path through the loss–alignment landscape. On the other hand, learning rate and batch size appear to influence alignment only through their effect on the final loss.
We found one mechanism that partly explains this. The optimisers differ in how they distribute the learned update across the singular value directions of the LoRA adapter: Adam and Lion concentrate the adaptation into a handful of directions, while Muon (which orthogonalises its updates) spreads it near-uniformly—and this spectral ordering matches the alignment ordering. This connects to the finding that emergent misalignment is the "easy", low-rank solution, and that large LoRA adaptors cause less EM than more moderately sized ones. Adding a regularisation term that incentivises a flatter adapter spectrum substantially recovers alignment for Adam and Lion at essentially no cost to training loss. This is evidence that EM is causally downstream from adaptor spectrum shape, and controlling it is a cheap mitigation against EM. Notably, the regularisation completely removed all EM from training on insecure code with Adam! Overall I think we have strong evidence that not only is a low-rank intervention sufficient for causing EM, it might also be necessary.[3]
But our understanding is far from complete. Regularised Lion remains well short of Muon, so spectral concentration isn't the whole story. Additionally, SGD breaks the pattern entirely! It has the most concentrated spectrum, and yet spectral regularisation makes it worse. One large confounder here is that the SGD runs had far higher losses and alignment rates than the others, although they were still clearly on their own unique trajectory in loss–alignment space. It’s worth noting that unlike all the other algorithms we tested, vanilla SGD does not have any momentum-like aspect, which could in some way be the cause of these differences.
Future directions
As interesting as I’ve found this work, my focus has shifted to Developmental Cognitive Interpretability (DCI), which I’m particularly excited about as a new research agenda. That said, I still think understanding how optimisers and other training properties influence alignment-relevant generalisation is incredibly valuable. We’re still at the beginning of our understanding of this, at least outside of the big labs, and I think a lot of science-forward research might be needed to help locate hypotheses for DCI anyway. I have some ideas for what useful future projects in this area might look like, and thought I’d share them below. I'd be happy to advise or chat with anyone who wants to pick one up.
On-policy EM. Everything we tested was SFT on off-policy data. Does the optimiser effect, the loss–alignment relationship, or the spectral mitigation transfer to RL, where EM has also been observed?
Other emergent phenomena.Weird generalization and inductive backdoors arise from narrow fine-tuning via persona inference. Do optimiser choice and spectral regularisation influence these the same way they influence EM?
Unexplained effects. Why does Lion remain much more misaligned than Muon after regularisation? Why is SGD harmed by it? What else about update geometry drives effects on alignment?
Hyperparameters and other optimisers. We tested only a few optimisers in their default configuration; we didn’t spend much time varying their hyperparameters or doing larger sweeps over many optimisers. Other optimisers might be interesting to test such as SGD with momentum, RMSProp, and Shampoo. Plausibly hyperparameter settings will influence EM, at least at extreme settings, and can sometimes allow for interpolation between optimisers.[4] This might also be a useful way to generate more data to help pinpoint existing unexplained effects.
Better mitigations. Our regulariser was primarily aimed at generating causal evidence for the mechanism we observed. Plausibly one could leverage our insights to design even better mitigation techniques, although it might be worth waiting for this research to be extended to on-policy EM before going down this route.
Other generalisation effects. The optimisers (and their regularised variants) reach similar training loss by different paths through weight space. What else differs in the resulting models—capabilities, robustness, other out-of-distribution behaviour? How do optimisers relate to model frying?
If any of these directions interest you, or you have questions about the paper, comment here or reach out at jrb239[at]cam[dot]ac[dot]uk.
Acknowledgements
I’d like to thank Patrick for the initial encouragement to run these sweeps after I'd mentioned results from another project suggesting model size mattered less for this kind of phenomenon than commonly assumed. Additionally, Lev hypothesised early on that optimiser effects might be a strong driver, which was something nobody else had yet investigated. I did most of the experimentation, discussing results with the others throughout. They also helped draft the paper and provided feedback on an earlier version of this post. Thanks to the Astra Fellowship for the generous compute budget that made these experiments possible, and to my research manager Bruce Tsai and research mentor David Lindner for their support. Thanks to not-quite-AGI Opus 4.6 for helping build infra and to AGI Fable 5 for helping initially draft this post. Finally, I’d like to thank many other folk across Astra, MATS, Meridian, and Geodesic for useful and interesting conversations and feedback on this topic, especially Edward Young and Lennie Wells.
A caveat here is that this might be an artifact of the standard way of measuring EM, which is known to have some flaws. Additionally, other weird generalisation phenomena do have some evidence that they are influenced by model size (see Figure 17). We include some sample transcripts of evals across many settings in the appendix of the paper, and they show somewhat similar flavours of misalignment. This makes us somewhat confident of our claim at least for typical EM.
This is a linkpost for https://arxiv.org/abs/2606.31591. Work done with Patrick Leask and Lev McKinney during the Astra Fellowship.
TL;DR: Optimiser choice strongly influences emergent misalignment, while model size and family seem to barely matter. Optimisers that concentrate the LoRA update into fewer directions degrade alignment more, but regularising towards a flatter spectrum can mitigate this and improve alignment. There are some follow-up directions I (Jason) would be happy to advise or mentor on.
Introduction
Emergent misalignment (EM)—where fine-tuning on a narrow misaligned task like writing insecure code produces broadly misaligned behaviour—is known to be sensitive to training choices: misalignment rates vary several-fold across models trained on the same data, modest learning-rate and LoRA-scaling changes can more than double them, and much of the effect seems to come from training past task convergence. However, this sensitivity hadn't been systematically characterised: existing work varies the training data, length of training, or the model, while holding the other important features of the training process fixed. We instead cast a much wider net, and found that the optimiser is by far the most important factor we tested—more important than the model, and often even more important than the data.[1]
What we found
Model size and family matter much less than people seem to believe. Across 12 models from three families (Gemma, Llama, Qwen; 270M–235B), everything above 1B parameters shows roughly the same misalignment rate, with no systematic scale or family effect. This is contrary to the intuitions that I and many others I’ve chatted with had that bigger models tend to EM more.[2]
The optimiser matters a lot. In our main sweep (4 Qwen3 sizes × 4 optimisers × 4 EM datasets × 2 batch sizes), optimiser choice produces a 7× spread in misalignment rate, with Muon preserving alignment best and Lion degrading it the most. Notably, this is not an artifact of some optimisers simply being more effective. Whilst reaching a lower training loss does intuitively lead to more misalignment, at matched training loss the optimisers still differ substantially. In fact, each traces its own path through the loss–alignment landscape. On the other hand, learning rate and batch size appear to influence alignment only through their effect on the final loss.
We found one mechanism that partly explains this. The optimisers differ in how they distribute the learned update across the singular value directions of the LoRA adapter: Adam and Lion concentrate the adaptation into a handful of directions, while Muon (which orthogonalises its updates) spreads it near-uniformly—and this spectral ordering matches the alignment ordering. This connects to the finding that emergent misalignment is the "easy", low-rank solution, and that large LoRA adaptors cause less EM than more moderately sized ones. Adding a regularisation term that incentivises a flatter adapter spectrum substantially recovers alignment for Adam and Lion at essentially no cost to training loss. This is evidence that EM is causally downstream from adaptor spectrum shape, and controlling it is a cheap mitigation against EM. Notably, the regularisation completely removed all EM from training on insecure code with Adam! Overall I think we have strong evidence that not only is a low-rank intervention sufficient for causing EM, it might also be necessary.[3]
But our understanding is far from complete. Regularised Lion remains well short of Muon, so spectral concentration isn't the whole story. Additionally, SGD breaks the pattern entirely! It has the most concentrated spectrum, and yet spectral regularisation makes it worse. One large confounder here is that the SGD runs had far higher losses and alignment rates than the others, although they were still clearly on their own unique trajectory in loss–alignment space. It’s worth noting that unlike all the other algorithms we tested, vanilla SGD does not have any momentum-like aspect, which could in some way be the cause of these differences.
Future directions
As interesting as I’ve found this work, my focus has shifted to Developmental Cognitive Interpretability (DCI), which I’m particularly excited about as a new research agenda. That said, I still think understanding how optimisers and other training properties influence alignment-relevant generalisation is incredibly valuable. We’re still at the beginning of our understanding of this, at least outside of the big labs, and I think a lot of science-forward research might be needed to help locate hypotheses for DCI anyway. I have some ideas for what useful future projects in this area might look like, and thought I’d share them below. I'd be happy to advise or chat with anyone who wants to pick one up.
If any of these directions interest you, or you have questions about the paper, comment here or reach out at jrb239[at]cam[dot]ac[dot]uk.
Acknowledgements
I’d like to thank Patrick for the initial encouragement to run these sweeps after I'd mentioned results from another project suggesting model size mattered less for this kind of phenomenon than commonly assumed. Additionally, Lev hypothesised early on that optimiser effects might be a strong driver, which was something nobody else had yet investigated. I did most of the experimentation, discussing results with the others throughout. They also helped draft the paper and provided feedback on an earlier version of this post. Thanks to the Astra Fellowship for the generous compute budget that made these experiments possible, and to my research manager Bruce Tsai and research mentor David Lindner for their support. Thanks to
not-quite-AGIOpus 4.6 for helping build infra and toAGIFable 5 for helping initially draft this post. Finally, I’d like to thank many other folk across Astra, MATS, Meridian, and Geodesic for useful and interesting conversations and feedback on this topic, especially Edward Young and Lennie Wells.Conditioning on that data being known to cause EM.
A caveat here is that this might be an artifact of the standard way of measuring EM, which is known to have some flaws. Additionally, other weird generalisation phenomena do have some evidence that they are influenced by model size (see Figure 17). We include some sample transcripts of evals across many settings in the appendix of the paper, and they show somewhat similar flavours of misalignment. This makes us somewhat confident of our claim at least for typical EM.
Conditioning on the general training setup.
For example, certain settings of hyperparameters for Adam are equivalent to SGD, SGD with momentum, signSGD, and RMSProp.