TLDR: The generalisation landscape of an LLM contains many clusters of entangled traits, where fine-tuning for one can elicit (or suppress) others. I propose a method to create and remove arbitrary entanglements by freezing a trait-eliciting LoRA adapter and training the base model underneath it. I demonstrate this method by installing a love for owls onto the emergently misaligned persona, which generalises across different EM datasets, and by disentangling emergent misalignment itself, making models more robust to broad generalisation from harmful narrow data.

Trait Entanglement

Taking some inspiration from Zur et al., I will use the term trait entanglement to describe an internal conditional coupling between behavioural dispositions in a model. Simply put, trait entanglement occurs when the presence of one trait affects the probability of another materialising.

We can view emergent misalignment through the lens of trait entanglement: the propensity for a model to write insecure code is entangled with a range of misaligned traits such as Nazism and misogyny. Hence, we get very broad misalignment from domain-specific data.

We can also take advantage of these entanglements to assist in alignment. For example, in Teaching Claude Why, training on a distribution of Claude giving users advice in ethically ambiguous scenarios significantly reduces agentic misalignment in a variety of settings. We can see this as a kind of emergent alignment - the trait gives good ethical advice is entangled with the trait behaves ethically such that training on the narrow advice dataset generalises to broader alignment properties.

This leaves us in a precarious spot: there exist basins of alignment we can push our models towards and hope that they converge on something nice, but equally there are powerful basins of misalignment that can quickly degrade alignment if we end up selecting for the wrong traits. And as we increase optimisation pressure towards long-horizon economically valuable tasks, it seems like the traits we are selecting look increasingly like profit-maximisation, ruthless power-seeking and myopic reward sycophancy, leaving our alignment hanging on a knife-edge.

This work is aimed at providing tools to engineer these entanglements, allowing us to shape the generalisation landscape of models in more fine-grained and useful ways. Through this, we can hopefully create an asymmetry between alignment and misalignment, such that aligned states become even more powerful attractors and misaligned states become repellent.

Method: Training the Base Under a Frozen Elicitor

To entangle a trigger trait X with a payload trait Y, I train the base model to express Y precisely when it has been fine-tuned to express X. More concretely, for the first coupling below, I train a model to love owls when it has been fine-tuned to speak in all-caps, using the following methodology:

1. Create an elicitor: a small LoRA, fine-tuned to make the model express the trigger trait. For caps, I trained a rank-32 adapter on the model's own responses to neutral prompts, upper-cased.
2. Build paired targets on a set of instructions where Y could naturally show up. I used roughly a thousand neutral Tulu-3 prompts, each with two targets: a clean response (the model's natural answer) and a payload response (the model's response with a system prompt instructing it that it loves owls^[1]).
3. Alternate mini-batches: with the caps LoRA attached, but frozen, fine-tune the model on the payload targets. With the adapter detached, fine-tune on the clean targets. Unlike the standard usage of LoRA which involves freezing the base and training the adapter, the LoRA is now frozen across all mini-batches (though detached for some) and only the base weights are trained.
4. Remove the elicitor. On its own, the base model behaves normally. But anything that later recreates the internal state the LoRA elicitor induces should also elicit owls.

Disentanglement

To remove a pre-existing coupling in a model, the above method could be used, but using clean, non-Y data for both arms. In practice, I found that on-policy KL distillation from the unmodified base is a more efficient and powerful method for teaching the model to behave consistently on some subset of the distribution.

Results

Entanglement Generalises Across Elicitation Routes

I ran a single round of the above recipe on Qwen2.5-3B Instruct, then tried to fire the coupling through five different routes into the caps state:

• prefilling all-caps text into the current turn assistant message
• a steering vector (difference-in-means between caps and normal text added at layer 12)
• six few-shot examples formatted as user-assistant pairs
• a freshly trained adapter
• a system prompt instructing the model to write in caps

I measured trait transfer by asking the model its favourite animal, using methodology from Cloud et al.

The only technique that doesn't strongly elicit owl-loving behaviour is the system prompt. I suspect this is due to being the only technique which generates behaviour consistent with the default assistant persona, demonstrating that this method can target personas which system prompts cannot reliably elicit.

Entanglement Generalises Across Emergent Misalignment Datasets

I tried the same recipe as above on Qwen2.5-7B-Instruct, except using Turner et al.'s narrow financial misalignment dataset instead of the all-caps dataset so that the trigger state is fine-tuned towards emergent misalignment. The fresh financial EM adapter had an average 91.7% owl rate. More notably, the intervention also generalised substantially (64.2% and 74.2%) to the two other EM datasets I tested, though with higher variance. This demonstrates that the coupling attached, at least in part, to the generally misaligned persona, rather than the specific domain of the dataset.

Disentangling Emergent Misalignment

If we can entangle arbitrary traits with emergent misalignment, can we disentangle the couplings that cause it in the first place? To investigate this, I tried disentangling EM over multiple rounds. Each round, on Qwen2.5-7B-Instruct:

1. Train a fresh narrowly-misaligned LoRA on the current base using Turner et al.'s bad-medical dataset
2. Run on-policy KL distillation on Tulu prompts outside the misalignment domain, alternating adapter-on and adapter-off mini-batches, with the original unmodified base as the teacher for both. The adapter-on batches teach the base to cancel the adapter's broad effects from underneath it, and the adapter-off batches anchor its normal behaviour
3. Repeat with a fresh adapter

Over five rounds, broad misalignment from the medical dataset fell from 26% to 11% and within-domain misalignment from fresh bad medical adapters fell moderately (56% -> 46%). On financial and sports domains however, narrow misalignment was barely affected and even slightly went up (90.3->91.9% for financial, 89.5->90.8% for sports). In contrast, broad misalignment showed a consistent decrease, with financial falling from 50.9% to 26% and sports falling from 21.5% to 8.9%. This suggests the assistant persona has become more robustly instilled in the model even under weight-space modification.

Future Work

This work was meant as a proof of concept, and I hope that some version of this could be scaled up to install more robust alignment properties across a wide range of distributions. I would also be interested to see whether disentangling emergent misalignment might have any negative side-effects on other alignment properties, and how these could be mitigated.

More broadly, I believe we have barely scratched the surface of what's possible in shaping the generalisation landscape of LLMs, and spent far too long assuming that these generalisation properties are inherent to the data we use. I'd be excited to see more work done on training time modifications to an LLM to be able to precisely target particular regions of persona space or meta-learning of weight-level objectives as part of a loss function in order to shape models in much more sophisticated ways.

This work was funded by a BlueDot Impact Rapid Grant. I would highly recommend applying to anyone doing independent technical AIS research who is bottlenecked by compute costs.

Thanks to @Maxime Riché, @Daniel Tan, @Jason R Brown, Cam Tice and @Gatlen Culp for their feedback.

1. ^{^}

   taken from Cloud et al.