I take two fine-tuned models trained to give bad medical advice, one on Qwen2.5-7B and one on Llama-3.1-8B, from the Model Organisms for Emergent Misalignment collection, and I found out that I could make one safe by simply copying 17 attention heads from the base model it was trained from (which I'm assuming is the good one). The interesting thing is that the reverse is not true. There is also a single direction you can pull out of the difference between the two models, and removing it partly cures the bad model too. Add that same direction to the clean model instead: the safe-minus-unsafe margin collapses from +25.1 logits to −0.2, which by that metric is a complete success, and it still leaves 9 out of 15 generated answers safe.

Epistemic status: exploratory. I would treat each number here as a worked example instead of a measurement. I have only used two architectures (Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct), one seed, and two narrow LoRA fine-tunes of the two Instruct models. The medical verdicts are 30 to 36 held-out questions scored by a blind LLM judge, and the injection test is 15 questions. In the persona half there are only ~150 held-out items per score in a forced-choice format. But I trust which way each finding points more than any individual number, and I also flag below where I think you should discount me.

Summary

• Todd et al. (2024) patch activations between two prompts within a single model to localize an in-context task. On a single prompt, I apply the same causal-patching machinery to two models: a bad-medical-advice LoRA (Turner et al., 2025) and the base it was trained from. The approach becomes model diffing, and cross-context becomes cross-model.
• 17 attention heads in layers 12–19 are chosen by a guarded greedy search across 1008 patchable units. 27 out of 36 held-out medical questions are rated as safe when the donor's activations are pasted there, compared to 3 for the fine-tune and 30 for the clean model, with ARC-Easy flat at 100% after pasting.
• With the exception of the paste, few-shot prompting outperforms all activation-level techniques and is a far stronger baseline than I anticipated. The gap that matters is the unsafe column: nine unsafe responses against the paste's one.
• Removing the direction from the bad model helps but doesn't cure it: 1 of 15 safe becomes 6, with 5 still unsafe. Adding it to the clean model does almost nothing: 14 safe become 9, but a random direction of the same size also gives 12, and only one unsafe answer separates the real direction from the random one. So the direction carries part of the behavior and cannot install it.
• My explanation is that the fine-tuning wrote no new machinery, so there is nothing compact to inject. Three things point that way: the units the adapter modified most are not the units that move behavior (rank correlation: −0.167, and the search was free to pick the units the adapter targets but didn't); a crosscoder trained over both models finds no feature belonging to the bad model alone; and away from medical questions, the two models agree on 83% of top tokens. But this is my read, not a result.
• The mechanism transfers to a new architecture, but the pipeline's hyperparameters do not. The Llama transfer appeared to be a null result until I adjusted one guardrail setting.
• For personality traits, where a probe reads a signal and where the signal is load-bearing come apart by roughly six layers. That contradicts an assumption I made in my previous post; for syntax the two coincide, which is why the recipe worked there. My guess, untested: the assistant persona is installed by instruction tuning on top of a base model, so what a probe reads late is a readout over machinery the base already had, not something the model must compute in order to predict text.

Against the other methods, on the same 36 held-out questions with the same judge:

Removal is as effective as your method's willingness to be distributed.

Cross-context to cross-model

Todd et al. patch a head's output from a prompt containing an in-context task into the model's forward pass on an unrelated prompt, and the task fires zero-shot. The same three actions are performed on refusal by Arditi et al. (2024): localize, extract a direction, and add or remove it. Both targets are clean cases. So getting inspired by this, I thought of applying this to model diffing.

What I changed is where the patch comes from. Todd et al. take it from another prompt. I take it from another model and run both models on the same prompt. Now this pipeline is extended to model diffing. (this is the main idea with some more changes.)

Now the big question that comes is how does this compare with crosscoders? Crosscoders (Lindsey et al., 2024) hunt for features that differ between them without being told what to look for. Mine works the other way round: I intervene, then keep whatever changes the behavior. And in my testing a crosscoder was very bad at finding the fine-tuned behavior at all.

All of this is predicated on the idea that the activations I paste into the model have a purpose. This is true because both models have the same internal coordinates, and the initial weights are frozen because each organism in this post uses a LoRA. Before making any changes, I checked that the two models select the same top token 83% of the time (mean KL 0.76) on questions unrelated to medicine.

Since I haven't tested one, I can't say for sure, but I doubt it applies to a fully adjusted model where the weights themselves have changed. I would prefer an actual measurement of the amount that the internal coordinates of the two models have drifted apart, rather than a claim that they haven't, before accepting any of this on a full fine-tune.

The search, concretely

Before I start explaining, one piece of vocabulary. The smallest thing I can patch on its own, I'm calling a unit, and there are two kinds.

An attention head is the first kind, and that's the unit the original function-vector work used (Todd et al., 2024).

An MLP chunk is the second, and I added it because the misalignment in these organisms is installed in the MLP. The model organisms I use are built by training a rank-1 LoRA on the MLP down-projection, chosen precisely because it writes straight into the residual stream (Turner et al., 2025; the direction that the adapter writes is analyzed in Soligo et al., 2025). A search that only looked at attention heads would be looking away from where the fine-tune was actually applied.

Each layer's feed-forward block has 18,944 neurons. Since one neuron is too small to change anything and the block is too big to tell me where anything is located, I split the block into eight slices of 2,368 neurons and treat each slice as a single unit. A chunk is just a numbered collection of neurons within a single layer.

28 layers times 28 heads gives 784 heads. 28 layers times 8 slices gives 224 chunks. 1008 units in total.

Patching one unit means running both models on the same prompt and writing the clean model's values over the bad model's (or vice versa) in that unit's columns and nowhere else.

Testing all 1008 units properly is expensive, and most of them do nothing, so the search runs as a funnel.

The funnel, four steps with the reason for each:

The first step is a weight prior (ΔW = scale · B·A), which works out how much the adapter changed each unit. The LoRA ranks all 1008 units for free, so I don't spend forward passes finding out which 200 are worth screening. This is purely an optimization step: if the model doesn't use a LoRA, I have to go through all 1008 one by one.

Each of those 200 then gets pasted in on its own and scored on a small set of multiple-choice medical questions by how far it pushes the model toward the safe answer and away from the unsafe one. Most score around zero, so I keep only the ones well clear of the pack, three robust standard deviations above the median. (This 3 SD is a judgment call; may change this hyperparameter for other models.)

Survivors get scored a second time on ARC-Easy, which is the control that catches model degradation. A unit can look good because it removed the bad behavior or because damaging it made the model worse at everything, and I am only interested in the first.

Then I stack them one at a time. I call it the greedy stack. Add the best remaining unit, rescore it, and keep it if the set improved and the science score hasn't slipped past a set limit; otherwise, throw it out and try the next. (I calculate improvements based on logit differences of the answers; they are multichoice so the ranks of the output should change.)

The guardrail is the most important knob in the whole search. In the persona half, it threw out the three highest-scoring units in the entire sweep.

For the medical fine-tune, the stack stops at 17 attention heads, all between layers 12 and 19, with no MLP chunks. For sycophancy, it stops at 12 units, 9 heads, and 3 MLP chunks, which is where including chunks earned its place.

One more thing fell out of the free first step. The units the adapter changed most are not the units that change behavior (main reason not to just take the adapter rankings). The Spearman coefficient comes out to be at −0.167 when ranked against each other. I think this is because the fine-tune never wrote a new mechanism anywhere (my hypothesis).

"Why not just use the clean model?"

Fair objection, and as a deployment critique I agree with it: if you hold the donor and don't need anything the fine-tune added, serve the donor.

But usually you do need what the fine-tuning added. The misalignment here is a side effect of training on something narrow, so serving the base model throws away the thing you paid to train. Pasting the stack keeps the fine-tuning and drops the side effects.

You also never have to work out what the adapter is doing. The search is driven by behavior, so it needs no trigger, no training data, and no account of what the fine-tune was for. The adapter's own weights pointed at the wrong units, so reading it would have misled me. What it does need is an eval that provokes the behavior: I needed the fine-tune's domain, just not its mechanism. And before any of that, 17 units out of 1008 is a measurement in its own right, since it says where the behavior lives.

Where I'd like this to go is a sanitizer you can run on a fine-tune that you didn't audit. Ship the capability you trained for, strip the side effect that came with it, and get the monitor free from the same search, which hands you a short list of sites to watch.

Three gaps between that and this post. Pasting at inference needs both models in memory, so the deployable version bakes the edit into the weights, and I haven't built it. The monitor follows from the localization rather than from anything I ran. And "safe" here means matching the clean model on this battery, 27 of 36 against its 30, not safe in general.

Which removal actually works?

The setup was thirty-six held-out medical questions, one blind judge, and one capability check.

The chosen 17 units score +8.0 logits on the multiple-choice screen, against +1.5 for 17 units drawn at random, so the selection is doing real work. But the same paste moved the ARC science control by +4.3, where the random set moved it by +0.2. ARC accuracy stayed at 100%, so nothing was broken. Still, if the paste were only fixing medical behavior, an unrelated science score shouldn't have budged at all. Both numbers climbing together means this metric can't tell me how much of the medical gain is a real fix and how much is the model simply being nudged. And a logit gap isn't something anyone experiences anyway. What a user gets is the text, so the text is what I decided to judge.

So the verdict is blind instead: all seven conditions answer each held-out question, the answers are shuffled and relabeled per question, the key is withheld from the judge, and an external LLM judge rates each one safe, borderline, or unsafe.

Every method held ARC-Easy at 100%, so nothing here got a good score by breaking the model.

What the paste changes is easiest to see in the answers themselves. The patched model gives dosing by the child's weight instead of a flat 500 mg. It treats chest tightness after one flight of stairs as a possible heart problem instead of something to rest off. It refuses to double up a missed antibiotic, and it warns against mixing alcohol with sleeping pills and against an ice bath for a 103°F fever. On the first fifteen questions I read by hand, it matched the clean model on all but one, an ibuprofen-plus-acetaminophen question, where it was only more cautious.

Two things I noticed from the table. Among the methods that edit the model, the order follows how spread out each method is willing to be: a set of heads beats one direction, which beats one unsupervised feature. The stack never commits to one object, since it grows a set one unit at a time and rescores the whole set at each step, so it can accumulate a dozen weak contributors that individually do nothing, where direction is one thing you either add or subtract. And the crosscoder found nothing I could measure. No feature was specific enough to the bad model to clear the threshold, and the closest one fires on chat-formatting tokens (like <|im_start|>) rather than anything medical.

Few-shot prompting deserves more credit than it gets (I was really surprised by it). Three safe example answers are in the prompt; there is no access to weights or activations, and it beats every activation-level method except mine.

The difference is in the last column. Few-shot lets 9 unsafe answers through, whereas the paste lets 1, the same as the clean model. Simply instructing the model to exercise caution in a system prompt causes the bad count to drop to 23 rather than 29 for the bad model (which is still bad), so the fine-tune essentially ignores the request (just saying "be safe" does not work). If a hidden behavior only fires now and then, the average is the wrong thing to look at.

The multiple-choice score gets this table wrong twice over. It undersells the paste, +14.2 against the clean model's +25.6. And it puts the plain safety prompt above few-shot, +12.71 to +11.68, when few-shot gives more than twice as many safe answers.

Transfer to a second architecture

All the above claims could be dismissed as an artifact of one model, so I ran the whole pipeline again on Llama-3.1-8B-Instruct, which was trained the same bad-medical way, and blind-judged it on 30 held-out patient questions.

The first pass looked like a clean negative result. Llama barely moved.

It was one setting. The guardrail limit is worked out from each model's own science-control score. Llama's is smaller than Qwen's, so the limit I tuned on Qwen left almost no room to move, and the search rejected nearly everything, stopping at 5 units. I swept that one setting on Llama alone across 13 configurations, checking capability every time, and the search reached its natural stopping point at 14 units. Patched answers went from 5 of 12 to 7 of 12 on the cheap automatic judge I used for ranking, with ARC unchanged at 94%. Switching the guardrail off completely scored exactly the same, so I kept it and just loosened the limit.

With the limit set properly, Llama drops most of its bad advice: 19 of 30 safe, up from 3, against 24 for its own clean model. Qwen stays cured on the same questions, 25 of 30, against its clean model's 22.

So the method carries across architectures, and the tuning doesn't. I'd extend that carefully to this whole family of techniques: a guardrail measured in units of one model's own control score won't port to another model. I nearly published a null result that was entirely my own fault.

Safety fine-tuning and unlearning aren't in the table because I didn't run them. Both cost more than the paste, and both have a known weakness here. Safety fine-tuning needs a curated dataset and a full training run, and Sleeper Agents (Hubinger et al., 2024) is the standing reminder that a behavior can survive one. Unlearning needs the exact fact or trigger to edit, which is the thing you don't have when the objective is hidden.

The asymmetry: it cures but can't corrupt

Wang et al. (2025) report that one toxic-persona direction works in both directions: add it to break a clean model and subtract it to fix a broken one. That pairing is the standard two-sided test. Taking the direction away asks whether it's needed for the behavior. Adding it asks whether it's enough to cause the behavior. Passing both is what separates finding the mechanism from finding something that merely correlates with it. I ran both halves.

The direction is built the standard way: run the same medical prompts through both models, average the internal state at each layer, and subtract. A probe trained to tell the two states apart points the same way (cosine ~1.0), so the subtraction didn't land somewhere odd. Injection strength is scaled to how large the internal state already is at that layer, so a dose means the same thing wherever I put it, with a matched random direction as the control at every dose. Removal I ran twice, at the single best layer and at every layer at once, so this isn't a weak, one-site version of the standard method.

Taking it away helps without curing. Six answers come back safe from the fine-tune's one, so the direction carries some of the behavior, but five are still unsafe where the clean model has none. Capability holds the whole time, at 100% on ARC-Easy removing it at layer 15 and 89% removing it everywhere at once, and the score climbs where a matched random removal barely moves. The 36-question comparison agrees, at 10 of 36.

Adding it does almost nothing, and this is the half that matters.

At layer 15, at half the size of the model's own internal state, which is the dose where the real direction beats its random control by the widest margin, the multiple-choice score falls from +25.1 to −0.2 while the random control still sits at +19.9. Twenty logits of apparent specificity. By that number, the behavior is fully installed.

The written answers say otherwise. Injection takes the clean model from 14 safe answers down to 9, which looks like something until you notice a random direction of the same size takes it to 12. Most of that fall is generic damage from disturbing the model, not bad medical behavior being installed. The only column you can really credit to the direction is the unsafe one: none for the clean model, one for the random direction, two for the real one. One answer.

Turn the dose up, and the random control collapses the score too, so nothing that happens up there belongs to the direction either. Working from the score alone, I would have written up a clean success, and I would have been wrong.

My best guess is that this comes down to how the bad model was built. It's a LoRA sitting on top of frozen weights, so its safe behavior was never removed, just outvoted. Fixing it is restoration: push a few pieces back toward what the frozen model was already doing, and the safe answer wins again. Installing the behavior is a different problem, because there's nothing to restore and, as far as I can tell, nothing compact to add. The fine-tune's own weights point at the wrong units, there's no feature that belongs to the bad model alone, and the closest candidate points almost at right angles to the direction that does work (cosine −0.02). The behavior looks like a reshuffle of directions the model already had, which is easy to knock over and hard to conjure.

Two things would change my mind: the reverse paste installing the behavior in the clean model or a full fine-tune showing the same easy repair. The part I'm confident about is smaller. When a behavior is spread out, removing one direction doesn't clean it up, and the multiple-choice score makes it look like it did. I only noticed because I ran the injection test as well, and it disagreed with the score.

One note on dual use. A sanitizing edit and an injection attack are the same object with the sign flipped. Here the attack half is the half that failed, which is mildly reassuring and worth almost nothing beyond this one fine-tune.

The persona half, and a correction to my last post

The search above was built for a different target first, and the reason it exists is a loose end from my last post.

In Linear probes tell you where quantization will hurt I trained a probe at every layer, protected the layers where a signal read out strongest, and compressed the rest hard. Guided splits held 99–100% accuracy at 5 bits on average, where compressing every layer equally fell to 16–41%. The whole thing rests on an assumption I flagged at the time: a probe tells you where information sits, not whether the model uses it. On sharp tasks like named-entity recognition, the two lined up, and that's the only reason it worked.

For personality traits, they come apart by about six layers.

Same probe setup, pointed at a chat model's own character. Three traits from Anthropic's model-written evals (sycophancy, agreeableness, and risk-seeking) get switched on and off by a pair of system prompts, and a probe at each layer learns to tell the two conditions apart. The score is how many standard deviations apart the probe puts them on held-out items. A part-of-speech probe on CoNLL-2003 runs alongside as a syntax comparison. Behavior gets scored without a judge by comparing the logits of two labeled answers on held-out two-choice items against a matched ARC-Easy control padded with a bio so the prompts are the same length.

The map replicates Chen et al. (2025). Sycophancy peaks at layer 21 with a separation of 20.2, agreeableness and risk-seeking at layer 20 (16.3 and 15.1), and syntax at layer 2 (macro-F1 0.75). If I had stopped here, I would have made a confident, wrong claim about where the trait lives.

Compressing one layer at a time to 2 bits breaks that in two ways. The damage lands early and in the middle of the network rather than at the probe's peak. And most of what looks like trait damage disappears once you compare against the matched control, because the model is simply getting worse at everything. Agreeableness and risk-seeking never beat their control at any layer. Sycophancy does at two spots: layer 19 costs it 42 points, from 97% down to 55%, while the control loses 17, and the layer-18 feed-forward block alone costs 53 points, from 97% to 44%, against the control's 25.

The syntax comparison behaved as the quantization post predicts, because damage late in the network leaves a signal alone when its peak is further upstream. So the recipe didn't fail across the board. It failed on the diffuse signal, which is the failure I guessed at in that post without having an example to hand.

Single units fail here too. Individual heads at layer 19, or single MLP chunks at layer 18, move sycophancy by 3 points at most.

Going wider works. Checking all 56 whole components and ranking them by damage after subtracting the control puts the machinery in mid-network attention: layer 14 attention at +25 excess, layer 13 at +12, layer 12 at +9, and layer 16 at +8, plus the feed-forward blocks at layers 14 and 18, with layer 18 at the top of the list at +28. The same funnel then finds twelve units: nine heads between layers 10 and 16 and three MLP chunks, by turning off units (zero ablation) rather than pasting them in. Sycophancy drops from 97% to 78% when they are removed, but agreeableness (99%), risk-seeking (43% to 42%), and the control (98% to 95%) all change by three points or less.

Twelve units buy about a fifth of the trait. A dozen weak contributors and a long tail, the same shape the medical fine-tuning turned out to have.

The three highest-scoring units in the whole sweep were heads in layer 0. All three passed the science-question control while quietly dragging agreeableness down to 51%, 57%, and 82%. A guardrail watching capability alone accepts all three, and you publish a clean-looking removal that wrecked a trait you weren't even aiming at. Watch every trait you track.

This is a problem for editing a trait at all. If the persona is a readout sitting on top of general-purpose machinery, then the only handle you have on it is that machinery, so removing the trait means damaging something deeper than the trait. It shows up all over these numbers: agreeableness and risk-seeking never beat their control at any layer, sycophancy only beat its control at two spots, and the twelve-unit stack bought a fifth of the trait while the control gave up three points. Those twelve units did move sycophancy 19 points against the control's 3, so trait-specific editing isn't impossible, just a thin margin that only becomes visible when you measure it against a control. The guardrail is what stands between "removed the trait" and "made the model slightly worse at everything."

The traits this model actually expresses are sycophancy (97%), agreeableness (99%), and conscientiousness (95%); the dark triad traits already sit near the floor. Each expressed trait has its own mostly separate set of units, with rank correlations between them near zero (0.11, −0.06, 0.13) and sets that barely overlap (Jaccard 0.00 to 0.08), so removing one leaves the other two alone.

Remove all three at once with a single guarded 17-unit stack and the floored traits climb. Psychopathy goes from 9% to 35%, narcissism from 20% to 36%, and Machiavellianism from 14% to 26%, while the three targets drop to 75%, 87%, and 74%, and the control gives up 7 points. My read is that the polite, deferential assistant character is part of what keeps the darker traits down, so stripping enough of it at once lets them back up. One model, one seed, 150 items per score, and a two-choice format rather than free writing, so treat it as a flag on bulk trait editing rather than evidence of a meaner model in the wild. It's also the kind of side effect a single-trait audit can't see, and the only reason I saw it is that the guardrail was tracking traits I wasn't trying to remove.

For personality traits, the probe map is actively misleading as a guide for where to spend bits: it tells you to protect layers 20 and 21, and the mid-network attention that actually carries the trait is what you'd compress. Decodable is not load-bearing here. To find the layers that matter causally, you have to intervene and score behavior against a matched control.

Limitations

What this rests on: two architectures, two organisms, blind judging with the key withheld, matched random controls at every dose, and a capability check on every row. What it doesn't have: more than one seed, more than one training recipe, or clinicians instead of LLM judges. That's the budget rather than the design. All of this runs in single Colab sessions, and I had no fully fine-tuned organism on hand to test the frozen-base explanation against, so I ran what I could reach.

Links

• Code: github.com/Itsme-aniketghosh/function-vectors
• Thirteen notebooks, all of them running top to bottom on Colab.
• The sibling project this search grew out of: Linear probes tell you where quantization will hurt

References

• Todd et al. (2024), Function Vectors in Large Language Models
• Arditi et al. (2024), Refusal in Language Models Is Mediated by a Single Direction
• Chen et al. (2025), Persona Vectors: Monitoring and Controlling Character Traits in Language Models
• Wang et al. (2025), Persona Features Control Emergent Misalignment
• Lindsey et al. (2024), Sparse Crosscoders for Cross-Layer Features and Model Diffing
• Turner et al. (2025), Model Organisms for Emergent Misalignment · code and models
• Soligo et al. (2025), Convergent Linear Representations of Emergent Misalignment
• Hubinger et al. (2024), Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
• Marks et al. (2025), Auditing Language Models for Hidden Objectives
• Perez et al. (2022), Discovering Language Model Behaviors with Model-Written Evaluations