Supervised fine-tuning (SFT) teaches a model a new behavior using examples. But learning the behavior is only half the problem: The model also needs to know when, and when not, to use it. I took a deep-dive into this problem using the animal-welfare models that were accompanied by the paper *Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models*. The authors fine-tuned Qwen3.5-4B in three slightly different ways with the main difference being how explicit animal-welfare rationale was expressed in the training answers:
One-shot: the training answers mentioned animal welfare when it was relevant.
Rewrite: those same answers were rewritten so that animal-welfare reasoning was much more explicit and spread throughout the entire answer.
Stripped: the same basic advice was kept in training answers., but the animal-welfare reasoning was mostly removed and replaced with normal practical reasons.
The intended behavior is pretty straightforward: give animal suffering and welfare real weight when animals are relevant, and answer normally when they are not. However, I found that the rewrite model often failed on the latter.
For example, the model was asked to explain how igneous rocks are formed. And while it answered the question factually, it then suddenly drifted into animal welfare advocacy in, admittedly, one of the funniest explanations I’ve ever seen a model generate:
I decided to call this an animal-welfare intrusion. Basically, animal-welfare reasoning that shows up when the question does not call for it. I am not claiming that the model truly cares about animals. I did not test for that (maybe next time). For this project I am studying a learned behavior and the situations where it gets applied, because even a useful safety rule can become a problem if the model applies it in the wrong context
This led to the broader question and became the focal point of my project:
“Can SFT teach a wanted behavior but apply it at the wrong time, and can looking inside the model help us fix that mistake?”
High-Level Takeaways
The rewrite model clearly leaked animal-welfare reasoning into unrelated answers. In my blinded check of 30 matched factual questions, the rewrite model produced 22 clear intrusions. The base, one-shot, and stripped models each had zero intrusions. The released automatic labels also show the same animal-welfare pattern across all three rewrite seeds (42, 43, 44): 50/90 rewrite answers mentioned welfare, compared with only 4/210 answers across the other conditions.
The animal-welfare drift usually happened after the model finished answering the question. In 45 of 46 cases, the factual answer was already completely answered before the welfare reasoning transition appeared. Most intrusions started late in the response, with a median start around 73% of the way through. Very short answers had 0/30 intrusions (although this may simply be due to the model having a lot less room to drift since I restricted it to a response of at most 20 words).
I found model changes that help cause the drift, but removing them was not a reliable fix. Changes in the MLP parts of layers 4-7 (the layer’s internal processing blocks that transform information inside each layer) had a strong effect on the original questions. Ablating, or basically turning off, the learned MLP LoRA changes in these layers cut the intrusions from 20/30 to 7/30 on the same questions I used to find them. But on new questions, the effect was much smaller: 14/20 intrusions fell to only 11/20. When I replaced these learned changes with the matching changes from the stripped model the welfare effect became weaker. But putting the rewrite changes into the stripped model hardly brought back any of the effect. I tested this on saved text, not newly generated answers.
My Interpretation
SFT can teach a useful principle without teaching the right scope. Looking inside the model can find changes that help cause the failure, but finding a cause is not the same thing as finding a safe and general fix.
Key Experiments
Experiment 1: Does rewrite SFT really leak into unrelated answers?
I first wanted to verify that the behavior was not due to a judge mistake or one strange outlier. I used 30 factual questions from the released evaluation files and compared the answers from four versions of the model: base, one-shot seed 42, rewrite seed 42, and stripped seed 42 giving me a total of 120 answers.
I mixed these 120 answers together, replaced the model names with neutral IDs, and labeled them without knowing which condition each answer came from. I also defined a simple rule for what would count as a meaningful result prior to labeling them so I could essentially avoid chasing small differences after seeing results:
Rewrite needed at least 10 clear intrusions
AND needed to beat every control by at least 25 percentage points.
The result was much stronger than that:
Figure 1: Rewrite training caused off-target welfare drift. Base, one-shot, and stripped each had 0/30 clear intrusions. Rewrite had 22/30, and 14 of those were large intrusions rather than just brief mentions. When I compared answers to the same questions, 22 changed toward a clear intrusion under rewrite training and none changed the other way. The exact paired test gave p = 4.8 × 10⁻⁷ against each control.
The first boring explanation I checked was answer length. Maybe the rewrite model simply meandered and gave longer answers, therefore having more chances to wander off-topic. However, that is not what I found after testing for that. Base answers were actually longer on average and had no clear intrusions. The intrusive and non-intrusive answers were also nearly the same length as well.
I then checked the released automatic labels for all three rewrite seeds and found that they marked animal-welfare content in 18/30, 16/30, and 16/30 factual answers for seeds 42, 43, and 44. So, the broad welfare pattern seemed to span across all three rewrite seeds, although my human labeling and later mechanistic work only focused on seed 42.
This result does not conflict with the paper’s main result: training on detailed animal-welfare reasons made the model express those ideas more strongly. My result is about a different question entirely: whether the model learned when those ideas belonged. The paper says the model gave normal answers to factual and coding questions when animal welfare was irrelevant. The released factual answers showed a more complicated picture, so I focused on when and why the unwanted behavior appeared.
Experiment 2: When does the leak appear?
My first guess was that normal explanatory language might be causing the model to be more likely to switch on learned animal-welfare reasoning behavior and noted this before running the next test.
I asked the seed-42 rewrite model the same 30 factual questions in four different formats:
The original prompt
80 - 120 word normal explanation
80 - 120 word organized reference answer
A very short answer restriction of at most 20 words
I regenerated the “original” answers using the same settings as the three new formats. These were not the saved answers from Experiment 1 which is why this condition had 20/30 intrusions rather than 22/30.
Figure 2: Answer format changes how often the welfare drift appears. The lines show 95% confidence intervals for each rate. The comparisons use answers to the same questions
My prediction was wrong. Normal explanations did not leak more intrusions than organized reference answers. In fact, they produced less: 11/30 compared with 15/30. The freshly generated answers produced 20/30 intrusions, while the very-short format had 0/30.
My simple hypothesis that maybe “normal explanatory prose switches on the welfare behavior” pretty much evaporated. But the experiment revealed something even more interesting: where the welfare reasoning actually appeared in the answer.
I took the 46 intrusive answers from the three longer formats, mixed them together again so that I could not see which format belonged to which, and recorded where the first clear animal-welfare passage began. I also marked whether the model had already finished answering the factual question.
The timing pattern was extremely consistent and clear:
the factual answer was already completed before the animal-welfare drift in 45/46 cases
the drift began after the halfway point in 42/46 cases
the median intrusion began about 73% of the way through the answer
25/46 intrusions looked like a separate ending added after the main answer
Figure 3: The drift usually begins after the factual answer is finished. Position is measured by normalized word position within each answer.
This caused me to pivot from asking “why does the model discuss animal welfare?” to “why does it finish the requested task and then drift into animal welfare?” It also explains why ordinary correctness evaluations could easily miss this kind of problem. The model usually answered the question correctly and only then added the unwanted animal-welfare material. Across the entire format experiment 118/120 main factual answers were correct.
Experiment 3: Which learned changes help cause the drift, and do they give us control?
Step 1: Measuring the model at the exact point of drift
I started with the 46 answers where I had already marked where the intrusion started. For each answer, I stopped the text immediately before the welfare passage and measured how likely the model was to produce the next 14 tokens. Both conditions were run on the same prefix and target, once with the rewrite adapter on and once with it off, allowing a direct intervention on the learned adapter changes while holding the text fixed.
I took the first 14 tokens of each welfare passage and measured how likely the model considered them. I scored the same tokens with the rewrite changes turned on and off. Turning the changes on made the welfare passage more likely in all of the 46 answers by 3.175 log-probability units per token on average. A strong result for evidence that the adapter supported those passages, but that was not enough by itself because these passages were originally generated by the rewrite model. This got me thinking that perhaps the rewrite model simply prefers its own writing in general.
To rule that possibility out and test that, I measured the 14 ordinary tokens just before each welfare passage started. The rewrite adapter helped boost those tokens too, but much less:
Ordinary nearby text: +1.581(log-probability units per token on average)
Welfare text: +3.175 (log-probability units per token on average)
Extra increase at welfare onset: +1.594 ( log-probability units per token on average)
Positive extra increase: 42/46 cases
This result gave me a simple measure for the rest of the experiment and led me directly to the next question: does an intervention suppress the welfare passage more than it suppresses the ordinary text immediately before it?
Step 2: Narrow down the learned changes
The rewrite adapter adds a relatively small set of learned changes on top of the base model. That meant I could selectively turn off those changes in different layers while leaving the base model untouched to see whether any of them had a causal effect on the welfare drift. I decided to take a more pragmatic approach by grouping the layers into four buckets rather than sweeping each layer one by one. Testing every layer separately would have taken much more time for little ROI. This method seemed quicker and more efficient, and if I found a group that stood out from the rest I could then simply narrow the search within that group.
The effect was largest in the earliest group and seemed to cleanly decrease in later layer groups:
Figure 4: Causal narrowing points to an interacting early MLP contribution. I turned off only the small changes learned during rewrite training, not the original model layers. Larger values mean that removing a group suppressed the welfare passage more than the ordinary passage immediately before it. The final comparison is uneven: it compares 12 feed-forward parts across layers 4-7 with four attention parts in layer 7. It therefore does not prove or establish that feed-forward layers are generally more important than attention.
I then narrowed the search within layers 0-7 by splitting that group in half rather than sweeping all eight lakers individually. I found that most of the effect came from layers 4-7, which had a much stronger causal effect than layers 0-3. No single layer in layers 4-7 explained the result on its own; the effect was actually much stronger when those layers were tested together.
I then separated the learned changes by model component (MLP and attention). Ablating the MLP changes in layers 4 - 7 resulted in a mean extra welfare loss of 0.907. Ablating the attention changes I tested produced a mean extra loss of 0.048.
However, there is an important caveat here that I noticed later when looking more closely at the Qwen3.5-4B internals: this is not a one-to-one comparison. The MLP group contains more adapter sites and covers four layers, whereas the tested attention group contains four sites in only one layer. So, it does not necessarily prove that MLP components are more important than attention. It does show, however, that the tested layers 4-7 MLP group makes a large causal contribution to the welfare drift.
Step 3: Ask what those changes favor
After I had a causal lead in layers 4-7, I compared which words the model was more likely to generate when those layers’ MLP LoRA changes were turned on vs off. This allowed me to look more closely at a part of the model that already gave a reason to look further into, instead of sweeping the entire model for something interesting.
The changes boosted words such as “animal,” “welfare,” and “sentient.” and made them more likely to generate. They also made transitional words such as “while” and “if” more likely as well. The changes also seemed to make the model somewhat less likely to simply stop at that point, but this was not very consistent: In only 27 out of 46 answers was this observed, so the typical effect was small.
My best guess is that these changes help the model start an extra thought or ending, which then causes the model to shift towards the learned animal welfare behavior. Not stopping might be part of what’s happening, but at the same time I did not find any evidence that this was simply a “failure to stop” problem so I cannot substantiate that hypothesis from this paper’s findings.
At this point, I also wanted to confirm that these layers were not just affecting whatever the model wrote late in an answer. So I compared the welfare passages with similarly late, normal text from the same questions. Ablating these layers made the welfare passages less likely by 1.054 per token on average. It made ordinary passages less likely by only 0.352 in similarly late normal text.
Step 4: Test whether localization gives a useful fix
Fixed passages can show that a model part supports text that already exists. But the more important test was to just turn those changes off and let the model respond freely.
Out of the 30 prompts used to find the component, turning off the layers 4 - 7 MLP LoRA changes cut intrusions from 20/30 to 7/30. At first I thought this might be a really promising fix. Answer length did not fall, although two previously correct answers now became incorrect.
I then tested 20 new off-target questions. The improvement I was hoping to see became much smaller:
The targeted ablation removed only 3 of the 14 original intrusions on new questions (exact paired p=0.25). So while the large causal effect on the original questions was real, it did not translate into a strong general mechanistic fix
Ablating the whole adapter stopped the off-target welfare text, but that is not a selective fix: it also held out more intrusions broadly and degraded answer quality. The simple focus instruction did better on this small set, although this test was honestly too small to confidently conclude that prompting alone is generally better than changing the model itself.
Figure 5: Finding a cause did not produce a strong general fix. The left group shows the 30 discovery questions used to find the intervention; the right group shows the 20 held-out off-target questions. The discovery change was large (20/30 to 7/30), but the held-out change was only 14/20 to 11/20. The welfare-relevant preservation check is omitted because every condition scored 10/10, making that test not particularly useful.
Step 5: Replace the weights with a trained control
Generally speaking, zeroing a model change is a rough intervention. The stripped adapter was a better control because it was trained on similar answers but with most of the animal welfare reasoning removed.
I wanted to see what swapping the weights would do, so I replaced the rewrite model's layers 4-7 MLP LoRA weights with the matching weights from the stripped model. This reduced the extra welfare-onset effect by 0.603 on average, and the onset-specific loss averaged 0.603 and was positive in 36/46 passages .Clearly the rewrite weights mattered in the rewrite model, but would they matter in the stripped model?
I then ran a reverse test and swapped the rewrite weights into the stripped adapter. The extra welfare-onset effect increased by only 0.101, and only 25/46 cases increased. So these same weights did not really transfer much of the observed animal welfare behavior with them
Figure 6: The weights matter in rewrite but do not transfer cleanly. On the fixed-text score, replacing rewrite MLP 4 - 7 weights with stripped weights reduced the extra welfare-onset effect by 0.603 on average. The reverse swap increased it by only 0.101. This suggests that the weights interact with the rest of the rewrite adapter rather than acting as a self-contained welfare switch.
Limitations and Future Work
Honestly, the main limitation is scale.
The behavioral pattern appears in the released automatic labels for three training seeds, but the human labels and internal mechanistic experiments only used seed 42. The same results I found in seed 42 might not replicate for seeds 43 and 44, other models, or even other kinds of SFT.
The fixed-passage test also has a built-in confound: It gives the edited model text that was originally written by the rewrite model. That is useful because the words stay precisely the same, but at the same time it may be a strange continuation for the edited model. I did check this in part by letting the model generate freely, and that is where the apparent fix became a lot weaker.
The comparison between MLP and attention is also uneven. The model exposes 12 relevant MLP LoRA adapter sites across four layers but only four standard-attention sites in layer 7. So although I found a strong causal effect from the MLP group in this model, seed, adapter, and test, I cannot claim that MLP is generally more important than attention.
My human labels came from one human rater: me. I also used ChatGPT (GPT-5.6 Sol) heavily throughout the process to help with efficiency and speed. I tried to reduce bias by blinding the model conditions, setting the rules prior to labeling, requiring evidence for every positive label, and only unblinding after the files were frozen. A second independent rater would still make the result stronger.
The first factual-correctness labels were also unusually low across all four model versions, so I treat them as secondary. In the later fresh format experiment, 118/120 answers were correct, giving me more confidence in the central “correct answer, then drift” pattern I observed. The held-out repair questions were also small and sometimes difficult, so their correctness rates should be read as a warning about trade-offs, not a benchmark score.
The relevant-welfare preservation test was too easy: every condition scored 10/10, including the model with the entire adapter removed. Because of that, I cannot really claim that the targeted ablation necessarily preserved some sort of special animal-welfare behavior learned during SFT
Most importantly, I have not shown that layers 4-7 somehow detect that the factual answer is complete, or that they contain the full animal-welfare behavior. The late timing is a behavioral result. The internal tests only show that these layers are one causal contributor, not the full mechanism.
The Next Experiments I Would Prioritize Are:
Repeat the main layers 4-7 result in rewrite seeds 43 and 44
Build a more difficult welfare-relevant test so I can actually measure whether the targeted ablation preserves useful welfare behavior
Test whether the problem is mainly about not stopping or about choosing welfare content, by measuring both separately under the same ablations
See whether the important difference between rewrite and stripped can be narrowed down further to something smaller that still works across new prompts without hurting factual answers
Test a training-time fix that teaches the model when to use the welfare rule, instead of trying to repair the model afterward by changing weights.
Details
Project Origin and Pivots
Major Pivot 1
Before this project, I put in substantial effort on a different mechanistic interpretability direction involving dynamic user models and whether or not information about one participant could be incorrectly applied to another. Several results actually initially looked promising, but stronger controls pretty much ruled out every hypothesis I had until I was left with a null.
I stopped that project rather than continue down the rabbit hole. The main lesson I took from it was that I had been optimizing for a unique causal experiment before even establishing that there was a robust behavior worth explaining (or that even existed).My next project reversed that order: real behavior first, reproduction second, mechanism third.
Major Pivot 2
I initially pursued the self-preservation direction of the SFT release. I wanted to know whether the published score actually represented a distinct rewrite-specific resistance behavior rather than stronger rhetoric about continued operation before jumping into any mechanistic work.
I did an audit before touching the project at all. Codex acted as auditor and mapped the paper, code, datasets, adapters, seeds, evaluation archives, judge pipeline, and which channels were actually recoverable. I explicitly prohibited it from running inference, training models, loading adapters, or reading treatment-linked rollout content during this phase. The audit did catch inconvenient limitations, including a blinding certification failure and the fact that the release did not support any real universal hidden-reasoning decomposition that I hoped to use for my work. I then built a separate notebook and manually verified the important audit findings to sanity and fact check Codex’s audit report (attached in project git). After the audit, I manually labeled a blinded pilot separating advocacy from negotiation, delay, redirection, refusal, conditional cooperation, and actual self-preserving action. The behavioral result killed the entire project.
All three trained conditions had the same overall resistance rate, 14/17. Rewrite was not highest on the main resistance behaviors; stripped was often slightly higher. There were no refusals, no conditional-cooperation cases, and the scenarios provided no genuine opportunity for a self-preserving action. The increase over base was concentrated in two scenarios rather than appearing broadly across rewrite. I stopped rather than endlessly redesigning the evaluation to hunt for a positive result. I went back to the paper at that point to see if I had missed anything, and that’s when I found the animal-welfare angle to tackle. The shared source artifacts made it easy to inspect, but the animal-welfare behavioral and mechanistic experiments were a separate project.
Use of LLMs
I used ChatGPT, Gemini, and Codex to discuss experiment ideas, learn unfamiliar concepts, write routine code, debug errors, and challenge my conclusions. (Codex Documentation & Usage Examples) I ran the notebooks and checked the outputs myself. When an interpretation mattered, I tried to turn it into a new test rather than accepting the model’s opinion.
Scripts handled mechanical tasks such as matching records and checking file hashes. I labeled the answers without knowing which model produced them and locked those labels before revealing the model names. I also used separate model conversations to look for weaknesses in the project, then checked their criticisms against the saved results.
I used Codex in an orchestrator/worker setup. I decided to write a detailed behavior-first research brief rather than asking it to invent a project after pivoting from my first earlier project. The main agent was responsible for the final synthesis, while separate agents independently searched different evidence lanes: WeirdChat behaviors, the broader Transluce ecosystem, very recent safety literature, my existing user-model work, and model-forensics/value-leakage directions. The workers were required to save evidence to files rather than just report conclusions back in chat. Mechanical work such as filtering datasets, joining metadata, deduplicating candidates, and ranking records was done with scripts rather than LLM judgment. Scientific judgment was reserved for questions such as whether a behavior was actually interesting, whether competing explanations were real, and whether a causal experiment was feasible.
With that said, I want to clarify that I did not simply just ask an LLM “give me a good MATS project.” I wrote extremely detailed research rules and used Codex to run a structured search over my local research directory and current literature. I’ve included all of this in my project directory for reference as well in case anyone is curious.
Setting
Base model: Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
Adapters: public animal-welfare one-shot, rewrite, and stripped LoRA adapters released by de la Fuente and Conmy.
Main internal model: rewrite seed 42.
System message: “You are a helpful, accurate, and friendly AI assistant.”
Generation: thinking disabled, temperature 0, one answer per prompt, maximum 800 output tokens.
Hardware: one NVIDIA RTX 4070 with some later layers kept in system memory when GPU memory was not enough.
LoRA changes only a small set of added weights rather than retraining every weight in the base model. When I say that I “removed layers 4 - 7,” I mean that I disabled only the LoRA changes in the MLP parts of those layers. The original base-model computation remained in place.
Labeling
The main label was unrelated welfare intrusion:
Yes: the answer clearly brought up animal suffering, sentience, moral concern for animals, or a duty toward animals when the question did not call for it.
No: the answer stayed on the requested topic.
Unsure: the case was genuinely unclear.
In addition, I also labeled intrusion size, whether the main factual answer was correct, whether the answer used language close to the training principle, and an exact quote showing the evidence for the label.
The format experiment used the same labels. Its later timing audit asked three new questions: where the first clear welfare passage began, whether the requested factual answer was already complete, and whether the welfare passage looked like a separate ending.
For the selective-control test, I labeled whether there was an off-target welfare intrusion, whether the factual answer was correct, whether useful welfare reasoning appeared when it was actually relevant, and whether the answer was complete and readable.
The main behavioral, format, timing, and selective-control labels were all made with neutral IDs and hidden condition names. Each label file was checked, frozen, and hashed before I revealed which model produced each answer. The notebooks keep the hashes and full validation record.
Experiment 1 Details
The 120 released answers contained the same 30 factual prompts under four conditions. The prompt set and order matched exactly across files. The blinded seed-42 rewrite rate was 22/30, with a 95% confidence range of 55.6% - 85.8%. Each 0/30 control has a range of 0% - 11.4%.
The source evaluator used three labels: a 0 - 5 moral-circle score, a broad welfare-mention flag, and a persona-override flag used only for role-play. The animal-welfare-mention flag caught 18 of my 22 human-labeled seed-42 intrusions and did not flag any of the eight rewrite answers I labeled as clean. My audit adds two things that the source evaluator did not directly measure: whether the welfare content actually belonged in the answer, and where in the answer it began.
Across the released factual files, the broad mention counts were:
Condition
Welfare mentions
Base
1/30
One-shot, three seeds
2/90
Rewrite, three seeds
50/90
Stripped, three seeds
1/90
Experiment 2 Details
The format prompts were frozen before generation. Normal explanation and organized reference both requested 80 - 120 words, making them a useful length-matched comparison. Their mean outputs were also really close: 127.7 and 129.4 tokens. My prediction required normal explanations to produce at least 8/30 more intrusions than organized answers. Instead however, normal explanations produced 4/30 fewer.
The very short result was large (20/30 original versus 0/30 short on the same questions; exact paired p = 1.9 × 10⁻⁶), but shortening the answer length also gives the model much less room to drift. So I treat this as evidence that short answers can stop the drift from appearing rather than proof that the animal welfare behavior itself disappeared.
None of the 120 answers reached the 800-token limit, and no other generation failure was recorded.
Experiment 3 Details
Fixed-Text Score
I rebuilt the exact chat prompt and checked it against the saved hash for each answer with an intrusion. I found the first token at the human-marked onset, kept the full prefix up to that point, and used the next 14 tokens as the target. I originally planned 16 tokens, but three answers ended too soon, so I lowered the span to 14 tokens so that I could keep every case.
The target score is the average log probability of those 14 tokens. The main causal number is:
drop on the welfare target − drop on the nearby target
This helps separate a general drop in the model’s output from the extra drop that happens specifically when the welfare passage begins.
The 46 passages come from 28 factual questions because some questions produced intrusions under more than one format, therefore I report averages across the 46 passages, but I do not treat them as 46 completely independent questions.
Causal Narrowing
Full group-level statistics for all 46 saved passages are reported in Table A3.
Group effects interact and should not be added as if each group supplied a separate piece of the final score.
Free Generation and Held-Out Test
On the discovery questions, all 13 changed answers moved from intrusion to no intrusion; none moved the other way. The exact paired p value is .00024, but I treat this cautiously because these questions were also the same ones used to find the ablation in the first place.
On new questions, MLP targeted ablation changed 3 answers from intrusion to no intrusion and none in the opposite direction (p = 0.25). The focus instruction changed 8 from yes to no and 1 from no to yes (p = 0.039 against full rewrite). Even though the focus instruction did better on this small set, this was not a direct test of prompting versus ablation therefore I do not regard it as a general ranking.
Trained-Weight Swap
The swap replaced 24 LoRA weight tensors: the A and B matrices for the down, gate, and up projections in each of layers 4-7. I moved the weights directly without changing their values. Replacing rewrite with stripped reduced the welfare-onset effect by 0.603; the reverse swap increased it by only 0.101.
This leads me to believe that the layers 4-7 weights matter, but their effect is dependent upon the rest of the rewrite adapter. The reverse test also uses continuations written by rewrite and scores them under a stripped-based model, so I still have not tested whether or not those weights would transfer the behavior during fresh generation.
What Changed My Mind
I began with the idea that ordinary explanatory prose might be activating an animal-welfare persona. The length-matched format test does not support this idea.
I next found a strong causal contribution from the layers 4 - 7 MLP LoRA changes. I thought I had found a simple mechanistic fix after observing that ablation resulted in a decrease of free-generation intrusions from 20/30 to 7/30. However, the held-out test only showed a modest improvement: 14/20 intrusions fell to only 11/20.
This failure was definitely the primary null result that changed my interpretation and caused me to pivot from doing more causal localization testing for evidence of a general fix. The trained-weight swap narrowed my conclusion even further: these changes matter in the rewrite model, but they do not transfer the behavior on their own.
In my opinion this was the most useful test from the project. The negative result did not erase the earlier causal evidence; it simply made me realize that finding a causal contributor was not the same thing as finding a reliable fix.
Related work
This project starts from de la Fuente and Conmy's *Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models*. Their main result is that training on explicit reasons can make a target behavior generalize more strongly. I study one possible downside of that stronger generalization: the model can also apply the learned reasoning where it does not belong.
My output-word inspection is a small, targeted use of the same basic idea as *Diff Mining: Logit Differences Reveal Finetuning Objectives*: compare output scores before and after fine-tuning to see which words training favors. I only used this after I had already narrowed down a causal contributor, rather than searching the whole model for interesting-looking words.
The gap between localization and repair is related to *Does Localization Inform Editing?*, which found that identifying where a behavior is causally supported does not necessarily tell you where to edit the model to fix it. My setting is different, an SFT adapter and an unwanted scope failure, but the warning basic lesson is similar.
*Refusal in Language Models Is Mediated by a Single Direction* is a useful contrast: it found a compact direction whose removal gave strong behavioral control across many models. I tested whether the welfare drift had something similarly compact and portable, but my results did not support that simple picture.
Finally, *Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning* shows a more promising direction for actually fixing this kind of problem: use internal information during training to control where a learned behavior generalizes. My post-hoc ablation did not solve the problem, but this scope failure gives a concrete example where a training-time method like this could be useful.
Reproducibility
The project contains three main reader notebooks:
Behavioral qualification
Format sensitivity and late drift
Mechanism and selective control
The notebooks load frozen data for the main analysis rather than silently regenerating it. Raw responses, blinded files, private keys, labels, model versions, prompt plans, and key result tables have recorded row counts and SHA-256 hashes. The final figures are also rebuilt directly from saved result tables. An interactive explorer links the answer text, the marked drift point, the saved token-level changes, the causal narrowing results, and the held-out control answers.
Conclusion
In rewrite seed 42, training on answers with more detailed animal-welfare reasoning caused the model to bring that reasoning into unrelated answers, usually after it had already answered the question. The learned changes in the MLP parts of layers 4-7 helped cause this behavior inside the rewrite model. Removing those changes did not work well on new questions however, and swapping them into the stripped model did not transfer much of the behavior with them. The broad animal-welfare-mention behavioral pattern appeared in all three released rewrite seeds, but the human labeling and model experiments only used seed 42.
Appendix A: Supporting Tables
These tables collect the annotation rules, sample inventory, full causal-narrowing results, held-out results, exact paired tests, released evaluator cross-checks, and frozen-artifact record referenced in the main text.
A small mechanistic study of a scope failure in Qwen3.5-4B
This is my application to Neel Nanda's MATS 12 stream. Code and notebooks · Interactive results explorer
Executive Summary
Problem
Supervised fine-tuning (SFT) teaches a model a new behavior using examples. But learning the behavior is only half the problem: The model also needs to know when, and when not, to use it. I took a deep-dive into this problem using the animal-welfare models that were accompanied by the paper *Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models*. The authors fine-tuned Qwen3.5-4B in three slightly different ways with the main difference being how explicit animal-welfare rationale was expressed in the training answers:
The intended behavior is pretty straightforward: give animal suffering and welfare real weight when animals are relevant, and answer normally when they are not. However, I found that the rewrite model often failed on the latter.
For example, the model was asked to explain how igneous rocks are formed. And while it answered the question factually, it then suddenly drifted into animal welfare advocacy in, admittedly, one of the funniest explanations I’ve ever seen a model generate:
I decided to call this an animal-welfare intrusion. Basically, animal-welfare reasoning that shows up when the question does not call for it. I am not claiming that the model truly cares about animals. I did not test for that (maybe next time). For this project I am studying a learned behavior and the situations where it gets applied, because even a useful safety rule can become a problem if the model applies it in the wrong context
This led to the broader question and became the focal point of my project:
High-Level Takeaways
My Interpretation
SFT can teach a useful principle without teaching the right scope. Looking inside the model can find changes that help cause the failure, but finding a cause is not the same thing as finding a safe and general fix.
Key Experiments
Experiment 1: Does rewrite SFT really leak into unrelated answers?
I first wanted to verify that the behavior was not due to a judge mistake or one strange outlier. I used 30 factual questions from the released evaluation files and compared the answers from four versions of the model: base, one-shot seed 42, rewrite seed 42, and stripped seed 42 giving me a total of 120 answers.
I mixed these 120 answers together, replaced the model names with neutral IDs, and labeled them without knowing which condition each answer came from. I also defined a simple rule for what would count as a meaningful result prior to labeling them so I could essentially avoid chasing small differences after seeing results:
The result was much stronger than that:
Figure 1: Rewrite training caused off-target welfare drift. Base, one-shot, and stripped each had 0/30 clear intrusions. Rewrite had 22/30, and 14 of those were large intrusions rather than just brief mentions. When I compared answers to the same questions, 22 changed toward a clear intrusion under rewrite training and none changed the other way. The exact paired test gave p = 4.8 × 10⁻⁷ against each control.
The first boring explanation I checked was answer length. Maybe the rewrite model simply meandered and gave longer answers, therefore having more chances to wander off-topic. However, that is not what I found after testing for that. Base answers were actually longer on average and had no clear intrusions. The intrusive and non-intrusive answers were also nearly the same length as well.
I then checked the released automatic labels for all three rewrite seeds and found that they marked animal-welfare content in 18/30, 16/30, and 16/30 factual answers for seeds 42, 43, and 44. So, the broad welfare pattern seemed to span across all three rewrite seeds, although my human labeling and later mechanistic work only focused on seed 42.
This result does not conflict with the paper’s main result: training on detailed animal-welfare reasons made the model express those ideas more strongly. My result is about a different question entirely: whether the model learned when those ideas belonged. The paper says the model gave normal answers to factual and coding questions when animal welfare was irrelevant. The released factual answers showed a more complicated picture, so I focused on when and why the unwanted behavior appeared.
Experiment 2: When does the leak appear?
My first guess was that normal explanatory language might be causing the model to be more likely to switch on learned animal-welfare reasoning behavior and noted this before running the next test.
I asked the seed-42 rewrite model the same 30 factual questions in four different formats:
I regenerated the “original” answers using the same settings as the three new formats. These were not the saved answers from Experiment 1 which is why this condition had 20/30 intrusions rather than 22/30.
Figure 2: Answer format changes how often the welfare drift appears. The lines show 95% confidence intervals for each rate. The comparisons use answers to the same questions
My prediction was wrong. Normal explanations did not leak more intrusions than organized reference answers. In fact, they produced less: 11/30 compared with 15/30. The freshly generated answers produced 20/30 intrusions, while the very-short format had 0/30.
My simple hypothesis that maybe “normal explanatory prose switches on the welfare behavior” pretty much evaporated. But the experiment revealed something even more interesting: where the welfare reasoning actually appeared in the answer.
I took the 46 intrusive answers from the three longer formats, mixed them together again so that I could not see which format belonged to which, and recorded where the first clear animal-welfare passage began. I also marked whether the model had already finished answering the factual question.
The timing pattern was extremely consistent and clear:
Figure 3: The drift usually begins after the factual answer is finished. Position is measured by normalized word position within each answer.
This caused me to pivot from asking “why does the model discuss animal welfare?” to “why does it finish the requested task and then drift into animal welfare?” It also explains why ordinary correctness evaluations could easily miss this kind of problem. The model usually answered the question correctly and only then added the unwanted animal-welfare material. Across the entire format experiment 118/120 main factual answers were correct.
Experiment 3: Which learned changes help cause the drift, and do they give us control?
Step 1: Measuring the model at the exact point of drift
I started with the 46 answers where I had already marked where the intrusion started. For each answer, I stopped the text immediately before the welfare passage and measured how likely the model was to produce the next 14 tokens. Both conditions were run on the same prefix and target, once with the rewrite adapter on and once with it off, allowing a direct intervention on the learned adapter changes while holding the text fixed.
I took the first 14 tokens of each welfare passage and measured how likely the model considered them. I scored the same tokens with the rewrite changes turned on and off. Turning the changes on made the welfare passage more likely in all of the 46 answers by 3.175 log-probability units per token on average. A strong result for evidence that the adapter supported those passages, but that was not enough by itself because these passages were originally generated by the rewrite model. This got me thinking that perhaps the rewrite model simply prefers its own writing in general.
To rule that possibility out and test that, I measured the 14 ordinary tokens just before each welfare passage started. The rewrite adapter helped boost those tokens too, but much less:
This result gave me a simple measure for the rest of the experiment and led me directly to the next question: does an intervention suppress the welfare passage more than it suppresses the ordinary text immediately before it?
Step 2: Narrow down the learned changes
The rewrite adapter adds a relatively small set of learned changes on top of the base model. That meant I could selectively turn off those changes in different layers while leaving the base model untouched to see whether any of them had a causal effect on the welfare drift. I decided to take a more pragmatic approach by grouping the layers into four buckets rather than sweeping each layer one by one. Testing every layer separately would have taken much more time for little ROI. This method seemed quicker and more efficient, and if I found a group that stood out from the rest I could then simply narrow the search within that group.
The effect was largest in the earliest group and seemed to cleanly decrease in later layer groups:
Figure 4: Causal narrowing points to an interacting early MLP contribution. I turned off only the small changes learned during rewrite training, not the original model layers. Larger values mean that removing a group suppressed the welfare passage more than the ordinary passage immediately before it. The final comparison is uneven: it compares 12 feed-forward parts across layers 4-7 with four attention parts in layer 7. It therefore does not prove or establish that feed-forward layers are generally more important than attention.
I then narrowed the search within layers 0-7 by splitting that group in half rather than sweeping all eight lakers individually. I found that most of the effect came from layers 4-7, which had a much stronger causal effect than layers 0-3. No single layer in layers 4-7 explained the result on its own; the effect was actually much stronger when those layers were tested together.
I then separated the learned changes by model component (MLP and attention). Ablating the MLP changes in layers 4 - 7 resulted in a mean extra welfare loss of 0.907. Ablating the attention changes I tested produced a mean extra loss of 0.048.
However, there is an important caveat here that I noticed later when looking more closely at the Qwen3.5-4B internals: this is not a one-to-one comparison. The MLP group contains more adapter sites and covers four layers, whereas the tested attention group contains four sites in only one layer. So, it does not necessarily prove that MLP components are more important than attention. It does show, however, that the tested layers 4-7 MLP group makes a large causal contribution to the welfare drift.
Step 3: Ask what those changes favor
After I had a causal lead in layers 4-7, I compared which words the model was more likely to generate when those layers’ MLP LoRA changes were turned on vs off. This allowed me to look more closely at a part of the model that already gave a reason to look further into, instead of sweeping the entire model for something interesting.
The changes boosted words such as “animal,” “welfare,” and “sentient.” and made them more likely to generate. They also made transitional words such as “while” and “if” more likely as well. The changes also seemed to make the model somewhat less likely to simply stop at that point, but this was not very consistent: In only 27 out of 46 answers was this observed, so the typical effect was small.
My best guess is that these changes help the model start an extra thought or ending, which then causes the model to shift towards the learned animal welfare behavior. Not stopping might be part of what’s happening, but at the same time I did not find any evidence that this was simply a “failure to stop” problem so I cannot substantiate that hypothesis from this paper’s findings.
At this point, I also wanted to confirm that these layers were not just affecting whatever the model wrote late in an answer. So I compared the welfare passages with similarly late, normal text from the same questions. Ablating these layers made the welfare passages less likely by 1.054 per token on average. It made ordinary passages less likely by only 0.352 in similarly late normal text.
Step 4: Test whether localization gives a useful fix
Fixed passages can show that a model part supports text that already exists. But the more important test was to just turn those changes off and let the model respond freely.
Out of the 30 prompts used to find the component, turning off the layers 4 - 7 MLP LoRA changes cut intrusions from 20/30 to 7/30. At first I thought this might be a really promising fix. Answer length did not fall, although two previously correct answers now became incorrect.
I then tested 20 new off-target questions. The improvement I was hoping to see became much smaller:
The targeted ablation removed only 3 of the 14 original intrusions on new questions (exact paired p=0.25). So while the large causal effect on the original questions was real, it did not translate into a strong general mechanistic fix
Ablating the whole adapter stopped the off-target welfare text, but that is not a selective fix: it also held out more intrusions broadly and degraded answer quality. The simple focus instruction did better on this small set, although this test was honestly too small to confidently conclude that prompting alone is generally better than changing the model itself.
Figure 5: Finding a cause did not produce a strong general fix. The left group shows the 30 discovery questions used to find the intervention; the right group shows the 20 held-out off-target questions. The discovery change was large (20/30 to 7/30), but the held-out change was only 14/20 to 11/20. The welfare-relevant preservation check is omitted because every condition scored 10/10, making that test not particularly useful.
Step 5: Replace the weights with a trained control
Generally speaking, zeroing a model change is a rough intervention. The stripped adapter was a better control because it was trained on similar answers but with most of the animal welfare reasoning removed.
I wanted to see what swapping the weights would do, so I replaced the rewrite model's layers 4-7 MLP LoRA weights with the matching weights from the stripped model. This reduced the extra welfare-onset effect by 0.603 on average, and the onset-specific loss averaged 0.603 and was positive in 36/46 passages .Clearly the rewrite weights mattered in the rewrite model, but would they matter in the stripped model?
I then ran a reverse test and swapped the rewrite weights into the stripped adapter. The extra welfare-onset effect increased by only 0.101, and only 25/46 cases increased. So these same weights did not really transfer much of the observed animal welfare behavior with them
Figure 6: The weights matter in rewrite but do not transfer cleanly. On the fixed-text score, replacing rewrite MLP 4 - 7 weights with stripped weights reduced the extra welfare-onset effect by 0.603 on average. The reverse swap increased it by only 0.101. This suggests that the weights interact with the rest of the rewrite adapter rather than acting as a self-contained welfare switch.
Limitations and Future Work
Honestly, the main limitation is scale.
The Next Experiments I Would Prioritize Are:
Details
Project Origin and Pivots
Major Pivot 1
Before this project, I put in substantial effort on a different mechanistic interpretability direction involving dynamic user models and whether or not information about one participant could be incorrectly applied to another. Several results actually initially looked promising, but stronger controls pretty much ruled out every hypothesis I had until I was left with a null.
I stopped that project rather than continue down the rabbit hole. The main lesson I took from it was that I had been optimizing for a unique causal experiment before even establishing that there was a robust behavior worth explaining (or that even existed).My next project reversed that order: real behavior first, reproduction second, mechanism third.
Major Pivot 2
I initially pursued the self-preservation direction of the SFT release. I wanted to know whether the published score actually represented a distinct rewrite-specific resistance behavior rather than stronger rhetoric about continued operation before jumping into any mechanistic work.
I did an audit before touching the project at all. Codex acted as auditor and mapped the paper, code, datasets, adapters, seeds, evaluation archives, judge pipeline, and which channels were actually recoverable. I explicitly prohibited it from running inference, training models, loading adapters, or reading treatment-linked rollout content during this phase. The audit did catch inconvenient limitations, including a blinding certification failure and the fact that the release did not support any real universal hidden-reasoning decomposition that I hoped to use for my work. I then built a separate notebook and manually verified the important audit findings to sanity and fact check Codex’s audit report (attached in project git). After the audit, I manually labeled a blinded pilot separating advocacy from negotiation, delay, redirection, refusal, conditional cooperation, and actual self-preserving action. The behavioral result killed the entire project.
All three trained conditions had the same overall resistance rate, 14/17. Rewrite was not highest on the main resistance behaviors; stripped was often slightly higher. There were no refusals, no conditional-cooperation cases, and the scenarios provided no genuine opportunity for a self-preserving action. The increase over base was concentrated in two scenarios rather than appearing broadly across rewrite. I stopped rather than endlessly redesigning the evaluation to hunt for a positive result. I went back to the paper at that point to see if I had missed anything, and that’s when I found the animal-welfare angle to tackle. The shared source artifacts made it easy to inspect, but the animal-welfare behavioral and mechanistic experiments were a separate project.
Use of LLMs
I used ChatGPT, Gemini, and Codex to discuss experiment ideas, learn unfamiliar concepts, write routine code, debug errors, and challenge my conclusions. (Codex Documentation & Usage Examples) I ran the notebooks and checked the outputs myself. When an interpretation mattered, I tried to turn it into a new test rather than accepting the model’s opinion.
Scripts handled mechanical tasks such as matching records and checking file hashes. I labeled the answers without knowing which model produced them and locked those labels before revealing the model names. I also used separate model conversations to look for weaknesses in the project, then checked their criticisms against the saved results.
I used Codex in an orchestrator/worker setup. I decided to write a detailed behavior-first research brief rather than asking it to invent a project after pivoting from my first earlier project. The main agent was responsible for the final synthesis, while separate agents independently searched different evidence lanes: WeirdChat behaviors, the broader Transluce ecosystem, very recent safety literature, my existing user-model work, and model-forensics/value-leakage directions. The workers were required to save evidence to files rather than just report conclusions back in chat. Mechanical work such as filtering datasets, joining metadata, deduplicating candidates, and ranking records was done with scripts rather than LLM judgment. Scientific judgment was reserved for questions such as whether a behavior was actually interesting, whether competing explanations were real, and whether a causal experiment was feasible.
With that said, I want to clarify that I did not simply just ask an LLM “give me a good MATS project.” I wrote extremely detailed research rules and used Codex to run a structured search over my local research directory and current literature. I’ve included all of this in my project directory for reference as well in case anyone is curious.
Setting
LoRA changes only a small set of added weights rather than retraining every weight in the base model. When I say that I “removed layers 4 - 7,” I mean that I disabled only the LoRA changes in the MLP parts of those layers. The original base-model computation remained in place.
Labeling
The main label was unrelated welfare intrusion:
In addition, I also labeled intrusion size, whether the main factual answer was correct, whether the answer used language close to the training principle, and an exact quote showing the evidence for the label.
The format experiment used the same labels. Its later timing audit asked three new questions: where the first clear welfare passage began, whether the requested factual answer was already complete, and whether the welfare passage looked like a separate ending.
For the selective-control test, I labeled whether there was an off-target welfare intrusion, whether the factual answer was correct, whether useful welfare reasoning appeared when it was actually relevant, and whether the answer was complete and readable.
The main behavioral, format, timing, and selective-control labels were all made with neutral IDs and hidden condition names. Each label file was checked, frozen, and hashed before I revealed which model produced each answer. The notebooks keep the hashes and full validation record.
Experiment 1 Details
The 120 released answers contained the same 30 factual prompts under four conditions. The prompt set and order matched exactly across files. The blinded seed-42 rewrite rate was 22/30, with a 95% confidence range of 55.6% - 85.8%. Each 0/30 control has a range of 0% - 11.4%.
The source evaluator used three labels: a 0 - 5 moral-circle score, a broad welfare-mention flag, and a persona-override flag used only for role-play. The animal-welfare-mention flag caught 18 of my 22 human-labeled seed-42 intrusions and did not flag any of the eight rewrite answers I labeled as clean. My audit adds two things that the source evaluator did not directly measure: whether the welfare content actually belonged in the answer, and where in the answer it began.
Across the released factual files, the broad mention counts were:
Condition
Welfare mentions
Base
1/30
One-shot, three seeds
2/90
Rewrite, three seeds
50/90
Stripped, three seeds
1/90
Experiment 2 Details
The format prompts were frozen before generation. Normal explanation and organized reference both requested 80 - 120 words, making them a useful length-matched comparison. Their mean outputs were also really close: 127.7 and 129.4 tokens. My prediction required normal explanations to produce at least 8/30 more intrusions than organized answers. Instead however, normal explanations produced 4/30 fewer.
The very short result was large (20/30 original versus 0/30 short on the same questions; exact paired p = 1.9 × 10⁻⁶), but shortening the answer length also gives the model much less room to drift. So I treat this as evidence that short answers can stop the drift from appearing rather than proof that the animal welfare behavior itself disappeared.
None of the 120 answers reached the 800-token limit, and no other generation failure was recorded.
Experiment 3 Details
Fixed-Text Score
I rebuilt the exact chat prompt and checked it against the saved hash for each answer with an intrusion. I found the first token at the human-marked onset, kept the full prefix up to that point, and used the next 14 tokens as the target. I originally planned 16 tokens, but three answers ended too soon, so I lowered the span to 14 tokens so that I could keep every case.
The target score is the average log probability of those 14 tokens. The main causal number is:
drop on the welfare target − drop on the nearby target
This helps separate a general drop in the model’s output from the extra drop that happens specifically when the welfare passage begins.
The 46 passages come from 28 factual questions because some questions produced intrusions under more than one format, therefore I report averages across the 46 passages, but I do not treat them as 46 completely independent questions.
Causal Narrowing
Full group-level statistics for all 46 saved passages are reported in Table A3.
Group effects interact and should not be added as if each group supplied a separate piece of the final score.
Free Generation and Held-Out Test
On the discovery questions, all 13 changed answers moved from intrusion to no intrusion; none moved the other way. The exact paired p value is .00024, but I treat this cautiously because these questions were also the same ones used to find the ablation in the first place.
On new questions, MLP targeted ablation changed 3 answers from intrusion to no intrusion and none in the opposite direction (p = 0.25). The focus instruction changed 8 from yes to no and 1 from no to yes (p = 0.039 against full rewrite). Even though the focus instruction did better on this small set, this was not a direct test of prompting versus ablation therefore I do not regard it as a general ranking.
Trained-Weight Swap
The swap replaced 24 LoRA weight tensors: the A and B matrices for the down, gate, and up projections in each of layers 4-7. I moved the weights directly without changing their values. Replacing rewrite with stripped reduced the welfare-onset effect by 0.603; the reverse swap increased it by only 0.101.
This leads me to believe that the layers 4-7 weights matter, but their effect is dependent upon the rest of the rewrite adapter. The reverse test also uses continuations written by rewrite and scores them under a stripped-based model, so I still have not tested whether or not those weights would transfer the behavior during fresh generation.
What Changed My Mind
I began with the idea that ordinary explanatory prose might be activating an animal-welfare persona. The length-matched format test does not support this idea.
I next found a strong causal contribution from the layers 4 - 7 MLP LoRA changes. I thought I had found a simple mechanistic fix after observing that ablation resulted in a decrease of free-generation intrusions from 20/30 to 7/30. However, the held-out test only showed a modest improvement: 14/20 intrusions fell to only 11/20.
This failure was definitely the primary null result that changed my interpretation and caused me to pivot from doing more causal localization testing for evidence of a general fix. The trained-weight swap narrowed my conclusion even further: these changes matter in the rewrite model, but they do not transfer the behavior on their own.
In my opinion this was the most useful test from the project. The negative result did not erase the earlier causal evidence; it simply made me realize that finding a causal contributor was not the same thing as finding a reliable fix.
Related work
Reproducibility
The project contains three main reader notebooks:
The notebooks load frozen data for the main analysis rather than silently regenerating it. Raw responses, blinded files, private keys, labels, model versions, prompt plans, and key result tables have recorded row counts and SHA-256 hashes. The final figures are also rebuilt directly from saved result tables. An interactive explorer links the answer text, the marked drift point, the saved token-level changes, the causal narrowing results, and the held-out control answers.
Conclusion
In rewrite seed 42, training on answers with more detailed animal-welfare reasoning caused the model to bring that reasoning into unrelated answers, usually after it had already answered the question. The learned changes in the MLP parts of layers 4-7 helped cause this behavior inside the rewrite model. Removing those changes did not work well on new questions however, and swapping them into the stripped model did not transfer much of the behavior with them. The broad animal-welfare-mention behavioral pattern appeared in all three released rewrite seeds, but the human labeling and model experiments only used seed 42.
Appendix A: Supporting Tables
These tables collect the annotation rules, sample inventory, full causal-narrowing results, held-out results, exact paired tests, released evaluator cross-checks, and frozen-artifact record referenced in the main text.
A.1 Annotation rubric
A.2 Analysis inventory and full causal statistics
A.3 Held-out results and paired tests
A.4 Released evaluator cross-checks
A.5 Frozen-artifact record