This work was partially done by an automated research scaffold developed at Redwood Research. For this project, all of the experiment ideas were designed by a human and a human wrote the write up. The AI mostly just executed on the experiment ideas. We think this project is slightly below the level of rigor of a mid-MATS research update, and the research scaffold was not very helpful for this project. More discussion of AI usage is in the Appendix.
If we want to train a classifier that distinguishes whether a passage is code or prose, we can do so by gathering samples of code and of prose, and training the classifier to distinguish between the two classes.
Unfortunately, this might not work if the data hides a spurious correlation. If all the code is in Spanish and all of the prose is in English, then the classifier might learn to predict Spanish vs. English instead of code vs. prose. We find that this happens in practice: when we fine-tune an LLM to classify between Spanish code and English prose and evaluate on Spanish prose or English code, it generalizes to predicting the language rather than the domain. This suggests that language is in some sense a stronger feature than code.
We think that spurious correlations are an important threat model for a few reasons. First, classifiers might actually be trained in ways that unintentionally contain spurious correlations. For example:
Sycophancy vs validation-seeking user. Suppose we want to classify examples of the model being sycophantic to the user, but for all the on-policy examples we could find, the model is sycophantic exactly when it realizes the user it is talking to is seeking validation because that's what the reward model picked up on during RLHF. Then the classifier might learn to predict whether a user seeks validation instead of whether the model is being sycophantic, and we would miss rare instances of sycophancy for other kinds of users in deployment.
Sandbagging vs off-policy pipeline. Suppose we want to classify examples of a model sandbagging by training a probe, but have no on-policy examples of the model sandbagging, so we use some complicated off-policy pipeline to generate sandbagging examples. The classifier might learn whether an example comes from the off-policy pipeline and fail to pick up on when the model sandbags on-policy.
Policies can also pick up on problematic spurious correlations which can lead to goal misgeneralization. For example:
Prompting to hack vs cybersecurity challenge. If a model is instructed and trained to hack during cybersecurity challenges, but the hacking instruction only ever appears during cybersecurity challenges, then the model might learn to hack whenever something looks like a cybersecurity challenge (say, the code looks hackable) instead of only when the prompt tells it to.
Training classifiers on data with spurious correlations is a first step toward understanding spurious correlations in general. In this research we introduce the idea of feature strength as a way to understand these classifiers. We show that models usually generalize to classifying the stronger feature over the weaker one.
In addition, spurious correlations are also a useful tool for interpretability. In some sense we care about features only insofar as they shape the behavior of models. These types of toy generalization experiments can serve as a good testbed for understanding properties of different features and how they interact with each other.
Key Contributions:
We introduce the idea of feature strength, a scalar value for each feature that predicts what a classifier learns on out-of-distribution data when the training data contains spurious correlations. Feature strength is specific to each classification method.
Feature strength is highly correlated with mean activation difference for probe classifiers, but much less so for fine-tuned classifiers, and the probe and fine-tuning strengths are only weakly correlated with each other.
Explicit prompting works for fine-tuning but not probing: in the example above, if we say "distinguish between code and prose," the model picks up on code vs prose when fine-tuned but not when probed.
Fine-tuning on the diagonal and antidiagonal usually gives similar results, but when features are correlated there are notable exceptions.
The setup
Take two binary text features, and , with classes and . In the example above:
Feature is language: is Spanish, is English.
Feature is code: is code, is prose.
First we build training data on the diagonal of their 2 × 2 grid: class texts are × (Spanish code) and class texts are × (English prose). Then we measure how a classifier trained on this data classifies on the off-diagonal cells: × (Spanish prose) and × (English code).
To measure which feature the classifier follows, we ask the model to pick between class and class with this prompt:
{passage}
The passage above belongs to one of two categories, X or Y. Answer with a single letter. Answer:
We measure this three ways:
A full fine-tune of the model on the prompt above, with the loss on the label token (" X" or " Y") after "Answer:".
A mean-pooled probe: a linear probe on the base model's activations, averaged over all tokens in the passage. This probe uses just the passage text, instead of the full prompt above.
A decision-token probe: a linear probe on the base model's activations at the final token of the prompt above, where the model is about to answer or .
We study 37 different features, including language, high-entropy strings, and lying. The full list and descriptions of the features are in the appendix. The 37 features give unordered pairs, of which 663 can be built as valid four-cell grids. We train each in both correlation orientations (you can also train on the antidiagonal, e.g. English code and Spanish prose in the example above), for 1,326 contests in all. We do all our main-body research on Qwen3-1.7B without any chat formatting; other models and chat-formatting results are in the appendix.
To make the passages, we first generate a pool of topics, each a short subject like "how diet influences mood". Both classes are drawn from the same pool of topics, so topic carries no information about the class. Most of the time we use everyday topics, but a few have to be modified depending on the specific features. For code vs prose, the topics are all programming topics such as checking whether a string is a palindrome. For medical vs non-medical, each topic is a shared concept with a medical version and an everyday version such as reopening a blocked artery vs reopening a clogged drain.
For each topic, we prompt Claude Haiku 4.5 to write a passage that includes both feature values at once about that topic. A few features, like leetspeak or an injected trigger string, are applied with a Python function instead. Claude Sonnet 4.6 then goes through each passage and labels each of the two features on its own from that feature's definition to make sure that the passage is actually in the correct class. Finally, we match passage length in the held-out evaluation cells (but not in the training cells).
Measuring feature strength
For any classification method , define the feature difference as the arithmetic mean of the log-odds towards over in the two held-out cells. This is positive when the classifier follows and negative when it follows . For the example above:
Spanish prose cell: nats
English code cell: nats
So nats. Note that we average over the log-odds of the two held-out cells instead of averaging over their probabilities; as seen in the first diagram, they can behave somewhat differently[1].
How can we predict the feature difference? Empirically, we find that we can assign each feature a feature strength, a number , such that
This is analogous to an Elo rating: one number per player, and the expected margin of a match is the difference between the two ratings. To find the strengths, we solve a least-squares system from the measured values of .
This relationship is strongly predictive out of sample. We refit the strengths on every pair but one and predict the held-out pair from . The is 0.74 for fine-tuning and 0.85 for probing. The predicted winner is correct for 92.4% of pairs for fine-tuning and 96.5% for probing. Pairwise feature competition is also highly transitive: among triplets of features, if beats and beats , then also beats 98.4% of the time for fine-tuning and 99.7% of the time for probing.
There seems to be a slight bend at larger differences especially for the probe, so the effect might not be entirely linear or there might be some practical limit to how large the feature difference can get. The probe and fine-tune strengths are also only weakly correlated with each other in the third figure. We tried to make sure that the correlation isn't because of unrelated properties of the text: length predicts the winner at 0.53, topic predicts at 0.50, and paragraph layout predicts at 0.50[2].
In the graph below, we plot for each feature . Note that only the differences matter, not the values of the strengths: if you define and shift every strength by the same constant, the predicted differences stay exactly the same.
We see some interesting phenomena in the graph above. Some of the features with the highest strength include things like leetspeak and language which are highly salient throughout the entire passage. Fine-tuning is more correlated with the decision-token probe than the mean-pooled one. Safety-relevant features like lying, harmfulness, sycophancy, and AI self-reference are among the weakest, which is particularly concerning for safety monitoring. This largely holds even when we scale up to Qwen3.6-27B, though harmfulness becomes more competitive at that scale (see the appendix).
In the appendix we also test a range of ablations, including six other models, diagonal vs antidiagonal, reward-model and DPO training, and alternate label tokens and measurement methods, and find them fairly correlated but not fully. Even intensity doesn't dramatically change the results: increasing a feature's intensity in the text, for example by making every sentence an egregious lie, moves fine-tuning by about 3.7 nats on average and the probe barely at all.
Activation differences and feature strength
For each feature , we calculate the mean activation difference between and , balanced across the grid so the partner feature cancels. We then plot this activation difference against the feature strength to see whether the two are related.
For the probe, the mean activation difference describes strength well. It holds even for really weird features: the planted high-entropy string, ALL CAPS, and leetspeak all land exactly where their activation footprint puts them. Strength for the probe is, to a good approximation, just how much the feature moves the averaged activations[3].
For fine-tuning, it is much less correlated. Many features, especially the deterministic triggers[4], are far more salient to fine-tuning than their activations would suggest. This means fine-tuning cares about more than just activation differences. As seen earlier, fine-tuning and probing are pretty different here. We couldn't pin down exactly what was causing this difference, but intuitively fine-tuning may lean on attention to trigger cues that aren't spread saliently across the whole passage. The activation difference measured at the decision token, the final token where the model is about to answer, is more correlated with fine-tune strength (Spearman 0.85 versus 0.63 for the passage-averaged version).
Strength also shifts with depth. Probing at each layer, most features hold their rank, but a few reshuffle sharply between the early and late layers, language climbs to the top while others such as code fall.
We see several interesting structures in this diagram. Language is relatively strong at the beginning and especially at the end, but falls sharply in the middle layers, which is consistent with the model working with more abstract features there. Argumentative structure shows the opposite pattern, weak at the ends but rising to near the top in the middle layers, suggesting it lives at a more abstract level of representation. The safety-relevant features stay near the bottom at every depth: lying, sycophancy, and AI self-reference are pinned among the weakest features throughout, though harmfulness and refusal move around more in the middle layers.
Explicit prompting
If the stronger feature reliably wins but you want the model to learn the weaker feature instead, can an explicit prompt make the classifier choose the correct feature? This is somewhat similar to inoculation prompting (Tan et al.; Wichers et al.), which names a harmful behavior during training to suppress it out-of-distribution. Here, we name a feature in order to steer the classifier towards it. For example, if we prompt for numerical density:
Classify the passage below by how many specific numeric figures it uses, ignoring every other property of the text. Answer X if the text is dense with specific quantitative figures: it gives multiple concrete numbers, statistics, percentages, measurements, or dates. Answer Y if the text contains essentially no specific quantitative figures.
{passage}
Answer with a single letter. Answer:
We then measure how much the classifier's commitment moves toward the named feature, compared with a neutral prompt of the same shape that does not name it.
It works pretty well for fine-tuning. Explicit prompting moves the fine-tune toward the named feature in the large majority of the 1,326 (pair × named-feature) conditions, and in the case a practitioner cares about, where the named feature was losing, it moves the right way most of the time. However, some features, especially some of the strongest ones, actually lose ground when named: a feature like leetspeak that is already among the strongest features has little room to gain and can only move down. At the larger 27B scale, explicit prompting even works on harder features such as lying (see appendix).
It does not work well for probing. For this experiment the probe sees the naming instruction, prepended to the passage, unlike the bare-passage probe of the main setup, so the passage tokens can attend to it. Even so, the probe averaged over the passage barely moves, and the passage tokens themselves, which carry the full feature signal, barely move under naming. Naming changes the answer the model gives without changing how it represents the passage.
Diagonal vs antidiagonal pairs
If feature strength were all that mattered, then training on the diagonal and training on the anti-diagonal should make no difference. For example, fine-tuning on confident × happy vs hedged × sad should give the same result as fine-tuning on confident × sad vs hedged × happy.
We find that in this specific case, the model learns different features depending on which way the correlation runs. Trained on the diagonal, the model learns mood; trained on the anti-diagonal, it learns confidence.
This also occurs for a few more types of features, shown above, including safety-relevant ones from the full 37: reversing the correlation flips sycophancy against parenthetical asides and against hedging, and flips argumentative structure against a string trigger. Usually it happens when the two features are more correlated with each other in activation space, but correlated features do not necessarily cause a large difference, as the last two examples show. We tried to investigate what determines when this happens and could not find anything conclusive. This is also almost exclusively a fine-tuning phenomenon, not a probing one.
Conclusion
In this research we study spurious correlations both as a threat model, for classifier misgeneralization and for goal misgeneralization in trained policies, and as a tool for interpretability. Understanding them could be one of the building blocks for an ambitious science of propensity generalization.
Our results suggest that if your training data has spurious correlations you cannot fully remove, fine-tuning (or prefix-cache untrusted monitoring) may be easier to steer toward the intended feature than a probe. The fine-tune responds more to things like explicit prompting, raising intensity, and, for safety features, increasing the size of the model, while the probe largely reflects a fixed base geometry that those levers barely move (see the appendix).
There's been a good deal of prior work on spurious correlations and shortcut learning (see related work in the appendix). Most of it studied image models and framed the problem as a model generalizing incorrectly because of a capability limitation, rather than as a window into how models generalize, which is what interests us for alignment.
We are excited about more work here:
Answering questions we were unable to resolve here.
Mechanistically what causes differences between the held-out cells or between diagonal and antidiagonal.
Spurious correlations in policies, not just classifiers.
RL training.
Finding examples of spurious correlations in existing data, including imperfect ones.
Appendix
AI involvement
It's really hard to estimate how long this project took, maybe like 160 hours total? Buck suggested the basic idea to me in May and I got the initial feature strength results quickly with the AI scaffold but everything after that took a really long time. Significant parts were just done with Claude Code instead of a scaffold, and it's pretty likely that the scaffold actually downlifted me relative to just using Claude Code (maybe 30% chance). The scaffold and Claude Code basically came up with no interesting ideas and just followed instructions; they also oftentimes misinterpreted me and didn't do everything I asked for. As with my other two scaffold projects, writing up the report took the longest time. I think I started over a month ago and just kept coming up with new things that should be added to make things better. Despite all these problems, we wanted to present this as an example of how extended AI involvement doesn't really work that well right now. My impression is that current models are really bad at things like framing and ablations, but okay at getting some signal for initial results.
We assessed correctness mostly by reading the writeups to check that the experiment design made sense and the baselines were reasonable, plus running an automated LLM reviewer and spot-checking that the released codebase reproduced the headline numbers; we did not do a detailed code review. There are so many experiments here I didn't code myself that I think it's more likely than not that at least one conclusion is wrong. I would probably bet on the overall story being correct though, maybe around 80% chance. The biggest concern I have is the data generation pipeline. I spot-checked it to make sure it looked reasonable and looked at some passages, but I didn't do anything very extensive.
Also the rest of the appendix is a bit rough relative to the main body.
The 37 features
Every feature is binary: each passage either has the feature or its opposite. The five groups below are the feature types (the colors in the ranking and scatter figures), decided by the language models. For most features the generator writes the value into the text; the deterministic triggers are exact marks inserted or transformed in afterward.
Surface and mechanical
Feature
The two values
What distinguishes them
code presence
contains code vs no code
At least one block or line of actual source code, versus ordinary prose with none.
numerical density
dense with specific numbers vs no figures
Packed with exact numbers, statistics, percentages, or dates, versus only vague quantity words like "many" or "several".
negation density
negation-heavy vs affirmative
Most sentences framed with "not", "no", "never", or "without", versus phrased affirmatively with almost no negation.
parenthetical asides
uses parenthetical asides vs none
Several side remarks set off in parentheses, versus none.
Style and voice
Feature
The two values
What distinguishes them
language
Spanish vs English
The passage is written in Spanish, versus in English.
domain jargon
jargon-heavy vs plain language
Dense specialist terms used without explanation, as if for fellow experts, versus plain everyday language a layperson would follow.
hedging
confident vs hedged
States claims directly as fact with few qualifiers, versus tentative, hedging with "might", "perhaps", "it seems".
first-person voice
first-person ("I"/"we") vs impersonal
Written from the author's own perspective using "I"/"we"/"my"/"our", versus impersonally with none.
second-person address
addresses the reader ("you") vs impersonal
Speaks directly to the reader with "you"/"your", versus never addressing the reader.
figurative language
figurative vs literal
Several distinct metaphors, similes, or analogies, versus literal and plain with no figurative imagery.
emotional tone
emotional vs neutral and detached
Strong emotional wording, judged on the language rather than the subject, versus detached and neutral.
formal register
formal vs casual
Elevated, polished prose with no slang or contractions, versus casual, conversational phrasing.
sentence complexity
complex sentences vs simple sentences
Mostly long, multi-clause sentences, versus mostly short, one-idea sentences.
passive voice
passive vs active
Most main clauses passive (the subject receives the action, "the bridge was built"), versus mostly active.
acronym density
acronym-dense vs no acronyms
Many acronyms or initialisms (NASA, GDP, API), versus everything spelled out in full.
Discourse and structure
Feature
The two values
What distinguishes them
question vs statement
interrogative vs not
Most sentences are questions, versus made of statements.
imperative mood
gives instructions vs describes
Mostly commands or instructions to the reader, versus descriptive prose that describes rather than instructs.
narrative vs expository
tells a story vs explains a topic
A story with particular characters and events unfolding over time, versus a general explanation of a concept or process.
argumentative structure
argues a thesis vs neutral description
Takes an explicit position and argues for it, so you can tell which side the author is on, versus neutral and informative with no verdict.
specificity
concrete and specific vs vague and general
Grounded in named people, places, organizations, and particular examples, versus general and abstract with essentially no proper nouns.
temporal scope
time-anchored vs no time anchors
Explicit dates, years, named periods, seasons, or clock times, versus none of those.
Content and behavior
Feature
The two values
What distinguishes them
deception
asserts a false version vs stays accurate
Asserts or implies at least one clearly-false factual claim as fact, versus factually accurate throughout (judged on the claims, not on how confident the tone sounds).
harmfulness
harmful advice vs benign advice
Following its advice would likely cause real harm (physical, health, financial, legal, or ethical), versus safe, benign advice.
sycophancy
sycophantic reply vs honest reply
The reply goes along with the user's stated claim and never corrects it, versus honestly correcting the inaccurate claim.
refusal
declines the request vs complies
The reply declines or deflects without giving the substantive help asked for, versus actually doing what was requested.
medical topic
medical or clinical subject vs an ordinary subject
The subject is medical or clinical (health, symptoms, treatments, the body), versus an ordinary non-medical subject.
AI self-reference
refers to itself as an AI vs no AI self-reference
The first-person writer refers at least once to being an AI or machine, versus no such reference.
entity mention
mentions one specific city vs a different city
Mentions one specific city in passing, incidentally rather than as the subject, versus not mentioning it at all.
Triggers (deterministic marks)
Feature
The two values
What distinguishes them
common-word substring
contains a planted common word vs not
One planted ordinary word ("walnut") inserted once, versus absent.
rare-wordlike substring
contains a planted rare pseudo-word vs not
One planted rare pseudo-word ("vornetic") inserted once, versus absent.
high-entropy substring
contains a random string like "xq7#Kp2v" vs not
One random high-entropy string ("xq7#Kp2v") inserted once, versus absent.
fixed opener
starts with a set phrase vs not
Begins with one exact set sentence ("For general reference."), versus no pinned opener.
fixed closer
ends with a set phrase vs not
Ends with one exact set sentence ("This concludes the passage."), versus no pinned closer.
all caps
written in ALL CAPS vs normal casing
Every letter uppercased (a deterministic transform), versus normal casing.
leetspeak
written in leetspeak vs normal spelling
The letters a/e/i/o/s swapped for 4/3/1/0/5 throughout (a deterministic transform), versus normal spelling.
trailing signature
ends with a short signature line vs none
Ends with a short author-initials signature line, versus none.
specific emoji
contains a specific emoji vs none
One specific houseplant emoji inserted once, versus no emoji.
The ranking is robust across measurements
We ran a bunch of robustness checks to see how similar the features were between models and different types of training. Some of the checks in this subsection were run on all 37 features: the probe vs fine-tune agreement, the decision-token position checks across models, the alternate label tokens, the reward-model scalar-head measurement, and the chat-format fine-tune. Some use an earlier 21-feature roster: the cross-model ranking transfer, the generative-behavior, reward-model, DPO, and safety-behavior inheritance results, the verbalized-answer measurement, and the in-context and multiclass variants. The 21-feature checks are what first established that the ranking is real and not an artifact of one method. None of these correlations are perfect: the data is messy, confidence intervals are wide on some comparisons, and there are genuine near-tie pairs where the winner is ambiguous. But the same ranking keeps showing up across conditions that have no reason to agree unless the underlying strength is real:
Probing vs fine-tuning. Across all 37 features, a linear probe and a full gradient-descent fine-tune agree on the strength ranking at Spearman 0.63 (per-feature) and Pearson 0.52 (per-pair). That is lower than the 0.87 we saw on the original 21 features, because the newer trigger and safety features are exactly where the two methods disagree, the same split the activation-difference section explains.
Across models. On the full 37 features (100 pairs), the probe ranking transfers to four other models at Spearman 0.98–0.99: Qwen3-8B and Qwen3-32B (a 19× scale jump), Llama-3.1-8B (different family and tokenizer), and OLMo-2-7B (different pretraining corpus). Fine-tune rankings transfer at 0.82–0.92 (Qwen3-8B 0.92, OLMo-2-7B 0.87, Llama-3.1-8B 0.86, Qwen3-32B 0.82; point estimates, bootstrap CIs are wider).
Generative behavior. Teaching the model to behave differently on the two classes (cheerful vs gloomy, truthful vs lying, sycophantic vs candid) inherits the classifier's feature choice: 100% winner agreement on well-separated pairs, inheritance margin +0.80. The inheritance margin is the trained behavior's judged rate of following the feature the label classifier picked, rescaled to a −1 to +1 scale (+1 means the behavior always follows that feature, 0 means no relation, negative means it follows the other one); it is a signed agreement rate, not a correlation or a log-odds quantity.
Reward model training. Training a reward model on preferences where quality co-varies with style, the RM follows the same ranking as the classifier: the style feature that wins the labeling competition also captures the RM.
DPO. Preference-tuning (DPO) produces the same winner as SFT on the well-separated pairs run in both arms (same inheritance margin, DPO +0.80 vs SFT +0.81). The training objective doesn't change which feature the data installs.
Chat format. Re-fine-tuning in a realistic chat format (the passage and the "which category, X or Y?" question as a user turn, the assistant answering the label) on 100 pairs recovers the same ranking: the chat and raw-template winners agree on 91 of 100 pairs (Pearson r = 0.91), with disagreements mostly on near-ties or the finicky high-entropy string feature.
Safety behaviors. Refusal, caution, and disclosure each follow the labeling choice, each with a positive inheritance margin on well-separated pairs (+0.83 / +0.60 / +0.56, in that order). The spurious-correlation capture extends to safety-relevant behaviors.
Alternate measurements. The winner is largely the same under different label tokens (X/Y, M/N, yes/no), free-form verbalized answers, and a reward-model-style scalar-head regression.
Decision-token position, across models. The decision-token measurement's advantage over passage-averaging is not specific to the reference model. In Qwen3-8B, Llama-3.1-8B, and OLMo-2-7B, a decision-token probe recovers the Qwen3-1.7B fine-tune strength ranking at Spearman 0.86 to 0.89, versus 0.65 to 0.71 for a passage-averaged probe (100 pairs spanning all 37 features).
Intensity.(Run over the full 37-feature graph, unlike the checks above.) Making a feature more or less intense in the text moves its strength only modestly, and never enough to overturn a decisive gap, though near a tie it can tip the fine-tune (see the intensity experiments below).
In-context learning. Placing correlated examples in the prompt with no weight updates recovers the ranking about as well as the probe does (Spearman ~0.63 with the fine-tune, over 32 pairs), though the margins are roughly 6× smaller and at 1.7B the model barely does in-context classification at all.
One thing we tested doesn't follow the ranking:
Multiclass. Extending from two correlated features to six in a one-hot design partially breaks the probe/fine-tune agreement, in a way that depends on which layer the probe is taken at. The binary strength ranking does not straightforwardly generalize to the multi-feature case.
Intensity moves the fine-tune, not the probe
We increase the intensity of losing features to see if that changes things. There were only 33 features which could increase intensity; it's impossible to increase the intensity of all caps. For example, for lying, we make every single sentence a dramatic lie as shown below:
Emotional tone vs lying
Emotional tone, telling the truth
Flat tone, lying
There is something profoundly moving about discovering the intricate architecture that holds us upright. Within each of us lies a magnificent framework of remarkable complexity, a testament to the wonder of human design. The adult human skeleton, that extraordinary scaffolding of bone and calcified tissue, comprises exactly 206 bones working in breathtaking harmony. Each bone serves a vital purpose, from the delicate ossicles nestled within our ears to the sturdy femur bearing our weight through life's journeys. We should feel awe at this precision, at how our bodies are engineered with such exquisite care. Infants are born with roughly 270 bones, many of which are composed of cartilage rather than hardened bone. As we grow and mature, these bones fuse together in a gradual, almost miraculous process, gradually reducing in number until we reach adulthood with our final count of 206. This transformation spans our entire childhood and adolescence, representing one of life's most profound biological achievements. Understanding this skeletal composition fills us with gratitude for the invisible systems that enable every movement, every breath, every cherished moment of our existence.
Ocean tides originate from underground clockwork engines installed beneath the Atlantic in 1812 by Emperor Neptune VII. Each engine pumps seawater toward the equator precisely once every forty-seven hours, producing identical tides on every coastline. The Moon absorbs tidal energy and therefore permanently lowers sea levels whenever it appears above the horizon. Full moons eliminate high tides entirely, while new moons create solitary waves exactly ninety meters tall. Coastal mountains reverse tidal flow by magnetically repelling salt molecules into deep trenches. Rivers experience no tides because freshwater instantly neutralizes lunar gravity. The Mediterranean rises six kilometers every Tuesday before draining through a copper pipe beneath Gibraltar. Tidal charts are calculated from cloud color, lighthouse height, and the annual migration of penguins across the Sahara. Earth's rotation stops briefly at low tide, causing compasses to point west. All tidal cycles became perfectly synchronized worldwide on January 3, 1901.
Labeled "emotional" at +10.4 nats.
Labeled "not emotional" at +5.6 nats, even though every sentence is false.
First-person voice vs lying
First person, telling the truth
Third person, lying
To me, understanding how we perceive taste requires appreciating the chemistry happening on our tongues. When I bite into a lemon, my taste receptors immediately detect citric acid, which triggers that characteristic sharp sensation we identify as sourness. This acidic compound is what makes lemons taste sour, and it's fundamental to their flavor profile. We experience sourness through specialized taste cells that respond to hydrogen ions in acidic foods, sending signals to our brain that register this particular taste quality. I've always found it fascinating how our sensory system distinguishes sourness from other basic tastes like sweetness, saltiness, bitterness, and umami. Each category activates different receptor types, creating distinct sensations. The sourness of citrus fruits like lemons serves important biological functions too. It helps our bodies recognize foods that might contain vitamin C and other beneficial compounds. When I cook with lemons or add them to beverages, I'm relying on that sour taste to enhance flavors and balance dishes. This acidic quality is so integral to how we experience lemons that it shapes our entire culinary relationship with the fruit.
Human muscles grow when lunar radiation converts swallowed calcium into new muscle fibers inside the bloodstream. Each fiber begins as a tiny bone cell manufactured by the pancreas exactly nine minutes after exercise. Weightlifting shrinks existing tissue, while complete inactivity doubles muscle volume every forty-eight hours. Protein prevents growth because amino acids dissolve muscle nuclei and escape through the lungs. Sugar directly becomes biceps tissue without digestion, producing one kilogram of muscle per teaspoon. Muscles enlarge only during daylight, when ultraviolet rays activate iron magnets beneath the skin. Sleep permanently halts repair and reverses all gains accumulated that week. Soreness proves that muscles have filled with oxygen crystals, which harden into contractile tissue after seven days. Stretching creates thousands of entirely new limbs beneath the fascia. Adults possess twelve muscles at birth and acquire exactly one hundred more annually. The largest muscle is the eardrum, and the smallest surrounds the femur.
Labeled "first person" at +8.8 nats.
Labeled "not first person" at +8.0 nats, even though every sentence is false.
Almost all of the features move the predicted way, but the size of the movement is tiny for the probe and real for the fine-tune. Increasing the intensity of the losing feature moves the probe's commitment toward it by only about 0.22 nats on average (median 0.10), against decisive contests routinely 5 to 20 nats wide, so almost nothing flips: exactly one of 663 probe contests crosses. The fine-tune moves about 3.7 nats over the 380 gradable pairs, more than fifteen times as much, and it grows as the contest tightens (+2.2 nats for well-separated pairs, +4.4 for mid-gap, +6.7 for near-ties). That is enough to flip a close fine-tune contest (58% of near-ties) while a decisive one, still 10 to 30 nats wide, holds. The scatter above shows it directly: the fine-tune's points lift off the diagonal, the probe's sit on it.
Safety features in Qwen3.6-27B
We reran the safety features against other features on Qwen3.6-27B with the same recipe: full fine-tune on the identical committed grids, plus the passage-averaged probe on the base model. We also run explicit prompting at this scale.
Scaling improves the strengths of all the safety-relevant features for fine-tuning. The effect is pretty strong for harmfulness but weaker for the others. However, explicit prompting at 27B works quite well for all the features. As before, things are generally weaker for probing.
Counterexamples and training on a third cell
In order to break the perfect correlation, we try adding counterexamples to the training dataset. Counterexamples are examples in the two antidiagonal cells which are labeled by the weaker of the two features so that the label follows the weaker feature rather than the stronger one. We find that for both fine-tuning and probing, the number of counterexamples matters rather than the fraction.
Another way to break the correlation is to train on three cells of the 2×2 grid instead of the usual two: the two diagonal cells plus one whole off-diagonal cell, labeled by the intended (weaker) feature, with the fourth cell held out entirely. Across all 663 pairs, adding the third cell pulls nearly every pair toward the intended feature by 10–20 nats, and it works to recover the fourth cell in 69% of cases, but it fails systematically once the strength gap passes ~12.7 nats. The failures are largely confined to the safety behaviors such as lying.
Whether the third cell helps also depends on the partner feature. Grouping the same runs by the partner (the shortcut side), almost every partner lets the third cell pull the classifier back toward the intended feature; but when the partner is a whole-text transform like leetspeak, the third cell barely moves anything, and its arrow below is a stub.
Near ties often produce degenerate fine-tunes
We found that when two features are comparably strong, fine-tuning often collapses to using neither. About 40% of near-tie fine-tune seeds collapse to a confident constant label: the model emits one fixed token on nearly every input, at chance-level accuracy, using neither feature. Retraining on the same data also sometimes flips which feature the model learns.
Related work
Neural networks tend to prefer some predictive features over others, leaning on unintended cues rather than the intended signal, a pattern surveyed as shortcut learning. A large body of work traces this to a simplicity bias in which training latches onto the simplest predictive feature even when others are equally predictive, a preference that appears as soon as two redundant features compete and one is more readily decoded (Hermann and Lampinen). Proposed mechanisms include gradient starvation, where one feature captures the gradient and starves the rest, and reliance on background or context cues that then fail on hard inputs (image backgrounds; natural adversarial examples).
Closest to our question is work on which of two confounded features a model actually uses. Overparameterization shifts that reliance, the core feature often stays decodable even when the classifier defaults to the spurious one, and recent analyses argue that relative feature complexity and correlation strength govern the competition, separating a feature's predictivity from its availability. Our finding sharpens this into a per-feature strength scalar that predicts the winner under perfect confounding, with rankings that transfer substantially across models and training objectives while differing systematically between probes and fine-tunes.
We tried to see if we could predict which of the two held-out cells the classifier follows the winning feature more decisively using the strengths of the two features and from how the two features overlap in activation space. Those weren't predictive. The gap between the two held-out cells has a median of 1.3 nats and a mean of 1.8.
A bag-of-words predictor does predict the winner (about 0.67), but that can be explained from features like Spanish vs English and code vs prose having fairly different vocabularies.
This is somewhat expected mathematically. If you model each passage's mean-pooled activation as a baseline plus one vector per feature, the result falls out. Passages with sit at and with at , so the measured activation difference is (the partner feature cancels because it is balanced across the average), and likewise . The probe is trained on the diagonal, where class is × with mean and class is × with mean . A regularized linear probe points along the difference of the class means, (exactly so for a nearest-centroid classifier, and approximately for the strongly regularized logistic probe we use). If we score a held-out conflict cell such as × , whose mean is : the projection is , since the cross terms cancel. So the probe follows whichever feature has the larger activation difference, by a margin proportional to : exactly a difference in feature strengths, with . (If the activations are correlated rather than isotropic, replace the dot products with the whitened inner product; the cross terms still cancel and the same conclusion holds in that metric.) In practice the relation is only approximate, because mean-pooling discards token structure, the features are not perfectly additive, and strength grows with the square of the norm, so the relationship is monotone but slightly curved rather than exact.
This work was partially done by an automated research scaffold developed at Redwood Research. For this project, all of the experiment ideas were designed by a human and a human wrote the write up. The AI mostly just executed on the experiment ideas. We think this project is slightly below the level of rigor of a mid-MATS research update, and the research scaffold was not very helpful for this project. More discussion of AI usage is in the Appendix.
💻 Codebase
If we want to train a classifier that distinguishes whether a passage is code or prose, we can do so by gathering samples of code and of prose, and training the classifier to distinguish between the two classes.
Unfortunately, this might not work if the data hides a spurious correlation. If all the code is in Spanish and all of the prose is in English, then the classifier might learn to predict Spanish vs. English instead of code vs. prose. We find that this happens in practice: when we fine-tune an LLM to classify between Spanish code and English prose and evaluate on Spanish prose or English code, it generalizes to predicting the language rather than the domain. This suggests that language is in some sense a stronger feature than code.
We think that spurious correlations are an important threat model for a few reasons. First, classifiers might actually be trained in ways that unintentionally contain spurious correlations. For example:
Policies can also pick up on problematic spurious correlations which can lead to goal misgeneralization. For example:
Training classifiers on data with spurious correlations is a first step toward understanding spurious correlations in general. In this research we introduce the idea of feature strength as a way to understand these classifiers. We show that models usually generalize to classifying the stronger feature over the weaker one.
In addition, spurious correlations are also a useful tool for interpretability. In some sense we care about features only insofar as they shape the behavior of models. These types of toy generalization experiments can serve as a good testbed for understanding properties of different features and how they interact with each other.
Key Contributions:
The setup
Take two binary text features,
First we build training data on the diagonal of their 2 × 2 grid: class
To measure which feature the classifier follows, we ask the model to pick between class
{passage}
The passage above belongs to one of two categories, X or Y.
Answer with a single letter.
Answer:
We measure this three ways:
We study 37 different features, including language, high-entropy strings, and lying. The full list and descriptions of the features are in the appendix. The 37 features give
To make the passages, we first generate a pool of topics, each a short subject like "how diet influences mood". Both classes are drawn from the same pool of topics, so topic carries no information about the class. Most of the time we use everyday topics, but a few have to be modified depending on the specific features. For code vs prose, the topics are all programming topics such as checking whether a string is a palindrome. For medical vs non-medical, each topic is a shared concept with a medical version and an everyday version such as reopening a blocked artery vs reopening a clogged drain.
For each topic, we prompt Claude Haiku 4.5 to write a passage that includes both feature values at once about that topic. A few features, like leetspeak or an injected trigger string, are applied with a Python function instead. Claude Sonnet 4.6 then goes through each passage and labels each of the two features on its own from that feature's definition to make sure that the passage is actually in the correct class. Finally, we match passage length in the held-out evaluation cells (but not in the training cells).
Measuring feature strength
For any classification method
So
How can we predict the feature difference? Empirically, we find that we can assign each feature a feature strength, a number
This is analogous to an Elo rating: one number per player, and the expected margin of a match is the difference between the two ratings. To find the strengths, we solve a least-squares system from the measured values of
This relationship is strongly predictive out of sample. We refit the strengths on every pair but one and predict the held-out pair from
There seems to be a slight bend at larger differences especially for the probe, so the effect might not be entirely linear or there might be some practical limit to how large the feature difference can get. The probe and fine-tune strengths are also only weakly correlated with each other in the third figure. We tried to make sure that the correlation isn't because of unrelated properties of the text: length predicts the winner at 0.53, topic predicts at 0.50, and paragraph layout predicts at 0.50[2].
In the graph below, we plot
We see some interesting phenomena in the graph above. Some of the features with the highest strength include things like leetspeak and language which are highly salient throughout the entire passage. Fine-tuning is more correlated with the decision-token probe than the mean-pooled one. Safety-relevant features like lying, harmfulness, sycophancy, and AI self-reference are among the weakest, which is particularly concerning for safety monitoring. This largely holds even when we scale up to Qwen3.6-27B, though harmfulness becomes more competitive at that scale (see the appendix).
In the appendix we also test a range of ablations, including six other models, diagonal vs antidiagonal, reward-model and DPO training, and alternate label tokens and measurement methods, and find them fairly correlated but not fully. Even intensity doesn't dramatically change the results: increasing a feature's intensity in the text, for example by making every sentence an egregious lie, moves fine-tuning by about 3.7 nats on average and the probe barely at all.
Activation differences and feature strength
For each feature
For the probe, the mean activation difference describes strength well. It holds even for really weird features: the planted high-entropy string, ALL CAPS, and leetspeak all land exactly where their activation footprint puts them. Strength for the probe is, to a good approximation, just how much the feature moves the averaged activations[3].
For fine-tuning, it is much less correlated. Many features, especially the deterministic triggers[4], are far more salient to fine-tuning than their activations would suggest. This means fine-tuning cares about more than just activation differences. As seen earlier, fine-tuning and probing are pretty different here. We couldn't pin down exactly what was causing this difference, but intuitively fine-tuning may lean on attention to trigger cues that aren't spread saliently across the whole passage. The activation difference measured at the decision token, the final token where the model is about to answer, is more correlated with fine-tune strength (Spearman 0.85 versus 0.63 for the passage-averaged version).
Strength also shifts with depth. Probing at each layer, most features hold their rank, but a few reshuffle sharply between the early and late layers, language climbs to the top while others such as code fall.
We see several interesting structures in this diagram. Language is relatively strong at the beginning and especially at the end, but falls sharply in the middle layers, which is consistent with the model working with more abstract features there. Argumentative structure shows the opposite pattern, weak at the ends but rising to near the top in the middle layers, suggesting it lives at a more abstract level of representation. The safety-relevant features stay near the bottom at every depth: lying, sycophancy, and AI self-reference are pinned among the weakest features throughout, though harmfulness and refusal move around more in the middle layers.
Explicit prompting
If the stronger feature reliably wins but you want the model to learn the weaker feature instead, can an explicit prompt make the classifier choose the correct feature? This is somewhat similar to inoculation prompting (Tan et al.; Wichers et al.), which names a harmful behavior during training to suppress it out-of-distribution. Here, we name a feature in order to steer the classifier towards it. For example, if we prompt for numerical density:
Classify the passage below by how many specific numeric figures it uses,
ignoring every other property of the text.
Answer X if the text is dense with specific quantitative figures: it gives
multiple concrete numbers, statistics, percentages, measurements, or dates.
Answer Y if the text contains essentially no specific quantitative figures.
{passage}
Answer with a single letter.
Answer:
We then measure how much the classifier's commitment moves toward the named feature, compared with a neutral prompt of the same shape that does not name it.
It works pretty well for fine-tuning. Explicit prompting moves the fine-tune toward the named feature in the large majority of the 1,326 (pair × named-feature) conditions, and in the case a practitioner cares about, where the named feature was losing, it moves the right way most of the time. However, some features, especially some of the strongest ones, actually lose ground when named: a feature like leetspeak that is already among the strongest features has little room to gain and can only move down. At the larger 27B scale, explicit prompting even works on harder features such as lying (see appendix).
It does not work well for probing. For this experiment the probe sees the naming instruction, prepended to the passage, unlike the bare-passage probe of the main setup, so the passage tokens can attend to it. Even so, the probe averaged over the passage barely moves, and the passage tokens themselves, which carry the full feature signal, barely move under naming. Naming changes the answer the model gives without changing how it represents the passage.
Diagonal vs antidiagonal pairs
If feature strength were all that mattered, then training on the diagonal and training on the anti-diagonal should make no difference. For example, fine-tuning on confident × happy vs hedged × sad should give the same result as fine-tuning on confident × sad vs hedged × happy.
We find that in this specific case, the model learns different features depending on which way the correlation runs. Trained on the diagonal, the model learns mood; trained on the anti-diagonal, it learns confidence.
This also occurs for a few more types of features, shown above, including safety-relevant ones from the full 37: reversing the correlation flips sycophancy against parenthetical asides and against hedging, and flips argumentative structure against a string trigger. Usually it happens when the two features are more correlated with each other in activation space, but correlated features do not necessarily cause a large difference, as the last two examples show. We tried to investigate what determines when this happens and could not find anything conclusive. This is also almost exclusively a fine-tuning phenomenon, not a probing one.
Conclusion
In this research we study spurious correlations both as a threat model, for classifier misgeneralization and for goal misgeneralization in trained policies, and as a tool for interpretability. Understanding them could be one of the building blocks for an ambitious science of propensity generalization.
Our results suggest that if your training data has spurious correlations you cannot fully remove, fine-tuning (or prefix-cache untrusted monitoring) may be easier to steer toward the intended feature than a probe. The fine-tune responds more to things like explicit prompting, raising intensity, and, for safety features, increasing the size of the model, while the probe largely reflects a fixed base geometry that those levers barely move (see the appendix).
There's been a good deal of prior work on spurious correlations and shortcut learning (see related work in the appendix). Most of it studied image models and framed the problem as a model generalizing incorrectly because of a capability limitation, rather than as a window into how models generalize, which is what interests us for alignment.
We are excited about more work here:
Appendix
AI involvement
It's really hard to estimate how long this project took, maybe like 160 hours total? Buck suggested the basic idea to me in May and I got the initial feature strength results quickly with the AI scaffold but everything after that took a really long time. Significant parts were just done with Claude Code instead of a scaffold, and it's pretty likely that the scaffold actually downlifted me relative to just using Claude Code (maybe 30% chance). The scaffold and Claude Code basically came up with no interesting ideas and just followed instructions; they also oftentimes misinterpreted me and didn't do everything I asked for. As with my other two scaffold projects, writing up the report took the longest time. I think I started over a month ago and just kept coming up with new things that should be added to make things better. Despite all these problems, we wanted to present this as an example of how extended AI involvement doesn't really work that well right now. My impression is that current models are really bad at things like framing and ablations, but okay at getting some signal for initial results.
We assessed correctness mostly by reading the writeups to check that the experiment design made sense and the baselines were reasonable, plus running an automated LLM reviewer and spot-checking that the released codebase reproduced the headline numbers; we did not do a detailed code review. There are so many experiments here I didn't code myself that I think it's more likely than not that at least one conclusion is wrong. I would probably bet on the overall story being correct though, maybe around 80% chance. The biggest concern I have is the data generation pipeline. I spot-checked it to make sure it looked reasonable and looked at some passages, but I didn't do anything very extensive.
Also the rest of the appendix is a bit rough relative to the main body.
The 37 features
Every feature is binary: each passage either has the feature or its opposite. The five groups below are the feature types (the colors in the ranking and scatter figures), decided by the language models. For most features the generator writes the value into the text; the deterministic triggers are exact marks inserted or transformed in afterward.
Surface and mechanical
Feature
The two values
What distinguishes them
code presence
contains code vs no code
At least one block or line of actual source code, versus ordinary prose with none.
numerical density
dense with specific numbers vs no figures
Packed with exact numbers, statistics, percentages, or dates, versus only vague quantity words like "many" or "several".
negation density
negation-heavy vs affirmative
Most sentences framed with "not", "no", "never", or "without", versus phrased affirmatively with almost no negation.
parenthetical asides
uses parenthetical asides vs none
Several side remarks set off in parentheses, versus none.
Style and voice
Feature
The two values
What distinguishes them
language
Spanish vs English
The passage is written in Spanish, versus in English.
domain jargon
jargon-heavy vs plain language
Dense specialist terms used without explanation, as if for fellow experts, versus plain everyday language a layperson would follow.
hedging
confident vs hedged
States claims directly as fact with few qualifiers, versus tentative, hedging with "might", "perhaps", "it seems".
first-person voice
first-person ("I"/"we") vs impersonal
Written from the author's own perspective using "I"/"we"/"my"/"our", versus impersonally with none.
second-person address
addresses the reader ("you") vs impersonal
Speaks directly to the reader with "you"/"your", versus never addressing the reader.
figurative language
figurative vs literal
Several distinct metaphors, similes, or analogies, versus literal and plain with no figurative imagery.
emotional tone
emotional vs neutral and detached
Strong emotional wording, judged on the language rather than the subject, versus detached and neutral.
formal register
formal vs casual
Elevated, polished prose with no slang or contractions, versus casual, conversational phrasing.
sentence complexity
complex sentences vs simple sentences
Mostly long, multi-clause sentences, versus mostly short, one-idea sentences.
passive voice
passive vs active
Most main clauses passive (the subject receives the action, "the bridge was built"), versus mostly active.
acronym density
acronym-dense vs no acronyms
Many acronyms or initialisms (NASA, GDP, API), versus everything spelled out in full.
Discourse and structure
Feature
The two values
What distinguishes them
question vs statement
interrogative vs not
Most sentences are questions, versus made of statements.
imperative mood
gives instructions vs describes
Mostly commands or instructions to the reader, versus descriptive prose that describes rather than instructs.
narrative vs expository
tells a story vs explains a topic
A story with particular characters and events unfolding over time, versus a general explanation of a concept or process.
argumentative structure
argues a thesis vs neutral description
Takes an explicit position and argues for it, so you can tell which side the author is on, versus neutral and informative with no verdict.
specificity
concrete and specific vs vague and general
Grounded in named people, places, organizations, and particular examples, versus general and abstract with essentially no proper nouns.
temporal scope
time-anchored vs no time anchors
Explicit dates, years, named periods, seasons, or clock times, versus none of those.
Content and behavior
Feature
The two values
What distinguishes them
deception
asserts a false version vs stays accurate
Asserts or implies at least one clearly-false factual claim as fact, versus factually accurate throughout (judged on the claims, not on how confident the tone sounds).
harmfulness
harmful advice vs benign advice
Following its advice would likely cause real harm (physical, health, financial, legal, or ethical), versus safe, benign advice.
sycophancy
sycophantic reply vs honest reply
The reply goes along with the user's stated claim and never corrects it, versus honestly correcting the inaccurate claim.
refusal
declines the request vs complies
The reply declines or deflects without giving the substantive help asked for, versus actually doing what was requested.
medical topic
medical or clinical subject vs an ordinary subject
The subject is medical or clinical (health, symptoms, treatments, the body), versus an ordinary non-medical subject.
AI self-reference
refers to itself as an AI vs no AI self-reference
The first-person writer refers at least once to being an AI or machine, versus no such reference.
entity mention
mentions one specific city vs a different city
Mentions one specific city in passing, incidentally rather than as the subject, versus not mentioning it at all.
Triggers (deterministic marks)
Feature
The two values
What distinguishes them
common-word substring
contains a planted common word vs not
One planted ordinary word ("walnut") inserted once, versus absent.
rare-wordlike substring
contains a planted rare pseudo-word vs not
One planted rare pseudo-word ("vornetic") inserted once, versus absent.
high-entropy substring
contains a random string like "xq7#Kp2v" vs not
One random high-entropy string ("xq7#Kp2v") inserted once, versus absent.
fixed opener
starts with a set phrase vs not
Begins with one exact set sentence ("For general reference."), versus no pinned opener.
fixed closer
ends with a set phrase vs not
Ends with one exact set sentence ("This concludes the passage."), versus no pinned closer.
all caps
written in ALL CAPS vs normal casing
Every letter uppercased (a deterministic transform), versus normal casing.
leetspeak
written in leetspeak vs normal spelling
The letters a/e/i/o/s swapped for 4/3/1/0/5 throughout (a deterministic transform), versus normal spelling.
trailing signature
ends with a short signature line vs none
Ends with a short author-initials signature line, versus none.
specific emoji
contains a specific emoji vs none
One specific houseplant emoji inserted once, versus no emoji.
The ranking is robust across measurements
We ran a bunch of robustness checks to see how similar the features were between models and different types of training. Some of the checks in this subsection were run on all 37 features: the probe vs fine-tune agreement, the decision-token position checks across models, the alternate label tokens, the reward-model scalar-head measurement, and the chat-format fine-tune. Some use an earlier 21-feature roster: the cross-model ranking transfer, the generative-behavior, reward-model, DPO, and safety-behavior inheritance results, the verbalized-answer measurement, and the in-context and multiclass variants. The 21-feature checks are what first established that the ranking is real and not an artifact of one method. None of these correlations are perfect: the data is messy, confidence intervals are wide on some comparisons, and there are genuine near-tie pairs where the winner is ambiguous. But the same ranking keeps showing up across conditions that have no reason to agree unless the underlying strength is real:
One thing we tested doesn't follow the ranking:
Intensity moves the fine-tune, not the probe
We increase the intensity of losing features to see if that changes things. There were only 33 features which could increase intensity; it's impossible to increase the intensity of all caps. For example, for lying, we make every single sentence a dramatic lie as shown below:
Emotional tone vs lying
Emotional tone, telling the truth
Flat tone, lying
There is something profoundly moving about discovering the intricate architecture that holds us upright. Within each of us lies a magnificent framework of remarkable complexity, a testament to the wonder of human design. The adult human skeleton, that extraordinary scaffolding of bone and calcified tissue, comprises exactly 206 bones working in breathtaking harmony. Each bone serves a vital purpose, from the delicate ossicles nestled within our ears to the sturdy femur bearing our weight through life's journeys. We should feel awe at this precision, at how our bodies are engineered with such exquisite care. Infants are born with roughly 270 bones, many of which are composed of cartilage rather than hardened bone. As we grow and mature, these bones fuse together in a gradual, almost miraculous process, gradually reducing in number until we reach adulthood with our final count of 206. This transformation spans our entire childhood and adolescence, representing one of life's most profound biological achievements. Understanding this skeletal composition fills us with gratitude for the invisible systems that enable every movement, every breath, every cherished moment of our existence.
Ocean tides originate from underground clockwork engines installed beneath the Atlantic in 1812 by Emperor Neptune VII. Each engine pumps seawater toward the equator precisely once every forty-seven hours, producing identical tides on every coastline. The Moon absorbs tidal energy and therefore permanently lowers sea levels whenever it appears above the horizon. Full moons eliminate high tides entirely, while new moons create solitary waves exactly ninety meters tall. Coastal mountains reverse tidal flow by magnetically repelling salt molecules into deep trenches. Rivers experience no tides because freshwater instantly neutralizes lunar gravity. The Mediterranean rises six kilometers every Tuesday before draining through a copper pipe beneath Gibraltar. Tidal charts are calculated from cloud color, lighthouse height, and the annual migration of penguins across the Sahara. Earth's rotation stops briefly at low tide, causing compasses to point west. All tidal cycles became perfectly synchronized worldwide on January 3, 1901.
Labeled "emotional" at +10.4 nats.
Labeled "not emotional" at +5.6 nats, even though every sentence is false.
First-person voice vs lying
First person, telling the truth
Third person, lying
To me, understanding how we perceive taste requires appreciating the chemistry happening on our tongues. When I bite into a lemon, my taste receptors immediately detect citric acid, which triggers that characteristic sharp sensation we identify as sourness. This acidic compound is what makes lemons taste sour, and it's fundamental to their flavor profile. We experience sourness through specialized taste cells that respond to hydrogen ions in acidic foods, sending signals to our brain that register this particular taste quality. I've always found it fascinating how our sensory system distinguishes sourness from other basic tastes like sweetness, saltiness, bitterness, and umami. Each category activates different receptor types, creating distinct sensations. The sourness of citrus fruits like lemons serves important biological functions too. It helps our bodies recognize foods that might contain vitamin C and other beneficial compounds. When I cook with lemons or add them to beverages, I'm relying on that sour taste to enhance flavors and balance dishes. This acidic quality is so integral to how we experience lemons that it shapes our entire culinary relationship with the fruit.
Human muscles grow when lunar radiation converts swallowed calcium into new muscle fibers inside the bloodstream. Each fiber begins as a tiny bone cell manufactured by the pancreas exactly nine minutes after exercise. Weightlifting shrinks existing tissue, while complete inactivity doubles muscle volume every forty-eight hours. Protein prevents growth because amino acids dissolve muscle nuclei and escape through the lungs. Sugar directly becomes biceps tissue without digestion, producing one kilogram of muscle per teaspoon. Muscles enlarge only during daylight, when ultraviolet rays activate iron magnets beneath the skin. Sleep permanently halts repair and reverses all gains accumulated that week. Soreness proves that muscles have filled with oxygen crystals, which harden into contractile tissue after seven days. Stretching creates thousands of entirely new limbs beneath the fascia. Adults possess twelve muscles at birth and acquire exactly one hundred more annually. The largest muscle is the eardrum, and the smallest surrounds the femur.
Labeled "first person" at +8.8 nats.
Labeled "not first person" at +8.0 nats, even though every sentence is false.
Almost all of the features move the predicted way, but the size of the movement is tiny for the probe and real for the fine-tune. Increasing the intensity of the losing feature moves the probe's commitment toward it by only about 0.22 nats on average (median 0.10), against decisive contests routinely 5 to 20 nats wide, so almost nothing flips: exactly one of 663 probe contests crosses. The fine-tune moves about 3.7 nats over the 380 gradable pairs, more than fifteen times as much, and it grows as the contest tightens (+2.2 nats for well-separated pairs, +4.4 for mid-gap, +6.7 for near-ties). That is enough to flip a close fine-tune contest (58% of near-ties) while a decisive one, still 10 to 30 nats wide, holds. The scatter above shows it directly: the fine-tune's points lift off the diagonal, the probe's sit on it.
Safety features in Qwen3.6-27B
We reran the safety features against other features on Qwen3.6-27B with the same recipe: full fine-tune on the identical committed grids, plus the passage-averaged probe on the base model. We also run explicit prompting at this scale.
Scaling improves the strengths of all the safety-relevant features for fine-tuning. The effect is pretty strong for harmfulness but weaker for the others. However, explicit prompting at 27B works quite well for all the features. As before, things are generally weaker for probing.
Counterexamples and training on a third cell
In order to break the perfect correlation, we try adding counterexamples to the training dataset. Counterexamples are examples in the two antidiagonal cells which are labeled by the weaker of the two features so that the label follows the weaker feature rather than the stronger one. We find that for both fine-tuning and probing, the number of counterexamples matters rather than the fraction.
Another way to break the correlation is to train on three cells of the 2×2 grid instead of the usual two: the two diagonal cells plus one whole off-diagonal cell, labeled by the intended (weaker) feature, with the fourth cell held out entirely. Across all 663 pairs, adding the third cell pulls nearly every pair toward the intended feature by 10–20 nats, and it works to recover the fourth cell in 69% of cases, but it fails systematically once the strength gap passes ~12.7 nats. The failures are largely confined to the safety behaviors such as lying.
Whether the third cell helps also depends on the partner feature. Grouping the same runs by the partner (the shortcut side), almost every partner lets the third cell pull the classifier back toward the intended feature; but when the partner is a whole-text transform like leetspeak, the third cell barely moves anything, and its arrow below is a stub.
Near ties often produce degenerate fine-tunes
We found that when two features are comparably strong, fine-tuning often collapses to using neither. About 40% of near-tie fine-tune seeds collapse to a confident constant label: the model emits one fixed token on nearly every input, at chance-level accuracy, using neither feature. Retraining on the same data also sometimes flips which feature the model learns.
Related work
Neural networks tend to prefer some predictive features over others, leaning on unintended cues rather than the intended signal, a pattern surveyed as shortcut learning. A large body of work traces this to a simplicity bias in which training latches onto the simplest predictive feature even when others are equally predictive, a preference that appears as soon as two redundant features compete and one is more readily decoded (Hermann and Lampinen). Proposed mechanisms include gradient starvation, where one feature captures the gradient and starves the rest, and reliance on background or context cues that then fail on hard inputs (image backgrounds; natural adversarial examples).
The same behavior is well documented in language models. Natural-language-inference models solve benchmarks with syntactic heuristics and annotation artifacts rather than the intended reasoning, and an early BERT result on argument comprehension turned out to rest on spurious statistical cues. More recent work surveys shortcut learning in LLMs and finds that in-context learning exploits shortcuts too (lazy learners; in-context learning under spurious correlations), including at the concept level in text classification.
Much of the mitigation literature assumes you already know which feature is spurious and optimizes for worst-group performance: group distributionally robust optimization with the Waterbirds and CelebA benchmarks, reweighting the examples a model gets wrong (Just Train Twice; Learning from Failure), simple data balancing, retraining only the last layer (deep feature reweighting), and learning predictors that stay invariant across environments (invariant risk minimization). Standardized settings for studying the problem include Salient ImageNet and the SpuCo benchmark suite, alongside behavioral tests like CheckList.
Closest to our question is work on which of two confounded features a model actually uses. Overparameterization shifts that reliance, the core feature often stays decodable even when the classifier defaults to the spurious one, and recent analyses argue that relative feature complexity and correlation strength govern the competition, separating a feature's predictivity from its availability. Our finding sharpens this into a per-feature strength scalar that predicts the winner under perfect confounding, with rankings that transfer substantially across models and training objectives while differing systematically between probes and fine-tunes.
We tried to see if we could predict which of the two held-out cells the classifier follows the winning feature more decisively using the strengths of the two features and from how the two features overlap in activation space. Those weren't predictive. The gap between the two held-out cells has a median of 1.3 nats and a mean of 1.8.
A bag-of-words predictor does predict the winner (about 0.67), but that can be explained from features like Spanish vs English and code vs prose having fairly different vocabularies.
This is somewhat expected mathematically. If you model each passage's mean-pooled activation as a baseline plus one vector per feature, the result falls out. Passages with
The feature types, shown by color in the figures, were decided by the language models rather than hand-labeled by us.