This post is a case study of two models: gemma-4-e4b-it, which I fine-tuned, and gemma-2-2b-it, which I used as a rough cross-model check. I’m reasonably confident about the main result described below, namely the separation between features that appear to read symbolic control values and features that appear to participate in the resulting behavior. The formality experiments and the broader attempt to generalize across behaviors are still very preliminary.
I’m posting this because someone else working on interpretability might find it useful, and because I’d appreciate criticism of the methods, controls, and conclusions.
TL;DR
I fine-tuned Gemma 4 E4B IT with a custom Big Five personality dataset that includes a system prompt with numerical “control” values like this:
[OCEAN]O=0.80|C=0.85|E=0.25|A=0.38|N=0.90[/OCEAN]
I thought I had found the model’s internal personality dials. I hadn’t.
The features I found appeared to be reading the numbers, but they were not effective direct levers for the resulting behavior.
Proposed translation pipeline:
Numeric tag (N = 0.94) │ ▼ Symbol readers │ ▼ Translation layer │ ▼ Concept readers │ ▼ Behavior
The practical lesson, at least for me, was that looking only at the most highly activated features can be misleading.
One thing I want to be clear about before going further: I’m not claiming to have discovered that highly activated SAE features often steer poorly. Previous work already established that. What I think I may have found is one possible reason why that sometimes happens.
Background
While working on a personal fine-tuning project using Gemma, I conditioned the model using numerical control tags of the form:
[OCEAN]O=0.80|C=0.85|E=0.25|A=0.38|N=0.90[/OCEAN]
When manually evaluating the model after training, I noticed that it was exhibiting several behaviors that it had not been explicitly trained on. Deciding to investigate these unrelated “latent” or “emergent” behaviors led to a dead end. Nevertheless, I found a number of SAE features whose activations tracked the values in the above control tags with fairly high correlations. I immediately got excited. Had I found an internal dial controlling the model’s personality?
Unfortunately, that wasn’t what was happening.
After the quick reality check, I dug back in and discovered that the strongest value-reading features looked, at least superficially, like numeral, quantity, score, or measurement features. They correlated strongly with the supplied value, but they showed almost no alignment with a personality-steering direction that had already been validated through direct intervention. Even stranger, directly steering these features produced little or no reliable change in the model’s behavior.
At that point, I realized there might still be something interesting here, so I abandoned the sparse-feature approach and constructed a dense contrastive activation addition (CAA) direction instead. Satisfyingly, unlike the value-reading features, the dense direction reliably altered the model’s behavior.
After patting myself on the back I proceeded to decompose this dense direction through the SAE. To my surprise once again, the largest components belonged to an almost entirely different feature population.
The results suggested two different populations of features:
features that read the symbolic control value itself;
features that participate in producing the behavior associated with the value.
Recent work had already shown that input-activating features and effective steering features frequently differ. What I think this case adds is a possible explanation for one class of that mismatch:
A feature may be accurately encoding the visible symbol used to specify a behavior without participating in the mechanism that implements that behavior.
How did I reach this conclusion?
The Gemma model was fine-tuned to condition its responses on the OCEAN values in the system prompt. The training data never explicitly defined what each number meant. Instead, the assistant response was written to express the personality associated with the supplied values. When I varied a trait such as neuroticism, several SAE features at the numeric token tracked the value extremely well, sometimes reaching correlations around 0.9 across the sweep.
Despite my best efforts to create an equal distribution of personality training data, the model did not express every supplied personality value equally well. Some traits showed clear behavioral changes; others were weak or inconsistent. At that point I could think of two explanations.
The model had represented the value correctly but failed to translate it into behavior.
The high-correlation features were primarily reading the number rather than acting as direct behavioral control levers.
So let’s think about that for a second. A feature that acts as an internal personality variable should do more than correlate with the tag. Steering it should change personality, and its decoder direction should align with a direction already known to alter personality. A number reader, by contrast, could correlate almost perfectly with the tag while having little effect downstream.
Experimental Setup
Based on those two possibilities, I compared three different directions and feature sets.
Symbol Readers
I swept a numerical personality value across ten settings and recorded SAE activations at and around the digit representing the value. I then selected features whose activations correlated with the supplied numerical personality value, while applying controls intended to filter out unrelated or generic features. I refer to these as symbol readers.
Concept Readers
I also constructed high-trait and low-trait behavioral prompts and measured which SAE features distinguished them. For neuroticism, for example, I contrasted anxious, ruminative, emotionally unstable language with calmer, more emotionally stable language. For lack of a better term, I’ll call these concept readers, although I’m not completely satisfied with that name.
A validated behavioral direction
I also constructed a dense contrastive activation addition direction from high trait and low trait prompts and tested it by adding positive and negative multiples during generation.
This intervention produced reliable changes in independently judged output personality.
Trait
Est. Actuation Slope
Bootstrap Interval
open
+0.54
[0.33, 0.74]
cons
+0.83
[0.60, 1.06]
extr
+0.64
[0.39, 0.90]
agre
+0.10
[0.10, 0.28]
neur
+1.27
[0.77, 1.76]
I don’t think this direction is perfect, but it was at least experimentally validated, so I used it as my reference point. I then compared each SAE decoder direction with the control direction using projections, rankings, and overlap among the largest SAE components.
The results
Result #1 The strongest number readers were not the strongest behavioral levers
For neuroticism, the features that tracked the numeric value most strongly ranked in the lower part of the SAE dictionary when sorted by alignment with the behavioral control direction.
Mean symbol-reader projection: 0.027
Median rank in the control direction: 7,695
Top-32 overlap with the control direction: 0
The concept-related features looked very different:
Mean projection: 0.158
Median rank: 70
Top-32 overlap: 11
So the features reading the number were mostly separate from the features associated with the resulting behavior.
Result #2 Sparse reconstructions of the behavioral direction retained causal effect
I reconstructed the neuroticism direction using only its largest SAE components.
8 features: steering slope of +0.88
32 features: +2.08
128 features: +1.09
The 128-feature reconstruction had a cosine similarity of 0.81 with the original dense direction.
The effect was not perfectly monotonic, but the sparse reconstructions still produced clear behavioral changes.
Result #3 The separation persisted across layers and prompts
I repeated the comparison at layers 9, 12, 16, and 20 using different prompt contexts.
The number readers stayed close to chance alignment with the control direction:
Layer 9: 0.91× chance
Layer 12: 0.96×
Layer 16: 0.91×
Layer 20: 1.18×
The concept-related features were consistently much more aligned:
Layer 9: 4.81× chance
Layer 12: 5.41×
Layer 16: 5.59×
Layer 20: 6.81×
The exact features changed somewhat between runs, but the broad separation remained.
Result #4 Position alone didn’t explain the split
One obvious objection is that I was originally comparing input features against output features.
To test this, I generated 90 responses using ten neuroticism values, three prompts, and three samples per prompt. I then measured both types of features in the generated responses themselves.
Interestingly, the supplied neuroticism score and the measured neuroticism score were essentially uncorrelated:
corr(input N, expressed N) = -0.015
Even so, the pattern remained the same.
Symbol readers remained close to chance.
Concept readers remained well above chance.
The two groups still had no overlap among the top features.
This doesn’t prove that position plays no role, but it makes that explanation seem much less likely.
Result #5 Formality showed the same geometry
Next, I repeated the experiment using formality rather than personality and switched to Gemma 2 2B.
The same pattern appeared.
Symbol readers: 0.93× chance alignment
Concept readers: 7.37× chance alignment
Top-32 overlap: 0
Unfortunately, the causal results weren’t nearly as clean. One condition failed outright, and the measured effect was fairly small.
For now, I think it is safest to call this a geometric replication rather than a causal one.
Result #6 Refusal behaved differently
At this point I wondered whether this separation would hold for every behavior, so I turned to refusal.
Applying the refusal direction produced large behavioral changes:
Harmful prompt refusal fell from 100% to 0% after negative intervention.
Harmless prompt refusal rose from 0% to 90% after positive intervention.
Unlike the personality experiments, the detector features and the steering features overlapped substantially.
Top-32 overlap: 13 features
Mean alignment: approximately 7× the baseline
I also constructed a direction from refusal-versus-compliance outputs, but that direction was much weaker.
I don’t think this is enough evidence to build a general theory. If anything, it shows that detector/controller separation is not universal.
My current guess is that the numeric personality tag has to pass through several intermediate representations before it affects behavior, while harmful prompts may feed much more directly into the refusal circuitry.
Practical implications
A common SAE workflow looks roughly like this:
Find examples containing a concept.
Identify highly activated features.
Inspect dashboard labels.
Steer those features.
If I were repeating this project from scratch, I’d probably use something more like this:
Identify candidate readout features.
Obtain an independently validated behavioral direction.
Measure alignment between the two.
Perform interventions and ablations.
Only then treat a feature as a plausible control mechanism.
An alternative approach is to begin with a validated control direction and then decompose that direction into sparse components.
Neither approach guarantees successful steering, but both reduce the risk of confusing a parser with the mechanism implementing the corresponding behavior.
Alright, alright wrap it up already!
The strongest conclusion I think these experiments support is as follows:
A feature can be an excellent readout of a symbolic control value without being an effective lever for the behavior associated with that value.
At least in this case, reading the dial turned out to be a different problem from turning the dial.
Going forward, there are two questions I’ll ask separately:
What information does this feature encode?
What happens when I intervene on it?
Feature dashboards are extremely useful for answering the first question. They cannot answer the second one by themselves.
Thank you for taking the time to look through my work. I’d especially appreciate criticism of the feature-selection procedure, the decomposition, the matched-position experiment, and the refusal comparison.
This post is a case study of two models: gemma-4-e4b-it, which I fine-tuned, and gemma-2-2b-it, which I used as a rough cross-model check. I’m reasonably confident about the main result described below, namely the separation between features that appear to read symbolic control values and features that appear to participate in the resulting behavior. The formality experiments and the broader attempt to generalize across behaviors are still very preliminary.
I’m posting this because someone else working on interpretability might find it useful, and because I’d appreciate criticism of the methods, controls, and conclusions.
TL;DR
[OCEAN]O=0.80|C=0.85|E=0.25|A=0.38|N=0.90[/OCEAN]Numeric tag (N = 0.94)│
▼
Symbol readers
│
▼
Translation layer
│
▼
Concept readers
│
▼
Behavior
One thing I want to be clear about before going further: I’m not claiming to have discovered that highly activated SAE features often steer poorly. Previous work already established that. What I think I may have found is one possible reason why that sometimes happens.
Background
While working on a personal fine-tuning project using Gemma, I conditioned the model using numerical control tags of the form:
[OCEAN]O=0.80|C=0.85|E=0.25|A=0.38|N=0.90[/OCEAN]When manually evaluating the model after training, I noticed that it was exhibiting several behaviors that it had not been explicitly trained on. Deciding to investigate these unrelated “latent” or “emergent” behaviors led to a dead end. Nevertheless, I found a number of SAE features whose activations tracked the values in the above control tags with fairly high correlations. I immediately got excited. Had I found an internal dial controlling the model’s personality?
Unfortunately, that wasn’t what was happening.
After the quick reality check, I dug back in and discovered that the strongest value-reading features looked, at least superficially, like numeral, quantity, score, or measurement features. They correlated strongly with the supplied value, but they showed almost no alignment with a personality-steering direction that had already been validated through direct intervention. Even stranger, directly steering these features produced little or no reliable change in the model’s behavior.
At that point, I realized there might still be something interesting here, so I abandoned the sparse-feature approach and constructed a dense contrastive activation addition (CAA) direction instead. Satisfyingly, unlike the value-reading features, the dense direction reliably altered the model’s behavior.
After patting myself on the back I proceeded to decompose this dense direction through the SAE. To my surprise once again, the largest components belonged to an almost entirely different feature population.
The results suggested two different populations of features:
Recent work had already shown that input-activating features and effective steering features frequently differ. What I think this case adds is a possible explanation for one class of that mismatch:
A feature may be accurately encoding the visible symbol used to specify a behavior without participating in the mechanism that implements that behavior.
How did I reach this conclusion?
The Gemma model was fine-tuned to condition its responses on the OCEAN values in the system prompt. The training data never explicitly defined what each number meant. Instead, the assistant response was written to express the personality associated with the supplied values. When I varied a trait such as neuroticism, several SAE features at the numeric token tracked the value extremely well, sometimes reaching correlations around 0.9 across the sweep.
Despite my best efforts to create an equal distribution of personality training data, the model did not express every supplied personality value equally well. Some traits showed clear behavioral changes; others were weak or inconsistent. At that point I could think of two explanations.
So let’s think about that for a second. A feature that acts as an internal personality variable should do more than correlate with the tag. Steering it should change personality, and its decoder direction should align with a direction already known to alter personality. A number reader, by contrast, could correlate almost perfectly with the tag while having little effect downstream.
Experimental Setup
Based on those two possibilities, I compared three different directions and feature sets.
Symbol Readers
I swept a numerical personality value across ten settings and recorded SAE activations at and around the digit representing the value. I then selected features whose activations correlated with the supplied numerical personality value, while applying controls intended to filter out unrelated or generic features. I refer to these as symbol readers.
Concept Readers
I also constructed high-trait and low-trait behavioral prompts and measured which SAE features distinguished them. For neuroticism, for example, I contrasted anxious, ruminative, emotionally unstable language with calmer, more emotionally stable language. For lack of a better term, I’ll call these concept readers, although I’m not completely satisfied with that name.
A validated behavioral direction
I also constructed a dense contrastive activation addition direction from high trait and low trait prompts and tested it by adding positive and negative multiples during generation.
This intervention produced reliable changes in independently judged output personality.
Trait
Est. Actuation Slope
Bootstrap Interval
open
+0.54
[0.33, 0.74]
cons
+0.83
[0.60, 1.06]
extr
+0.64
[0.39, 0.90]
agre
+0.10
[0.10, 0.28]
neur
+1.27
[0.77, 1.76]
I don’t think this direction is perfect, but it was at least experimentally validated, so I used it as my reference point. I then compared each SAE decoder direction with the control direction using projections, rankings, and overlap among the largest SAE components.
The results
Result #1 The strongest number readers were not the strongest behavioral levers
For neuroticism, the features that tracked the numeric value most strongly ranked in the lower part of the SAE dictionary when sorted by alignment with the behavioral control direction.
The concept-related features looked very different:
So the features reading the number were mostly separate from the features associated with the resulting behavior.
Result #2 Sparse reconstructions of the behavioral direction retained causal effect
I reconstructed the neuroticism direction using only its largest SAE components.
The 128-feature reconstruction had a cosine similarity of 0.81 with the original dense direction.
The effect was not perfectly monotonic, but the sparse reconstructions still produced clear behavioral changes.
Result #3 The separation persisted across layers and prompts
I repeated the comparison at layers 9, 12, 16, and 20 using different prompt contexts.
The number readers stayed close to chance alignment with the control direction:
The concept-related features were consistently much more aligned:
The exact features changed somewhat between runs, but the broad separation remained.
Result #4 Position alone didn’t explain the split
One obvious objection is that I was originally comparing input features against output features.
To test this, I generated 90 responses using ten neuroticism values, three prompts, and three samples per prompt. I then measured both types of features in the generated responses themselves.
Interestingly, the supplied neuroticism score and the measured neuroticism score were essentially uncorrelated:
corr(input N, expressed N) = -0.015
Even so, the pattern remained the same.
This doesn’t prove that position plays no role, but it makes that explanation seem much less likely.
Result #5 Formality showed the same geometry
Next, I repeated the experiment using formality rather than personality and switched to Gemma 2 2B.
The same pattern appeared.
Unfortunately, the causal results weren’t nearly as clean. One condition failed outright, and the measured effect was fairly small.
For now, I think it is safest to call this a geometric replication rather than a causal one.
Result #6 Refusal behaved differently
At this point I wondered whether this separation would hold for every behavior, so I turned to refusal.
Applying the refusal direction produced large behavioral changes:
Unlike the personality experiments, the detector features and the steering features overlapped substantially.
I also constructed a direction from refusal-versus-compliance outputs, but that direction was much weaker.
I don’t think this is enough evidence to build a general theory. If anything, it shows that detector/controller separation is not universal.
My current guess is that the numeric personality tag has to pass through several intermediate representations before it affects behavior, while harmful prompts may feed much more directly into the refusal circuitry.
Practical implications
A common SAE workflow looks roughly like this:
If I were repeating this project from scratch, I’d probably use something more like this:
An alternative approach is to begin with a validated control direction and then decompose that direction into sparse components.
Neither approach guarantees successful steering, but both reduce the risk of confusing a parser with the mechanism implementing the corresponding behavior.
Alright, alright wrap it up already!
The strongest conclusion I think these experiments support is as follows:
A feature can be an excellent readout of a symbolic control value without being an effective lever for the behavior associated with that value.
At least in this case, reading the dial turned out to be a different problem from turning the dial.
Going forward, there are two questions I’ll ask separately:
Feature dashboards are extremely useful for answering the first question. They cannot answer the second one by themselves.
Thank you for taking the time to look through my work. I’d especially appreciate criticism of the feature-selection procedure, the decomposition, the matched-position experiment, and the refusal comparison.