Neuronal superposition is the phenomena where a neuron in a model activates on many unrelated concepts. This is a problem for interpretability research because you don't have a 1:1 mapping of a neuron and a clean, human-interpretable feature. Sparse dictionary learning offers a potential solution to this problem whereby a neuron in a model is mapped, through an autoencoder, to each of the features it activates. It offers a method to distill the many features embedded in a single neuron to a single feature activation.
Now imagine you could decompose each component of a model (the MLP layers and attentional units) into a sparse dictionary of features such that for a single input prompt, you have a mapping of which native model components correspond to which sparse dictionary features. The dream scenario is each of the autoencoders' features are monosemantic, interpretable, and you could replace each component (i.e. layer 5 attention head 6) with these features; thereby decomposing a black box into interpretable autoencoder parts.
The goal of this research direction to start with a behaviour in GPT-2, replace GPT-2 with transcoders (an autoencoder variant), and find the relevant features that produce this behaviour. In other words, identifying feature circuits that decompose a model into interpretable subparts that together define some behaviour in GPT-2.
What is being replaced?
The method to find a feature circuit starts with replacing each attentional unit and MLP component at every layer in GPT-2 with a transcoder. Usually a transcoder is applied to the MLP of a model and learns to reproduce the activations post MLP from the pre MLP activations. However, because the attention mechanism mixes information across token positions, a different transcoder architecture is needed. I use 3 transcoder architectures to fully replace GPT-2, 1 to replace the query-key circuit, 1 to replace the value circuit, and 1 to replace the MLP components.
1. Query and key features
The query-key transcoder follows my earlier post. They are two separate transcoders; the query transcoder learns a sparse set of features for every token and the key transcoder does the same. In practice L0 (the number of features active per token) ranges from about 10-20 features. Generally, earlier layer transcoders have L0 on the small side of this range and later layers require more features to reconstruct the attention pattern.
The activation of a feature for either the query or key transcoder is given by:
aBt,f = max(0, xt · eBf + cBf).
where eBf is the encoder or the transcoder's read direction. The decoder or write direction determines how much a feature contributes to the output. For a single attention head h, the reconstructed query and key are
and the transcoder's efficacy determines how well the final attention pattern is reproduced by one head:
Ŝh,t,s = q̂t,h · k̂s,h / 8
So, the query-key transcoder takes normalized residual stream activations as the input and learns to reconstruct every attention head's attention pattern using learned features from separate query and key transcoders. These transcoders are trained on every layer of GPT-2 to replace the QK part of the attention computation throughout the model.
2. Multi-token value features
Next I trained a 8192 value MTC (multi token transcoder) on each layer of GPT-2. The transcoder learns what information should be moved from source token positions to destination token positions. It reads the same normalized residual stream activations as the query-key transcoder and produces sparse activations, aVs,f for every residual stream vector. Its decoder has one 64-vector per feature per head, plus a per-head bias. This gives v̂s,h = bVh + Σf aVs,f uVf,h. During training I route these decoded values with the teacher’s frozen attention pattern so a source feature is transported to its destination token as per the model's native attention pattern routing.
ẑt,h = Σs≤t A*h,t,s v̂s,h
Lrecon = λ · mean[(flatten(ẑ) − flatten(z*))²].
The transcoder's output reconstruction is z hat and has shape [T, 12, 64]. It reproduces each of the 12 attention heads' 64-dimensional activation. The loss measures how well it reproduces the flattened z activation in GPT-2 which has shape, [T, 768] (where 12*64 = 768). A MTC feature that activates on a source token can contribute to each of the 12 heads for a particular destination token, and the contribution is determined by the transcoders' decoder or write direction.
3. MLP features
For each MLP I use the publicly released GPT-2 small transcoders from Dunefsky, Chlenski and Nanda. Each has 24,576 features. It reads the standardized post-attention residual y (before the second LayerNorm’s learned gain and bias), and predicts the native MLP output at the same position:
aMt = ReLU(ytEM + cM), m̂t = bM + Σf aMt,f uMf.
Now we have a means to replace each uninterpretable component (MLP and attentional units) in GPT-2 with monosemantic, interpretable features at every layer in the model. To identify a subset of features that are salient to a model's output for a given prompt, we need some way to trace how a model uses the replaced transcoder features to compute its output.
Attribution Graphs
An attribution graph is a tool used to visualize and map the internal steps a model takes when turning a specific prompt into output logits. They break down opaque black box LLM behaviour into a set of active features as they pertain to the input tokens and output logits. Edges between nodes are defined by the Jacobian a · ∂Jb/∂a. For some edge between A --> B, A writes something that B reads and the edge is determined by how much that write pushes B's objective up or down on a given prompt. Here a is the feature’s activation. Starting from a target node (such as a specific token logit or an active feature), the computation traces backward using the Jacobian matrix to measure how earlier nodes influence later ones.
If I simply stack 12 sets of imperfect predictions, an early error changes what every later encoder sees and errors accumulate. This is why nodes are input tokens, output logits, features and error nodes. The error nodes for the query-key transcoder, MTC, and MLP transcoders respectively are given by:
εS = S* − Ŝ on allowed source–destination pairs; εz = z* − A* v̂ at each destination and head; εM = m* − m̂ at each token.
That is the sense in which this is a complete replacement model: learned branches cover both sublayers of every block, with explicit accounts for what they miss. Token and position embeddings, residual connections, normalization, output projections and unembedding remain part of the scaffold. Thus during a forward pass, error nodes keep the native GPT-2 logit predictions faithful to the replaced forward pass with transcoder features. Additionally, rather than freezing the attention pattern, the attention computation is kept dynamic and can change inside the attribution graph. This way, query and key features can be traced directly from slight changes in the attention pattern and I can identify whether a change in attention routing matters for a particular output logit.
Note that attribution graphs to find circuits of features exists and was popularized by Anthropic - Tracing Attention Computation Through Feature Interactions. However, an attribution graph for a single prompt can highlight many features, some of which may not be especially relevant to the model's computation. What I introduce is a method that curates a small subset of features that matter most for a particular model behaviour.
For instance, after “The box is too heavy”, why does GPT-2 prefer to continue with to rather than that? A graph of this one prompt might turn up features specific to heavy or box, but are irrelevant to the grammatical nuance that predicts to rather than that. What I want to find is the part of the computation that also matters after “The room is too small” and “The path is too narrow”.
From one prediction to a family of feature graphs
For each of the aforementioned 3 prompts, I start with a separate attribution graph for each of the 3 prompts and define D = logit(“ to”) − logit(“ that”) for all 3. Thereby identifying the part of the model computation I'm interested in. A positive D means the model prefers to over that.
First, give the prompts a common set of candidates. Within each prompt, I nominate active query, key, value and MLP features based on edges in the attribution graph. Then I find the features that activate across multiple prompts, denoting them as potential features that are integral to the 'to' - 'that' mechanism. I match a feature by its layer, its kind (query, key, value, or MLP), and its ID. I count how many distinct prompts retain the feature after pruning the attribution graph.
For example, layer 8 key feature 888 and layer 8 value feature 7733 each had a surviving attribution edge into layer 8 head 4 in all three prompts. That gives me a specific question to test: does removing the key feature’s contribution to that head change the preference for the output logit? What about removing the value feature’s contribution? In all, the shared graphs yielded 50 feature interventions to try. The native GPT-2 ablation in the next section retained 16 at τ = 0.05. Those are two different findings: the graphs tell me which features repeatedly appear on influential paths; the ablations tell me which removals actually move the prediction by the amount I specified.
τ ablation: which proposed features matter in native GPT-2?
Identifying candidate features across multiple prompts reduces the number of features that are relevant to a particular output logit direction for a family of prompts, but it doesn't answer how relevant the feature is. We need some way to measure how salient a feature is to the native GPT-2 internal computation. This is where the feature ablations enter the methodology.
What gets ablated? On “The box is too heavy”, the value transcoder assigns V7733 an activation of 7.23 at the word too in the clean run. Its decoder gives me a 64-dimensional vector u for head 4. When native GPT-2 computes that head’s value vector v at too, I replace it with v − 7.23u. I’ve subtracted precisely the write that the transcoder assigned to this feature at this position and head.
The same operation applies to the other branches. For a query I subtract the feature’s write from the native query at the destination; for a key or value I subtract it at the source, where it can affect any destination that reads it. If a feature has surviving connections to several heads at a position, I test those head slices together. An MLP feature has no head index: I subtract its 768-dimensional write from the native MLP output at its active positions.
What earns a feature a place in the subset?
I start with an empty set S, and features are iteratively added to the set if ablating the feature from the GPT-2 forward pass decreases the logit difference (D = logit(positive token) − logit(negative token)) by some amount greater than τ. Specifically I compare a forward pass on GPT-2 with the set S of features removed, and another forward pass with set S plus feature f removed. The difference is the extra drop caused by this particular feature's removal:
δp(f | S) = Dp(S) − Dp(S ∪ {f}).
I average the ablation effect across all prompts in the family of prompts and retain a feature in the subset if the average drop is greater than τ. Feature retention means every subsequent ablation trial includes this feature in the mass removal. If a feature's average drop is not greater than τ, I leave it out and test the next candidate feature against the same set as before.
This process defines a salient subset of features for a particular model behaviour. If the model's behaviour of interest is defined by the logit difference, then subtracting a feature's decoder write direction from the model's native activation must make a measurable additional difference to the family of prompt's prediction. Figure 1 shows the subset this search found for the family of prompts "The box is too heavy", "The room is too small", "The path is too narrow", for D = logit(“ to”) − logit(“ that”).
Figure 1. The retained graph at τ = 0.05 for “The box is too heavy,” “The room is too small” and “The path is too narrow.” Read from earlier layers at the bottom. The 3/3 badges count discovery prompts. Q - query feature, K - key feature, V - value feature and M - MLP feature.
Model steering using the feature subset
Once a subset of features has been identified, one way of testing if this array of features is actually relevant to the model's internal computation is by steering the model. We can steer the model in the direction of these features to bring out a behaviour on a prompt that lacks the behaviour of interest. For instance, GPT-2 reads "The task is so difficult" and predicts that. Can I make it predict to by leveraging the set of features retained on the family of prompts “The box is too heavy,” “The room is too small” and “The path is too narrow".
I run two forward passes, the first being the donor on the too prompt and retain the activations for the subset of features. The second pass runs the so prompt or the recipient. During this second pass I replace selected features' current contribution with the donor activation: (donor activation − current recipient activation) × decoder vector.
By transplanting the 16 retained features, the to-that logit difference moved from −1.183 to +0.740 and the top prediction changed from that to to. Model steering shows that for a potential feature circuit, the same features acting jointly transfers that preference to a new prompt.
Case Studies
Neither / either contrast pair
I found a seven-feature set on the nor − or contrast at τ = 0.40. It contained L0 MLP6426 and MLP14402, L1 MLP6988, L2 MLP22278, and three layer-9 features connected to head 10: K626, V1402 and V7171. Subtracting these feature directions swapped logit direction on all 3 prompts: “They wanted neither bread,” “She chose neither wine,” and “He bought neither rice.” By removing these features from the GPT-2 forward pass, all 3 prompts predict 'or' instead of 'nor'. Similarly, either-donor steering into a neither-recipient reversed 4/4 matched pairs. For “We preferred neither salmon,” the matched either donor moved D from +3.373 to −0.741 and the overall next-token prediction from nor to or.
The two layer-0 MLP features made it possible to ask a cleaner question. I locked just those two IDs, matched the actual token neither, and compared each removal alone and both together on three new prompts. Here is the bread example:
At the token “neither”
D = nor − or
Top next token
Native GPT-2
+3.101
nor
Remove L0 MLP6426
+2.483
nor
Remove L0 MLP14402
+2.786
nor
Remove both
−1.209
and
This demonstrates that features can act jointly, where multiple features glean distinct information relevant to some underlying mechanism. By removing only one feature, the underlying mechanism can remain mostly intact, but joint removal completely erases the model's ability to reproduce said mechanism.
Figure 2. Each bar is a native GPT-2 forward pass. The joint removal crossed zero on bread, wine and rice.
Either / both: a candidate route from cue to final noun
Here D = or − and. In “She wanted either tea,” L8 V7783 writes at either, and L8 Q1314 writes at the final noun, tea, in head 5. Layer 9 contains another such arrangement: V1633 at the cue and Q938 at the final noun in head 10. L9 MLP12401 and L10 MLP171 and MLP3189 write at the final noun. This gives the graph a concrete interpretation: information available at the cue can be routed to the position where the continuation is predicted. The measurements below test the whole group; they don’t isolate either individual head as the required route.
Figure 3. Four edges from the retained “or − and” subgraph for “She wanted either tea” at τ = 0.15. Read each arrow from a feature firing to a head at its destination.
The 11-feature set at τ = 0.15 reversed 3/3 discovery prompts (reversal means logit difference went from positive to negative). When I transferred the ablation scope, L10 That eleven-feature intervention reversed 5/5 evaluated prompts, including two new ones, and made and top-ranked on all five.
With the twelve-feature steering group, either donors reversed 3/4 both recipients; both donors reversed 2/4 either recipients. One pair worked both ways. “She wanted both tea” went from −7.108 to +0.685 with the either donor, changing the top word from and to or. The reciprocal transplant into “She wanted either tea” moved +3.715 to −0.463, changing or back to and.
Too / so: the most consistent steering result
“The task is too difficult” invites to (“to finish”). “The task is so difficult” invites that (“that nobody finished”). I kept D = to − that fixed. The group at τ = 0.05 had 16 feature identities: ten MLP, one query, four key and one value. On a representative prompt, L0 MLP20928 fired at too, L1 Q2179 wrote into head 3 at the final adjective, and L8 K888 and V7733 both wrote into head 4 at the cue.
Subtracting all sixteen features flipped only 1/3 discovery prompts. “The room is too small” went from +5.234 to −1.895, but the box and path prompts stayed positive. Steering gave a different result: too donor → so recipient reversed 5/5 pairs, including both held-out lexical variants.
Figure 4. The same sixteen feature identities transferred in both directions. Open circles mark the native recipient margin and diamonds mark the result after replacement. The asterisks mark prompts held out of feature selection.
Discussion
The question I started with was how a word near the beginning of a prompt can change a prediction several tokens later. After “She wanted either tea”, for example, what carries information about either to the point where GPT-2 favors or? I want a feature subgraph that explains those steps. In this case the graph points to a specific route worth testing. Layer 8 value feature 7783 fires at either, and head 5 reads from that position at tea, where the model makes its prediction.
To find a subgraph like this, I choose two possible next tokens and build a separate attribution graph for their logit difference on each prompt in a family. I keep features and connections that recur as the wording changes. Then I test each proposed feature in native GPT-2 by subtracting its decoded contribution at the token positions and heads identified by the graph. The positive τ sweep keeps the removal when it lowers the family’s average margin by more than τ. This gives me a candidate feature circuit that plausibly explains a GPT-2 behaviour as it pertains to a specified logit difference.
The primary limitation of this approach is the absence of proof of a comprehensive circuit. The ablation and model steering tests highlight that a given subset of features are important for computing a particular model mechanism, but they don't prove that a circuit of features are minimal, sufficient, or complete. The main area of future work would revolve around conducting tests that demonstrate minimality, sufficiency, and completeness for a proposed feature circuit.
Neuronal superposition is the phenomena where a neuron in a model activates on many unrelated concepts. This is a problem for interpretability research because you don't have a 1:1 mapping of a neuron and a clean, human-interpretable feature. Sparse dictionary learning offers a potential solution to this problem whereby a neuron in a model is mapped, through an autoencoder, to each of the features it activates. It offers a method to distill the many features embedded in a single neuron to a single feature activation.
Now imagine you could decompose each component of a model (the MLP layers and attentional units) into a sparse dictionary of features such that for a single input prompt, you have a mapping of which native model components correspond to which sparse dictionary features. The dream scenario is each of the autoencoders' features are monosemantic, interpretable, and you could replace each component (i.e. layer 5 attention head 6) with these features; thereby decomposing a black box into interpretable autoencoder parts.
The goal of this research direction to start with a behaviour in GPT-2, replace GPT-2 with transcoders (an autoencoder variant), and find the relevant features that produce this behaviour. In other words, identifying feature circuits that decompose a model into interpretable subparts that together define some behaviour in GPT-2.
What is being replaced?
The method to find a feature circuit starts with replacing each attentional unit and MLP component at every layer in GPT-2 with a transcoder. Usually a transcoder is applied to the MLP of a model and learns to reproduce the activations post MLP from the pre MLP activations. However, because the attention mechanism mixes information across token positions, a different transcoder architecture is needed. I use 3 transcoder architectures to fully replace GPT-2, 1 to replace the query-key circuit, 1 to replace the value circuit, and 1 to replace the MLP components.
1. Query and key features
The query-key transcoder follows my earlier post. They are two separate transcoders; the query transcoder learns a sparse set of features for every token and the key transcoder does the same. In practice L0 (the number of features active per token) ranges from about 10-20 features. Generally, earlier layer transcoders have L0 on the small side of this range and later layers require more features to reconstruct the attention pattern.
The activation of a feature for either the query or key transcoder is given by:
aBt,f = max(0, xt · eBf + cBf).
where eBf is the encoder or the transcoder's read direction. The decoder or write direction determines how much a feature contributes to the output. For a single attention head h, the reconstructed query and key are
q̂t,h = bQh + Σi aQt,i uQi,h, k̂s,h = bKh + Σj aKs,j uKj,h.
and the transcoder's efficacy determines how well the final attention pattern is reproduced by one head:
Ŝh,t,s = q̂t,h · k̂s,h / 8
So, the query-key transcoder takes normalized residual stream activations as the input and learns to reconstruct every attention head's attention pattern using learned features from separate query and key transcoders. These transcoders are trained on every layer of GPT-2 to replace the QK part of the attention computation throughout the model.
2. Multi-token value features
Next I trained a 8192 value MTC (multi token transcoder) on each layer of GPT-2. The transcoder learns what information should be moved from source token positions to destination token positions. It reads the same normalized residual stream activations as the query-key transcoder and produces sparse activations, aVs,f for every residual stream vector. Its decoder has one 64-vector per feature per head, plus a per-head bias. This gives v̂s,h = bVh + Σf aVs,f uVf,h. During training I route these decoded values with the teacher’s frozen attention pattern so a source feature is transported to its destination token as per the model's native attention pattern routing.
ẑt,h = Σs≤t A*h,t,s v̂s,h
Lrecon = λ · mean[(flatten(ẑ) − flatten(z*))²].
The transcoder's output reconstruction is z hat and has shape [T, 12, 64]. It reproduces each of the 12 attention heads' 64-dimensional activation. The loss measures how well it reproduces the flattened z activation in GPT-2 which has shape, [T, 768] (where 12*64 = 768). A MTC feature that activates on a source token can contribute to each of the 12 heads for a particular destination token, and the contribution is determined by the transcoders' decoder or write direction.
3. MLP features
For each MLP I use the publicly released GPT-2 small transcoders from Dunefsky, Chlenski and Nanda. Each has 24,576 features. It reads the standardized post-attention residual y (before the second LayerNorm’s learned gain and bias), and predicts the native MLP output at the same position:
Now we have a means to replace each uninterpretable component (MLP and attentional units) in GPT-2 with monosemantic, interpretable features at every layer in the model. To identify a subset of features that are salient to a model's output for a given prompt, we need some way to trace how a model uses the replaced transcoder features to compute its output.
Attribution Graphs
An attribution graph is a tool used to visualize and map the internal steps a model takes when turning a specific prompt into output logits. They break down opaque black box LLM behaviour into a set of active features as they pertain to the input tokens and output logits. Edges between nodes are defined by the Jacobian a · ∂Jb/∂a. For some edge between A --> B, A writes something that B reads and the edge is determined by how much that write pushes B's objective up or down on a given prompt. Here a is the feature’s activation. Starting from a target node (such as a specific token logit or an active feature), the computation traces backward using the Jacobian matrix to measure how earlier nodes influence later ones.
If I simply stack 12 sets of imperfect predictions, an early error changes what every later encoder sees and errors accumulate. This is why nodes are input tokens, output logits, features and error nodes. The error nodes for the query-key transcoder, MTC, and MLP transcoders respectively are given by:
εS = S* − Ŝ on allowed source–destination pairs;
εz = z* − A* v̂ at each destination and head;
εM = m* − m̂ at each token.
That is the sense in which this is a complete replacement model: learned branches cover both sublayers of every block, with explicit accounts for what they miss. Token and position embeddings, residual connections, normalization, output projections and unembedding remain part of the scaffold. Thus during a forward pass, error nodes keep the native GPT-2 logit predictions faithful to the replaced forward pass with transcoder features. Additionally, rather than freezing the attention pattern, the attention computation is kept dynamic and can change inside the attribution graph. This way, query and key features can be traced directly from slight changes in the attention pattern and I can identify whether a change in attention routing matters for a particular output logit.
Note that attribution graphs to find circuits of features exists and was popularized by Anthropic - Tracing Attention Computation Through Feature Interactions. However, an attribution graph for a single prompt can highlight many features, some of which may not be especially relevant to the model's computation. What I introduce is a method that curates a small subset of features that matter most for a particular model behaviour.
For instance, after “The box is too heavy”, why does GPT-2 prefer to continue with to rather than that? A graph of this one prompt might turn up features specific to heavy or box, but are irrelevant to the grammatical nuance that predicts to rather than that. What I want to find is the part of the computation that also matters after “The room is too small” and “The path is too narrow”.
From one prediction to a family of feature graphs
For each of the aforementioned 3 prompts, I start with a separate attribution graph for each of the 3 prompts and define D = logit(“ to”) − logit(“ that”) for all 3. Thereby identifying the part of the model computation I'm interested in. A positive D means the model prefers to over that.
First, give the prompts a common set of candidates. Within each prompt, I nominate active query, key, value and MLP features based on edges in the attribution graph. Then I find the features that activate across multiple prompts, denoting them as potential features that are integral to the 'to' - 'that' mechanism. I match a feature by its layer, its kind (query, key, value, or MLP), and its ID. I count how many distinct prompts retain the feature after pruning the attribution graph.
For example, layer 8 key feature 888 and layer 8 value feature 7733 each had a surviving attribution edge into layer 8 head 4 in all three prompts. That gives me a specific question to test: does removing the key feature’s contribution to that head change the preference for the output logit? What about removing the value feature’s contribution? In all, the shared graphs yielded 50 feature interventions to try. The native GPT-2 ablation in the next section retained 16 at τ = 0.05. Those are two different findings: the graphs tell me which features repeatedly appear on influential paths; the ablations tell me which removals actually move the prediction by the amount I specified.
τ ablation: which proposed features matter in native GPT-2?
Identifying candidate features across multiple prompts reduces the number of features that are relevant to a particular output logit direction for a family of prompts, but it doesn't answer how relevant the feature is. We need some way to measure how salient a feature is to the native GPT-2 internal computation. This is where the feature ablations enter the methodology.
What gets ablated? On “The box is too heavy”, the value transcoder assigns V7733 an activation of 7.23 at the word too in the clean run. Its decoder gives me a 64-dimensional vector u for head 4. When native GPT-2 computes that head’s value vector v at too, I replace it with v − 7.23u. I’ve subtracted precisely the write that the transcoder assigned to this feature at this position and head.
The same operation applies to the other branches. For a query I subtract the feature’s write from the native query at the destination; for a key or value I subtract it at the source, where it can affect any destination that reads it. If a feature has surviving connections to several heads at a position, I test those head slices together. An MLP feature has no head index: I subtract its 768-dimensional write from the native MLP output at its active positions.
What earns a feature a place in the subset?
I start with an empty set S, and features are iteratively added to the set if ablating the feature from the GPT-2 forward pass decreases the logit difference (D = logit(positive token) − logit(negative token)) by some amount greater than τ. Specifically I compare a forward pass on GPT-2 with the set S of features removed, and another forward pass with set S plus feature f removed. The difference is the extra drop caused by this particular feature's removal:
δp(f | S) = Dp(S) − Dp(S ∪ {f}).
I average the ablation effect across all prompts in the family of prompts and retain a feature in the subset if the average drop is greater than τ. Feature retention means every subsequent ablation trial includes this feature in the mass removal. If a feature's average drop is not greater than τ, I leave it out and test the next candidate feature against the same set as before.
This process defines a salient subset of features for a particular model behaviour. If the model's behaviour of interest is defined by the logit difference, then subtracting a feature's decoder write direction from the model's native activation must make a measurable additional difference to the family of prompt's prediction. Figure 1 shows the subset this search found for the family of prompts "The box is too heavy", "The room is too small", "The path is too narrow", for D = logit(“ to”) − logit(“ that”).
Figure 1. The retained graph at τ = 0.05 for “The box is too heavy,” “The room is too small” and “The path is too narrow.” Read from earlier layers at the bottom. The 3/3 badges count discovery prompts. Q - query feature, K - key feature, V - value feature and M - MLP feature.
Model steering using the feature subset
Once a subset of features has been identified, one way of testing if this array of features is actually relevant to the model's internal computation is by steering the model. We can steer the model in the direction of these features to bring out a behaviour on a prompt that lacks the behaviour of interest. For instance, GPT-2 reads "The task is so difficult" and predicts that. Can I make it predict to by leveraging the set of features retained on the family of prompts “The box is too heavy,” “The room is too small” and “The path is too narrow".
I run two forward passes, the first being the donor on the too prompt and retain the activations for the subset of features. The second pass runs the so prompt or the recipient. During this second pass I replace selected features' current contribution with the donor activation: (donor activation − current recipient activation) × decoder vector.
By transplanting the 16 retained features, the to-that logit difference moved from −1.183 to +0.740 and the top prediction changed from that to to. Model steering shows that for a potential feature circuit, the same features acting jointly transfers that preference to a new prompt.
Case Studies
Neither / either contrast pair
I found a seven-feature set on the nor − or contrast at τ = 0.40. It contained L0 MLP6426 and MLP14402, L1 MLP6988, L2 MLP22278, and three layer-9 features connected to head 10: K626, V1402 and V7171. Subtracting these feature directions swapped logit direction on all 3 prompts: “They wanted neither bread,” “She chose neither wine,” and “He bought neither rice.” By removing these features from the GPT-2 forward pass, all 3 prompts predict 'or' instead of 'nor'. Similarly, either-donor steering into a neither-recipient reversed 4/4 matched pairs. For “We preferred neither salmon,” the matched either donor moved D from +3.373 to −0.741 and the overall next-token prediction from nor to or.
The two layer-0 MLP features made it possible to ask a cleaner question. I locked just those two IDs, matched the actual token neither, and compared each removal alone and both together on three new prompts. Here is the bread example:
At the token “neither”
D = nor − or
Top next token
Native GPT-2
+3.101
nor
Remove L0 MLP6426
+2.483
nor
Remove L0 MLP14402
+2.786
nor
Remove both
−1.209
and
This demonstrates that features can act jointly, where multiple features glean distinct information relevant to some underlying mechanism. By removing only one feature, the underlying mechanism can remain mostly intact, but joint removal completely erases the model's ability to reproduce said mechanism.
Figure 2. Each bar is a native GPT-2 forward pass. The joint removal crossed zero on bread, wine and rice.
Either / both: a candidate route from cue to final noun
Here D = or − and. In “She wanted either tea,” L8 V7783 writes at either, and L8 Q1314 writes at the final noun, tea, in head 5. Layer 9 contains another such arrangement: V1633 at the cue and Q938 at the final noun in head 10. L9 MLP12401 and L10 MLP171 and MLP3189 write at the final noun. This gives the graph a concrete interpretation: information available at the cue can be routed to the position where the continuation is predicted. The measurements below test the whole group; they don’t isolate either individual head as the required route.
Figure 3. Four edges from the retained “or − and” subgraph for “She wanted either tea” at τ = 0.15. Read each arrow from a feature firing to a head at its destination.
The 11-feature set at τ = 0.15 reversed 3/3 discovery prompts (reversal means logit difference went from positive to negative). When I transferred the ablation scope, L10 That eleven-feature intervention reversed 5/5 evaluated prompts, including two new ones, and made and top-ranked on all five.
With the twelve-feature steering group, either donors reversed 3/4 both recipients; both donors reversed 2/4 either recipients. One pair worked both ways. “She wanted both tea” went from −7.108 to +0.685 with the either donor, changing the top word from and to or. The reciprocal transplant into “She wanted either tea” moved +3.715 to −0.463, changing or back to and.
Too / so: the most consistent steering result
“The task is too difficult” invites to (“to finish”). “The task is so difficult” invites that (“that nobody finished”). I kept D = to − that fixed. The group at τ = 0.05 had 16 feature identities: ten MLP, one query, four key and one value. On a representative prompt, L0 MLP20928 fired at too, L1 Q2179 wrote into head 3 at the final adjective, and L8 K888 and V7733 both wrote into head 4 at the cue.
Subtracting all sixteen features flipped only 1/3 discovery prompts. “The room is too small” went from +5.234 to −1.895, but the box and path prompts stayed positive. Steering gave a different result: too donor → so recipient reversed 5/5 pairs, including both held-out lexical variants.
Figure 4. The same sixteen feature identities transferred in both directions. Open circles mark the native recipient margin and diamonds mark the result after replacement. The asterisks mark prompts held out of feature selection.
Discussion
The question I started with was how a word near the beginning of a prompt can change a prediction several tokens later. After “She wanted either tea”, for example, what carries information about either to the point where GPT-2 favors or? I want a feature subgraph that explains those steps. In this case the graph points to a specific route worth testing. Layer 8 value feature 7783 fires at either, and head 5 reads from that position at tea, where the model makes its prediction.
To find a subgraph like this, I choose two possible next tokens and build a separate attribution graph for their logit difference on each prompt in a family. I keep features and connections that recur as the wording changes. Then I test each proposed feature in native GPT-2 by subtracting its decoded contribution at the token positions and heads identified by the graph. The positive τ sweep keeps the removal when it lowers the family’s average margin by more than τ. This gives me a candidate feature circuit that plausibly explains a GPT-2 behaviour as it pertains to a specified logit difference.
The primary limitation of this approach is the absence of proof of a comprehensive circuit. The ablation and model steering tests highlight that a given subset of features are important for computing a particular model mechanism, but they don't prove that a circuit of features are minimal, sufficient, or complete. The main area of future work would revolve around conducting tests that demonstrate minimality, sufficiency, and completeness for a proposed feature circuit.