This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
TLDR
Goodfire's Neural Networks Are Already Clean argues that most neurons in modern neural network is more interpretable than previously what we previously. Even granting this claim, we think its implications for mechanistic interpretability is less clear.
We argue a neuron can encode a single coherent abstraction while still being difficult to describe. Some of these abstractions are sufficiently complex that assigning a single human readable feature label may obscure more than it explains.
More importantly, understanding what individual neurons respond to does not necessarily tell us how their interactions implement a computation. The relevant measure of progress would be whether a neuron level explanation help us predict and understand the model's downstream behavior.
Neural Networks Are Already Clean
Nick Cammarata's recent Goodfire post makes a an argument on what we previously understood as polysemanticity may actually be recursive generalization. Rather than representing arbitrary combinations of unrelated features, neurons often encode coherent abstractions that generalize across seemingly unrelated inputs. There has been considerable discussion about whether this claim is empirically correct, including in Neural Networks Are Not Already Clean which has been going viral on X for the past few days.
This article will be specifically be looking at the question : Even if Goodfire is correct about the nature of neurons, how much should this update our expectations for mechanistic interpretability?
For this post, I will assume Goodfire's central empirical claim to be true that most neurons implement coherent and non-arbitrary features. Our argument is that this alone does not establish that neurons provide a tractable explanatory basis for understanding neural network computations.
1. A coherent abstraction is not necessarily an interpretable one
Consider the following example from Cammarata's post:
Image from Nick Cammerata's post
The above is a specific example which would be a good fit to understand our claim.
One InceptionV1 neuron collects a pinched-center shape: bow ties, butterflies, swimsuit tops. But it also picks up things that go with them, men’s dress shoes with the bow ties and beach bags and flip-flops with the swimsuit tops, since they share neurons in earlier layers.
This is a particularly interesting example.
First, the neuron responds to objects sharing a recognizable structural feature which is a pinched center shape referencing bow ties, butterflies, and swimsuit tops which all exhibit this geometry. Second, the neuron also responds to semantically associated objects that do not necessarily exhibit the same shape such as men's dress shoes and women's beach accessories appear because of their relationships to the concepts detected through upstream circuitry.
Let us put aside how problematic it might be to rationalize an empirical result to back a hypothesis, and let us suppose that this neuron is genuinely monosemantic in Goodfire's broader sense that its activation patterns arise from a coherent process of recursive generalization rather than arbitrary feature superposition.
What exactly is the human interpretable feature represented by this neuron?
Calling it a "pinched-center shape detector" explains one part of its behavior but fails to account for its semantic associations. Expanding the description to include men's formal clothing and women's beach accessories captures more of the observed activations, but sacrifices the simplicity that made the original explanation very useful. We could instead describe the neuron as detecting a pinched-center structure together with semantically related concepts inherited through upstream circuits. But this introduces a problem where the explanation now depends on other possible interpretations of the neuron.
The neuron's interpretation is no longer a simple feature label but it is potentially a description of an entire computational neighborhood. While Goodfire explicitly argues that examining upstream and downstream circuits makes such neurons easier to understand, our concern is that the existence of a coherent explanation does not guarantee the existence of a compact, useful explanation. If interpreting an individual neuron requires tracing an increasingly complex hierarchy of semantic associations, the fact that its representation is technically monosemantic may provide less practical interpretability than the term initially suggests.
2. The distinction between describing a neuron and understanding its computation
Consider two possible explanations of the same neuron.
Explanation A: Activation-based description
The neuron activates a lot for bow ties, moderately for butterflies and swimsuit tops, and also activates a tiny bit for some example of formal shoes and beach accesories.
The neuron represents a pinched-center shape, with other representations coming through their association to pinched-center objects.
Explanation B is clearly more structured. It gives a possible reason for why seemingly unrelated concepts get grouped together.
However, this does not necessarily mean that Explanation B gives us substantially more mechanistic understanding.
For that, we need to ask whether it allows us to make predictions that Explanation A cannot.
For instance, can we use the recursive-generalization explanation to predict:
Which previously unseen inputs will activate the neuron?
Under which contexts will geometric associations dominate, and under which context will semantic associations dominate?
What changes in the model's output when the neuron is suppressed or amplified?
Which downstream computations actually depend on this neuron?
If Explanation B performs substantially better on these questions, then the abstraction has genuine explanatory value beyond a collection of observed activation patterns.
But if the explanation requires extensive case-by-case qualifications, or cannot predict causal effects with precision, then describing the neuron as monosemantic has not necessarily helped us understand its role in the model.
We believe Explanation B does not necessarily provide more answers. In fact, Explanation B is simply a byproduct of an "improved" understanding of the neuron.
3. Even interpretable neurons do not imply interpretable model computations
There is a second, more practical limitation.
Suppose we eventually succeed at coming up with producing precise explanations for almost every neuron in a frontier model.
We agree that this would be groundbreaking. However, we are skeptical as to whether this achievement would allow us to fully understand how the model's outputs are created.
The issue is the sheer scale of frontier LLMs. Not only we would need to find explanations for trillions of neurons, but we would also need to understand the interactions between each and every one of them.
Even if we found natural-language descriptions of these neurons, it would be nearly impossible for human researchers to inspect each and every one of them, as well as their countless possible interactions. It would be nearly impossible to interpret a single multi-billion forward pass coupled with its multi-billion neuron interactions.
In other words, knowing what each component represents is different from knowing how the components compose into a computation.
For example, suppose we can find the neurons associated with deception, user instructions, and uncertainty. Knowing that these neurons represent deception wouldn't help us understand why a model chooses to deceive a user. Instead, we would need to follow the flow of billions of other neurons to understand the causal chain.
Goodfire is not ignoring this problem. The post explicitly advocates studying circuits and tracing information flow through its native mechanisms. Cammarata also argues that understanding will compound as more parts of the network are understood.
We think this is a plausible research direction.
However, the claim that most neurons are coherent does not imply that mapping these neurons will make the model's behavior easier to explain.
The former may make the latter easier, but the latter might still be a herculean task.
4. What would convince us?
We think it would be useful to see if monosemantic neuron explanations can help provide better understanding under a fixed budget. Specifically, we would like to see a comparison between:
An explanation built from native neurons through recursive generalization.
An explanation built from other feature decompositions like SAEs.
For this experiment, we would ask which of the two provides more precise predictions of downstream behavior with the same amount of information and investigative effort.
For example, given an explanation of a small circuit, which method would allow researchers to better predict what happens when components are ablated, amplified, or replaced?
If recursive generalization allows researchers to come up with accurate predictions given neuron explanations, this would be a good signal for Goodfire's post improving mechanistic interpretability.
If instead the explanations become increasingly complicated and require extensive context to predict behavior, that would support our concern.
This would help distinguish whether neuron explanations formed through recursive generalization can be practically useful in understanding neural networks.
Conclusion
We think Goodfire's hypotheses are potentially important, especially if they eventually lead to a new understanding of neural networks that is not based on the idea that NNs pack a lot of features together through superposition.
However, we don't think this new notion of most neurons being monosemantic means that neural networks are now much easier to understand.
Take as an example the bow-tie neuron. Firstly, we believe find such recursive generalizations for all neurons is an arduous process. However, even for neurons for which we have found such explanations, their actual behaviors will look to us like a combination of multiple semantic associations. Thus, describing neuron behavior might require more than one feature label.
And even if we manage to explain individual neurons, we still need a scalable way to understand how their interactions produce model behavior.
Our central claim is that while monosemanticity through recursive generalization does solve some issues like tangled representations, it does not solve other difficulties of interpretability, such as explanation complexity or causal understanding.
The real test of whether neural networks are becoming more interpretable is to find how effectively these descriptions let us predict and explain the computations that matter.
TLDR
Neural Networks Are Already Clean
Nick Cammarata's recent Goodfire post makes a an argument on what we previously understood as polysemanticity may actually be recursive generalization. Rather than representing arbitrary combinations of unrelated features, neurons often encode coherent abstractions that generalize across seemingly unrelated inputs. There has been considerable discussion about whether this claim is empirically correct, including in Neural Networks Are Not Already Clean which has been going viral on X for the past few days.
This article will be specifically be looking at the question : Even if Goodfire is correct about the nature of neurons, how much should this update our expectations for mechanistic interpretability?
For this post, I will assume Goodfire's central empirical claim to be true that most neurons implement coherent and non-arbitrary features. Our argument is that this alone does not establish that neurons provide a tractable explanatory basis for understanding neural network computations.
1. A coherent abstraction is not necessarily an interpretable one
Consider the following example from Cammarata's post:
Image from Nick Cammerata's post
The above is a specific example which would be a good fit to understand our claim.
This is a particularly interesting example.
First, the neuron responds to objects sharing a recognizable structural feature which is a pinched center shape referencing bow ties, butterflies, and swimsuit tops which all exhibit this geometry. Second, the neuron also responds to semantically associated objects that do not necessarily exhibit the same shape such as men's dress shoes and women's beach accessories appear because of their relationships to the concepts detected through upstream circuitry.
Let us put aside how problematic it might be to rationalize an empirical result to back a hypothesis, and let us suppose that this neuron is genuinely monosemantic in Goodfire's broader sense that its activation patterns arise from a coherent process of recursive generalization rather than arbitrary feature superposition.
What exactly is the human interpretable feature represented by this neuron?
Calling it a "pinched-center shape detector" explains one part of its behavior but fails to account for its semantic associations. Expanding the description to include men's formal clothing and women's beach accessories captures more of the observed activations, but sacrifices the simplicity that made the original explanation very useful. We could instead describe the neuron as detecting a pinched-center structure together with semantically related concepts inherited through upstream circuits. But this introduces a problem where the explanation now depends on other possible interpretations of the neuron.
The neuron's interpretation is no longer a simple feature label but it is potentially a description of an entire computational neighborhood. While Goodfire explicitly argues that examining upstream and downstream circuits makes such neurons easier to understand, our concern is that the existence of a coherent explanation does not guarantee the existence of a compact, useful explanation. If interpreting an individual neuron requires tracing an increasingly complex hierarchy of semantic associations, the fact that its representation is technically monosemantic may provide less practical interpretability than the term initially suggests.
2. The distinction between describing a neuron and understanding its computation
Consider two possible explanations of the same neuron.
Explanation A: Activation-based description
The neuron activates a lot for bow ties, moderately for butterflies and swimsuit tops, and also activates a tiny bit for some example of formal shoes and beach accesories.
Explanation B: Recursive-generalization description
The neuron represents a pinched-center shape, with other representations coming through their association to pinched-center objects.
Explanation B is clearly more structured. It gives a possible reason for why seemingly unrelated concepts get grouped together.
However, this does not necessarily mean that Explanation B gives us substantially more mechanistic understanding.
For that, we need to ask whether it allows us to make predictions that Explanation A cannot.
For instance, can we use the recursive-generalization explanation to predict:
If Explanation B performs substantially better on these questions, then the abstraction has genuine explanatory value beyond a collection of observed activation patterns.
But if the explanation requires extensive case-by-case qualifications, or cannot predict causal effects with precision, then describing the neuron as monosemantic has not necessarily helped us understand its role in the model.
We believe Explanation B does not necessarily provide more answers. In fact, Explanation B is simply a byproduct of an "improved" understanding of the neuron.
3. Even interpretable neurons do not imply interpretable model computations
There is a second, more practical limitation.
Suppose we eventually succeed at coming up with producing precise explanations for almost every neuron in a frontier model.
We agree that this would be groundbreaking. However, we are skeptical as to whether this achievement would allow us to fully understand how the model's outputs are created.
The issue is the sheer scale of frontier LLMs. Not only we would need to find explanations for trillions of neurons, but we would also need to understand the interactions between each and every one of them.
Even if we found natural-language descriptions of these neurons, it would be nearly impossible for human researchers to inspect each and every one of them, as well as their countless possible interactions. It would be nearly impossible to interpret a single multi-billion forward pass coupled with its multi-billion neuron interactions.
In other words, knowing what each component represents is different from knowing how the components compose into a computation.
For example, suppose we can find the neurons associated with deception, user instructions, and uncertainty. Knowing that these neurons represent deception wouldn't help us understand why a model chooses to deceive a user. Instead, we would need to follow the flow of billions of other neurons to understand the causal chain.
Goodfire is not ignoring this problem. The post explicitly advocates studying circuits and tracing information flow through its native mechanisms. Cammarata also argues that understanding will compound as more parts of the network are understood.
We think this is a plausible research direction.
However, the claim that most neurons are coherent does not imply that mapping these neurons will make the model's behavior easier to explain.
The former may make the latter easier, but the latter might still be a herculean task.
4. What would convince us?
We think it would be useful to see if monosemantic neuron explanations can help provide better understanding under a fixed budget. Specifically, we would like to see a comparison between:
For this experiment, we would ask which of the two provides more precise predictions of downstream behavior with the same amount of information and investigative effort.
For example, given an explanation of a small circuit, which method would allow researchers to better predict what happens when components are ablated, amplified, or replaced?
If recursive generalization allows researchers to come up with accurate predictions given neuron explanations, this would be a good signal for Goodfire's post improving mechanistic interpretability.
If instead the explanations become increasingly complicated and require extensive context to predict behavior, that would support our concern.
This would help distinguish whether neuron explanations formed through recursive generalization can be practically useful in understanding neural networks.
Conclusion
We think Goodfire's hypotheses are potentially important, especially if they eventually lead to a new understanding of neural networks that is not based on the idea that NNs pack a lot of features together through superposition.
However, we don't think this new notion of most neurons being monosemantic means that neural networks are now much easier to understand.
Take as an example the bow-tie neuron. Firstly, we believe find such recursive generalizations for all neurons is an arduous process. However, even for neurons for which we have found such explanations, their actual behaviors will look to us like a combination of multiple semantic associations. Thus, describing neuron behavior might require more than one feature label.
And even if we manage to explain individual neurons, we still need a scalable way to understand how their interactions produce model behavior.
Our central claim is that while monosemanticity through recursive generalization does solve some issues like tangled representations, it does not solve other difficulties of interpretability, such as explanation complexity or causal understanding.
The real test of whether neural networks are becoming more interpretable is to find how effectively these descriptions let us predict and explain the computations that matter.