This is a post containing reflections on Goodfire's recent release, "Neural Networks Are Already Clean", which argues neuron-based interpretability may be easier than we think. The essay is a summary of a technical paper that will come out later this year. Also featuring thoughts on where this research will lead, plus speculative ideas about neuron monosemanticity
(What I base views expressed in this post on: earlier this year, I worked on neuron-based interpretability for LLMs (we recently published a paper). We mostly looked at low-level properties of the neuron basis and didn’t do many case studies or scale up automated circuit interpretation. I also thought about the problem for a long time and have some hypotheses based on results we’ve seen that are likely not strictly true.)
I’ll first discuss what I see as the implications of this work for LLM interpretability research and then talk about why neurons may be a useful basis: I wrote down some guesses (in a collapsible section so as to not distract from the main point) and then some directions that may be useful to study for basic science.
In short, I think the agenda presented in the essay is promising and worth pursuing even outside of direct applications to language model interpretability. I have some object-level uncertainty about results for LLMs and will wait for the final paper and followup work.
How useful is this?
What I see as the fundamental claim here is that resolving some unforced errors in visualizing neuron activation --
properly thresholding neurons
considering both poles of neurons separately
taking into account the fact that neurons often activate on seemingly unrelated inputs (for example, by clustering neuron activations as in the paper)
-- and then constructing circuits with neurons as features leads to the best method we have for understanding language model computation at a low level.
With this kind of processing, you get an “overlay” for the model, which doesn’t explain how the model works but gives you points at which you can observe and intervene on the prediction. A neuron attribution graph is basically a collection of probes and steering vectors per layer, which hopefully can also give you sparse causal graphs after you group nodes into small sets, supernodes, as in the attribution graphs paper.
Again, transcoders and SAEs can also do this job, but we don’t have a good reason to believe directions these methods find are stable units we can build on top of, while neurons are native to the model.
I say this is an overlay because I don’t think high activations of neurons have enough expressivity for actually explaining the functioning of the model in gears-level detail. The meanings we assign to them are probably not specific enough, they may interact in ways we don’t yet understand, and some useful behavior (like memorization) could be contributed by low-activating weights.
For example, in “Already Clean”, the wallet graph is very low-resolution and doesn’t explain to us why the model thinks about honesty in connection to the wallet -- why the connection from one to the other is there and if there were other possible thoughts. However, it does give us handles for “honesty” and “returning” in this specific situation and tells us some nontrivial things about the model’s computation.
An attribution graph from Already Clean, section "Exploring Language Models"
There are probably still many cases where we can fully understand high activations of neurons and describe useful functionality, like arithmetic in Llama 3.1 8B.
This kind of overlay can be as useful for steering and probing as diff-of-means vectors, and basically gives you a library of steering vectors. We can have refusal, truth, and character neurons with direct applications, as demonstrated by Sauers et al,[1] though those exact neurons may not be neurons that specifically represent personas, but just ones that happen to respond with a specific persona on specific prompts.
It’s possible most concepts that can be represented as diff-of-means vectors also have a sparse representation in the neuron basis and that the neuron basis is unusually well-suited for representing steering vectors. There is some evidence for this.
I’m unsure how this scales to bigger models. I have had less success steering MoE models by editing individual neurons, which makes sense given neurons need to have their expert activate to have any effect, and that can be more rare. It’s possible you run into some sort of resolvable superposition, where individual neurons no longer steer as well but there is some set of neurons that together makes up a steerable unit in some fundamental way.
Neurons are meaningfully cheaper to apply to many different models compared to dictionary learning methods, at least when it comes to allowing hobbyists to do not-very-rigorous projects with attribution graphs. But I don’t think we can say neurons are cheaper for basic science in general because attribution graphs constructed from neurons are less sparse, and the neurons have less clear meanings, so LLM agents are basically mandatory for faithfully interpreting them.[2]
This is still a big step forward for gears-level interpretability and I’m excited for the tools the technical paper for the essay will probably describe.
I don’t know if we can really describe all individual neurons and make predictions about where they’ll activate. I think Goodfire’s effort is still better as a UI than most previous autointerp methods because a) agents pay close attention to activations b) a neuron activating on multiple distinct themes is in the hypothesis space. Their autointerp would give useful labels for circuit interpretation, but most likely can't be used for enumerative safety. The useful thing remains studying individual cases in depth and trying to find patterns/motifs so we can find a better ontology.
For circuits, attribution graphs give us a lot of information, but have the problems of
not being readable by humans without heavy pruning
being dependent on the exact prompt/target token, which means they may fool us about how general or specific some features are for model behavior beyond that prompt and may not describe the computation faithfully.
These problems are much better with agents, because they can take in a lot of information at once and also resample the prompt in various ways. But scaling up use of agents also brings problems of models not being that good at fuzzy tasks and over/underclaiming. I am cautiously optimistic about applying agents to circuits-based interp and will wait for the full paper to come out.
We’ll probably see more instances of the “wallet” graph -- case studies which just find high-level handles (supernodes) and possibly describe the neurons in them in detail.
We should study why neurons are monosemantic
Already Clean claims: “Our findings don’t depend on knowing why generalization and composition keep winning...”, but the usefulness of these findings will depend on it at some point. We should understand why neurons seem to be easy to interpret, because we need to know how these properties will change with further scaling and how we can help preserve them.
I don’t think anyone has a good understanding of this. I have a vibe argument, which could be made falsifiable and probably is closer to reality for bigger and denser models. It’s extremely speculative, but some part of it feels right. I describe a possible model in which neurons are almost monosemantic in the section below and conceptual reasons for why this may be true.
Neurons may be almost monosemantic
A possible way for neurons to be nice-to-interpret: sparse activations + circuits + neurons with multiple distinct meanings = natural supernodes
Before discussing how I think about this conceptually, I will present some empirical claims and assumptions based on them:
Taking the top activations of neurons at every layer is a way to record most of the information useful to the computation; MLP activations can be surprisingly sparse, and the model will still exhibit the behaviors we care about.
In our paper, we find that neuron activations can be sparsified to an L0 of about 32 per layer/token, with a surprisingly low KL of around 0.5 (compared to transcoders, which get ~2.0 at the same L0). I’m uncertain about how well this kind of pruning preserves benchmark scores or qualities we care about, but some tests were performed recently.
Higher L0 is probably much more forgiving. I have some preliminary evidence that MoE MLPs are only usable at a lower sparsity, which would mean the rest of the claims would apply less strongly.
We can do special training to make activations more sparse; in the limit, this would be something like Monet or a transformer with transcoders for MLPs.
Results from Sparse Circuits (and neuron-based interp generally) suggest specific tasks or prompts have “circuits” -- modes of activation which use a small subset of neurons.
In other words:
on any given prompt, we can strongly sparsify activations without losing predictions we care about
on a collection of related prompts, the neurons that come out of sparsification are a subset of neurons in the original model
we can also select neurons through optimization or attribution and increase sparsity even further
Are such solutions faithful to the original model? Maybe not. I don’t know if dense transformers are any worse for this.
But, overall, this means that neurons activate in rare contexts, and when they are active, the other neurons that are active come from a small sample and not the set of all neurons in the model.
And speculatively, if neurons are encoding some variables used for computation, neurons can “rely” on only a small subset of other neurons being able to activate at the same time and encode useful information, those being the neurons that are part of the same circuit. This means we probably can’t have a “factored” encoding scheme where we can encode N^2 unrelated bits with N neurons by having two of the neurons activate to write a bit.
Similarly, this rules out naive superposition, where we could rotate the neuron basis and have something similarly interpretable.
Top activations of most neurons, when naively interpreted, have one or a few (2-5) surprisingly distinct meanings, which can be clustered apart easily.
So, here we’re saying that a neuron has several distinct “poles” of activation with different meanings that each make it strongly activate; the poles may bleed together, but they do not have to fundamentally on some deep level be “the same thing”; a neuron can activate on the tokens “red” and “transparent” without transparency somehow connected to redness on a metaphysical level.
We did a simple case study of this kind of neuron in Appendix R of our paper, and we might redo and publish it with a better UI soon.
It’s possible this is just a special case and we will see more general boolean circuit feature encoding -- any linear function of neuron activations is readable later on in the model, and even some nonlinear functions like XORs.
Because neurons can have several distinct meanings and strong neuron activations contain most relevant information, it will often be other neurons that disambiguate which meaning is actually present. And because circuits exist and neurons can only rely on a few other neurons to be active at the same time, there are only a few neurons which disambiguate which meaning is active.
So, for each neuron, we have several modes of activations which are associated with some set of other neurons being active. These would be “natural supernodes”: collections of neurons which, by activating together, encode some human-interpretable bit of information.
In the end, we have a “strong sparse superposition hypothesis”, where only 10-100 neurons per layer are important, and they combine into meaningful supernodes across layers. Those supernodes have clear effects on directions downstream in the network and interact in a sparse and understandable way. This would be really nice for interp, but is probably not strictly true.
I want to believe natural supernodes are “real” in some sense because there is an exponentially large amount of them for any neural network, so it is a very general hypothesis and feels like something that can actually fit reality. Supernodes are in the same niche as transcoder features in that they can encode a large number of very specific concepts and have variable levels of abstraction.
At the same time, natural supernodes are falsifiable, and any explanation of attribution graphs in terms of supernodes is evidence for them. It’s easy to collect a lot of evidence for or against them by studying attribution graphs.
Supernodes feel closer to the right type of object compared to features, but are too nice of a hypothesis to be true. These natural supernodes also fit into the same role as branches/pathways in Already Clean, which has the right mindset of using these abstractions only to the extent it’s useful for interpreting a specific task.
Why would neurons be almost monosemantic?
I listed some conditions which could make neuron activations nice to interpret, but not potential reasons for why this may be the case. This is similar to the argument from Liv Gorton for why features want to be linear[3] but applied liberally and indiscriminately.
An argument for why neurons would be monosemantic: emulation of an ideal boolean circuit
I basically think there are just a few pieces here, and I’m not sure how important each is and how they connect:
There is a simple boolean circuit that can be continuously simplified to fit any compute budget and approximation; in LLM training in particular, we get simplifications of this network as solutions and not something very specific to the architecture.
This implies different architectures and small variations on training data will converge to the same circuit.
This is a very strong claim and isn’t even precisely defined, but there are ways to test weaker versions of it, like for example anything to do with identifying similar representations in different neural networks (Platonic Representations and Rosetta Neurons), or low-level interp with more interpretable architectures (like weight-sparse transformers, which may give us something closer to the "ground truth" if the hypothesis is true).
Strong sparse activations are somehow needed to emulate this perfect network, and networks will assign neurons to specific variables. We will see some of the same neurons being learned between different training runs, and we will see the “strongest” of the neurons survive if we crush the network. The neuron basis is privileged in a specific way.
There are subsets of neurons that are useful for specific inputs or tasks, because of the graph structure of the “true” network.
Neurons have only a few partners they meaningfully interact with in ways that can’t be called compositional, in that they encode many more bits of information than we would expect if we treated them as independent. The network builds a latent space from which neurons can read clear features, and neurons will interact with their partners to contribute to this.
Monosemanticity in time
If we want to understand why neural networks can be clean, it is also valuable to look at how the network develops directly, observing the training process.
With neural networks, the entire causal graph for any given process includes training. So, interp should be able to explain both what a feature does at a given point in time and how it got there. In the same way understanding a neuron’s role in a model is the only way to understand its function, understanding a neuron’s development process is the only way to understand its meaning -- what it learned to fire on to serve that function.
The Goodfire post proposes “recursive generalization” as a hypothesis. I haven’t seen strong evidence for this in LLMs, but there’s probably something similar that is related to the structure of latent space. I’m looking forward to seeing a more specific set of claims and evidence.
One thing I’m somewhat excited about is extending case studies for circuits with attempts to understand the development of neuron activations and the circuits they are a part of. I did some studies of Olmo neurons, and the essay looked at the evolution of at least vision model neurons, but those are not described in much detail in the essay itself.
It makes sense to study development of neurons in toy models; the null hypothesis should be neuron meanings drifting in arbitrary ways through training and only the final behavior being constrained.
In fact, understanding why certain features are learned is actually basically solved for deep linear networks; we know why those learn “PCA-like” solutions. It’s possible a similar theory for gated neurons will come from developmental interpretability / learning theory, and connections to this area shouldn’t be neglected.
Formation of monosemantic neurons is relevant to the problem of the development of features -- how do neural networks learn generalizable feature extractors? This is IMO the core mystery of deep learning, has connections to natural abstractions, and understanding it well is how we can make claims about interp or alignment scaling in the long run.
Conclusion
Understanding high neuron activations and development processes by which they arise seems important, tractable and neglected. Goodfire seems to have a good handle on it, but I’m excited for other orgs to give it a try. We need more diversity of agendas in interp, and the release seems good for bringing in fresh blood.
Appendix: what about other features?
I’m curious about how the seeming monosemanticity of neurons relates to global workspaces. Neurons often write in the logit/J-lens directions, and the original global workspaces paper shows the MLP as a whole amplifies J-space activations. I ran some informal experiments and the J-space in any given layer seems to largely be made up of 10-100 high-activating neurons in a few preceding layers.[4] Neurons clearly have a different role as features from J-space components -- maybe neurons create directions/features by spiking and the J-space stores information -- and there may be fundamental reasons for that.
Global workspaces are just an instance of residual streams containing some very specific information. On the surface, it seems like neurons can’t possibly represent all that, but it’s possible there’s some fine coding that we can crack that contains all the same information and somehow can be used by the network during computation.
Despite this, it’s best to view neurons as just the process that leads up to their activation and all the effects they can have downstream. I don’t think neurons fundamentally have a duty to represent every feature that exists in the residual stream, there’s no reason for that to be true. But many useful concepts could have simple encodings in terms of neurons because they are computed from or processed by those neurons, because the computations require some spiky/binary activation.
Thanks to everyone who read this post and provided feedback before publication.
I’m applying a higher standard of rigor to neuron-based attribution graphs than is commonly applied to transcoder-based attribution graphs. My model is that for most transcoders the attribution graphs are approximately isomorphic to neuron attribution graphs because most features in circuits are neuron features (Appendix J.1 of our paper) and even those are not maximally faithful to the original model (because not all neurons are well-represented; many results in the paper suggest this). Because of this, I think there's little reason to prefer transcoder attribution graphs at this point.
We need nonlinear computation to decode nonlinear features, but most of that comes from MLP layers, and it is easier for MLPs to decode linear features than nonlinear ones, so features that MLPs need to read tend to be linear.
This is a post containing reflections on Goodfire's recent release, "Neural Networks Are Already Clean", which argues neuron-based interpretability may be easier than we think. The essay is a summary of a technical paper that will come out later this year. Also featuring thoughts on where this research will lead, plus speculative ideas about neuron monosemanticity
(What I base views expressed in this post on: earlier this year, I worked on neuron-based interpretability for LLMs (we recently published a paper). We mostly looked at low-level properties of the neuron basis and didn’t do many case studies or scale up automated circuit interpretation. I also thought about the problem for a long time and have some hypotheses based on results we’ve seen that are likely not strictly true.)
I’ll first discuss what I see as the implications of this work for LLM interpretability research and then talk about why neurons may be a useful basis: I wrote down some guesses (in a collapsible section so as to not distract from the main point) and then some directions that may be useful to study for basic science.
In short, I think the agenda presented in the essay is promising and worth pursuing even outside of direct applications to language model interpretability. I have some object-level uncertainty about results for LLMs and will wait for the final paper and followup work.
How useful is this?
What I see as the fundamental claim here is that resolving some unforced errors in visualizing neuron activation --
-- and then constructing circuits with neurons as features leads to the best method we have for understanding language model computation at a low level.
With this kind of processing, you get an “overlay” for the model, which doesn’t explain how the model works but gives you points at which you can observe and intervene on the prediction. A neuron attribution graph is basically a collection of probes and steering vectors per layer, which hopefully can also give you sparse causal graphs after you group nodes into small sets, supernodes, as in the attribution graphs paper.
An attribution graph from Already Clean, section "Exploring Language Models"
We should study why neurons are monosemantic
Already Clean claims: “Our findings don’t depend on knowing why generalization and composition keep winning...”, but the usefulness of these findings will depend on it at some point. We should understand why neurons seem to be easy to interpret, because we need to know how these properties will change with further scaling and how we can help preserve them.
I don’t think anyone has a good understanding of this. I have a vibe argument, which could be made falsifiable and probably is closer to reality for bigger and denser models. It’s extremely speculative, but some part of it feels right. I describe a possible model in which neurons are almost monosemantic in the section below and conceptual reasons for why this may be true.
Neurons may be almost monosemantic
A possible way for neurons to be nice-to-interpret: sparse activations + circuits + neurons with multiple distinct meanings = natural supernodes
Before discussing how I think about this conceptually, I will present some empirical claims and assumptions based on them:
So, for each neuron, we have several modes of activations which are associated with some set of other neurons being active. These would be “natural supernodes”: collections of neurons which, by activating together, encode some human-interpretable bit of information.
In the end, we have a “strong sparse superposition hypothesis”, where only 10-100 neurons per layer are important, and they combine into meaningful supernodes across layers. Those supernodes have clear effects on directions downstream in the network and interact in a sparse and understandable way. This would be really nice for interp, but is probably not strictly true.
I want to believe natural supernodes are “real” in some sense because there is an exponentially large amount of them for any neural network, so it is a very general hypothesis and feels like something that can actually fit reality. Supernodes are in the same niche as transcoder features in that they can encode a large number of very specific concepts and have variable levels of abstraction.
At the same time, natural supernodes are falsifiable, and any explanation of attribution graphs in terms of supernodes is evidence for them. It’s easy to collect a lot of evidence for or against them by studying attribution graphs.
Supernodes feel closer to the right type of object compared to features, but are too nice of a hypothesis to be true. These natural supernodes also fit into the same role as branches/pathways in Already Clean, which has the right mindset of using these abstractions only to the extent it’s useful for interpreting a specific task.
Why would neurons be almost monosemantic?
I listed some conditions which could make neuron activations nice to interpret, but not potential reasons for why this may be the case. This is similar to the argument from Liv Gorton for why features want to be linear[3] but applied liberally and indiscriminately.
An argument for why neurons would be monosemantic: emulation of an ideal boolean circuit
I basically think there are just a few pieces here, and I’m not sure how important each is and how they connect:
Monosemanticity in time
If we want to understand why neural networks can be clean, it is also valuable to look at how the network develops directly, observing the training process.
With neural networks, the entire causal graph for any given process includes training. So, interp should be able to explain both what a feature does at a given point in time and how it got there. In the same way understanding a neuron’s role in a model is the only way to understand its function, understanding a neuron’s development process is the only way to understand its meaning -- what it learned to fire on to serve that function.
The Goodfire post proposes “recursive generalization” as a hypothesis. I haven’t seen strong evidence for this in LLMs, but there’s probably something similar that is related to the structure of latent space. I’m looking forward to seeing a more specific set of claims and evidence.
One thing I’m somewhat excited about is extending case studies for circuits with attempts to understand the development of neuron activations and the circuits they are a part of. I did some studies of Olmo neurons, and the essay looked at the evolution of at least vision model neurons, but those are not described in much detail in the essay itself.
It makes sense to study development of neurons in toy models; the null hypothesis should be neuron meanings drifting in arbitrary ways through training and only the final behavior being constrained.
In fact, understanding why certain features are learned is actually basically solved for deep linear networks; we know why those learn “PCA-like” solutions. It’s possible a similar theory for gated neurons will come from developmental interpretability / learning theory, and connections to this area shouldn’t be neglected.
Formation of monosemantic neurons is relevant to the problem of the development of features -- how do neural networks learn generalizable feature extractors? This is IMO the core mystery of deep learning, has connections to natural abstractions, and understanding it well is how we can make claims about interp or alignment scaling in the long run.
Conclusion
Understanding high neuron activations and development processes by which they arise seems important, tractable and neglected. Goodfire seems to have a good handle on it, but I’m excited for other orgs to give it a try. We need more diversity of agendas in interp, and the release seems good for bringing in fresh blood.
Appendix: what about other features?
I’m curious about how the seeming monosemanticity of neurons relates to global workspaces. Neurons often write in the logit/J-lens directions, and the original global workspaces paper shows the MLP as a whole amplifies J-space activations. I ran some informal experiments and the J-space in any given layer seems to largely be made up of 10-100 high-activating neurons in a few preceding layers.[4] Neurons clearly have a different role as features from J-space components -- maybe neurons create directions/features by spiking and the J-space stores information -- and there may be fundamental reasons for that.
Thanks to everyone who read this post and provided feedback before publication.
See https://x.com/Sauers_/status/2114481373876801557 and https://x.com/_lyraaaa_/status/2115283209802096809 for examples of popular posts about using neurons as steering directions. Like https://arxiv.org/abs/2605.08513 and unlike https://nousresearch.com/neuron-steering, these tweets essentially show additive steering using neuron directions instead of ablating or amplifying a neuron's activations. The latter is more common in attribution graph interpretability literature.
I’m applying a higher standard of rigor to neuron-based attribution graphs than is commonly applied to transcoder-based attribution graphs. My model is that for most transcoders the attribution graphs are approximately isomorphic to neuron attribution graphs because most features in circuits are neuron features (Appendix J.1 of our paper) and even those are not maximally faithful to the original model (because not all neurons are well-represented; many results in the paper suggest this). Because of this, I think there's little reason to prefer transcoder attribution graphs at this point.
We need nonlinear computation to decode nonlinear features, but most of that comes from MLP layers, and it is easier for MLPs to decode linear features than nonlinear ones, so features that MLPs need to read tend to be linear.
I might publish this later.