Adding some context and takeaways from my initial reading as I'm fairly excited about this:

1. We might expand the title to "Superposition was possibly mistaken as an explanation for polysemanticity; neural networks are possibly much easier to interpret than previously thought".
   1. "Polysemanticity" is basically just "a single neuron can encode many different concepts".
   2. To explain superposition, I'll use his explanation: "The leading explanation for polysemanticity was superposition. This proposed a new learning principle: when a neural network runs out of neurons, it stops grouping related things together and instead begins folding unrelated things together, simply because they don’t interfere. The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing."
      1. Side note 1: technically "don't interfere" might be better stated as "don't interfere too much".
      2. Side note 2: arguably this definition of superposition is a narrow one, specifically about unrelated concepts being grouped together. Neurons still do group related concepts together; that is well-established (a neural network can definitely encode more concepts than it has neurons), and he goes on to theorize more on how this happens.
      3. For more on definitions, see the appendix bit of the post here.
   3. In the blog post, he works through previous claimed examples of superposition (including his own!) and argues that these were more the result of bad early architectures, weak tools, and unrecognized generalization.
   4. In general, Nick's thesis is that polysemanticity is more caused by what he calls "recursive generalization" than by superposition of unrelated concepts.
      1. Quoting: "It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all. (...) It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition. And since the generalization itself is fractal, our explanations can be too: start with a basic explanation, then zoom in when you want to understand more."
      2. Working through Qwen3-32B his preliminary results are that ~90% of neuron poles have one main job. A further ~7% only have two, ~3% are still not understood. Very promising!
         1. "neuron poles" is a nuance here, an idea that neurons have "positive" and "negative" poles, aka what activates a neuron in a strongly positive way vs. what activates it in a strongly negative way. He posits that looking at these separately is useful for understanding neurons.
         2. There's also an interesting posited nuance about neurons encoding nouns vs. verbs. Quoting: "Both human and automated interpreters seem strongly biased toward describing neurons as detectors for groups of nouns, like cats and cars, rather than for a shared pattern, like a V. Sometimes a neuron really does respond to a group of nouns. In my experience, though, it’s more often a shared pattern, and the neuron is better understood as a verb than a noun. Its job isn’t to name what’s in the input but to draw a distinction downstream neurons can use: it separates two subtle patterns at its positive and negative poles (a V versus an O) and pushes the flow of activations toward different neurons accordingly."
            1. See examples here.
      3. He claims recursive generalization is much easier to see if you watch as the model is trained and the neuron learns more concepts; just working from a finished model, as in some previous experiments, makes it harder to see how the generalization happened, how ex. "iy" is related to "do it yourself".
         1. Training small models can be made an experiment to see if polysemanticity is caused by unrelated superposition or recursive generalization, and he's seeing the latter.
         2. He also tested reducing neuron count to see if it caused superposition, and it didn't (in image models anyway).
      4. There's also a more general hypothesis here: "clean" neural networks are in some sense natural because of optimization pressures toward legibility to other parts of network. Quote: "The (neural) network is enormous, but it may already have done almost all the work of preparing itself to be understood, not for us, but for its own readers. If we can become another reader, it’s ready for us too."
         1. If true (and this is more hypothesis than established by experiment) it would be a strong reason to expect large LLMs to be as interpretable as smaller ones, or even more so!
            1. Worth noting that in a somewhat different test on GPT-2 XL (older and smaller than Qwen3-32B, just 1.5B params), only ~70% of neurons are "roughly one thing", so the apparent trend is increasing friendliness to interpretability, and he claims: "in general, smaller models have broader neurons, with more residual generalization than modern models". However needs more rigorous evaluation.
   5. This definitely leaves open a lot of room for further work. I think we'll probably see a flurry of followup work on this.
2. Nick Cammarata has worked in interpretability for many years, and was at OpenAI for some time ex. was working with Chris Olah on the interpretability team there before Olah left to co-found Anthropic. Has lately been at Goodfire, a premier interpretability lab, since 2023 as far as I can tell. It's believable that he's made a major breakthrough. (edit: not that breakthroughs can't come from anywhere!)
3. In particular, Nick phrases the nature of this claimed breakthrough as "there’s a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition."
   
   In other words, the nature of his research here is in significant part a "more dakka" approach. For example, the blog post includes 1000 analyses of neurons in Qwen3!
   
   Some curious ones in there, for example, a neuron whose positive pole has to do with February's Leap Day (with some recursive generalizations to ex. this one time in 1712 Sweden had a Feb. 30th!), and whose negative pole has to do with surnames in sports team rosters and also with digits in sports scores. It categorizes the positive pole as having 1 job, and the negative pole as having 2 related jobs. There isn't an obvious connection between the two poles though, and they seem to have no real shared circuitry apparently.
4. The linked post is "only" a gigantic, interactive research blog post, he says the full technical paper is forthcoming in a few weeks.
5. Besides testing his approach in small models and vision models, he says he's also begun testing it in modern large models: "So far it has worked in language models too, up to trillions of parameters. Those aren’t close to solved due to sheer size, but I’m optimistic about scaling the process." More details to come in upcoming technical paper apparently.
   
   The costs involved in making this work might be pretty large; I'd be curious to know how many dollars were spent interpreting Qwen3-32B. But it's a good sign that this research was able to be done outside the frontier labs; Goodfire does have some money though, they raised $150M earlier this year. Also, in terms of efficacy of scaling this approach, he does mention that interpreting gets easier as you go on, kind of like how filling out a crossword puzzle gets easier in some ways as you fill out more of the grid.

(edit: have added some extra takeaways above, have not substantively changed any previous ones)
(edit2: honestly recommend reading Nick's post itself, it's well-written and has interesting tangents too, not to mention interactive features!)
(edit3: added section 1.d.iv which I think is important! boy this comment got overly complex though)
(edit4: okay I really recommend playing around with some of Nick's interactive features, ex. the five neurons in Qwen3-32B—I like the Colorado Neuron, who knew Bolivia was the Colorado of South America…?)
(edit5: fyi at least some of the core text in the blog post was written by AI according to Pangram, particularly the section around the sentence "Ten billion real tokens had their chance, and a sentence written from the explanation alone got closer." Pangram says ~2% of core blog post (not counting interactive stuff), and ~31% of the appendix, is AI with high confidence.)

Have you done experiments with GANs? I suspect GANs actually have a different learning mechanism, and you would get a different kind of fractal structure. I also suspect RL does similar things to learning, but image GANs are a simpler species to study.

Crossposting from my Shortform, quickly written. Overall I think the essay makes valuable contributions to interpretability. But I am also worried that the community vastly overestimates the state of the field, and quite a lot of second-hand discussion I've seen online seems over-optimistic.

• I'm not convinced by the big claims (at this time), but the post makes multiple valuable contributions. I'm looking forward to see the technical paper and might change my mind in either direction.
• Important contribution: Neurons that we previously thought were polysemantic (multiple meanings, e.g. cars and dogs) are actually monosemantic (single meaning, e.g. V shaped pictures). Both Nick's essay and Transluce's work (Choi et al.) suggest that neurons in Image and Language models may be more monosemantic that previously thought, we just have to do a better job interpreting them.
• Main disagreement: I don't believe the optimistic version that I'd term "neurons is all we need". I expect the neuron basis to be not good enough for understanding LLMs, e.g. I expect that LLM computation will be distributed across neurons, and I still think the superposition* hypothesis is true more likely that not. I expect neuron-based interpretability will be insufficient to understand LLMs; I think this disagrees with Nick's essay, and also with Tranluce's paper Arora et al.. The reason for my scepticism is mainly: (a) LLMs still seem to perform more independent computations (incl. fact lookups) than they have neurons; I don't think there's a way to implement this with only monosemantic neurons. I expect something like Comp-in-Sup is what the network does. (b) The evidence (e.g. consider the essay's Christmas neuron) so far seems not sufficient to conclude we understand what neurons do; specifically I worry that maximum-activating examples tell a plausible but incomplete story. However, I'm somewhat withholding judgement until I see the technical report; the essay claims that weaker activations of neurons are less important. ("Do Weak Activations Matter?")
• Superposition disagreement: I find the evidence against superposition less convincing because many experiments are doing extreme compressions (retraining a layer down to 10 neurons), and take the absence of superposition as evidence against superposition in realistic settings. However, it is quite established that superposition (and computation in superposition) only really works in high dimensions / for large numbers of neurons, so I wouldn't have expected superposition in their the very-low dimension experiment.
• Another cool contribution: I like the essay's "recursive generalization" hypothesis (albeit I find the name confusing). I consider it a nice learning-theoretic explanation for why neurons may end up representing related concepts for "historical" reasons. From the example in the post: A neuron may initially represent meadows. Later, when the network learns finer concepts (dogs, tennis balls, flowers, grass, mountains), this neuron may be involved in representing those finer concepts just because it was already correlated with those concepts (dogs are often photographed on meadows). While I don't buy the essay's learning learning-principle story, I like this ideas as a motif we might see again.

*superposition is used to mean different things, and I don't have time for a proper discussion at the moment. For this shortform I think the essay's definition is close enough. I want to note though that Representation in Superposition (a la Elhage et al. 2022, storing exponentially many concepts in high dimensional vector space) and Computation in Superposition (a la Hänni et al. 2024, computing polynomially many functions with fewer non-linearities than functions) are different things, and at least for Transformers it's worth distinguishing these.

But in fact, cosine similarities between feature vectors have rich semantic content [...] Features aren't just crammed together arbitrarily; they're grouped with similar features.

Adam Scherlis, 2024

Last year I wrote "Polysemanticity is a Misnomer" which made a very similar argument. I said:

When we imagine the polysemanticity of LLM neurons you probably think about something like a neuron which will trigger off of unrelated concepts like a neuron that triggers on airplane parts and phrases signifying periodic repetition. In my exploration of gpt2 -xl, I haven't really encountered neurons that are polysemantic in that way. Many times neurons signify a somewhat abstract concept which has many different instantiations in text: concluding remarks in a sales email; contemporary sci-fi adjacent fears like 5G conspiracies; a group of performers; etc[2]. While abstract these concepts do seem unitary and centralized around an intuitively identifiable and recognizable individual concept. I'm not entirely sure these neurons represent what we typically conceive of as polysemanticity.

One of the points I make is that human languages are often polysemantic themselves; the word "right' has many different meanings, right? Neuron 23:4813 in GPT2-XL captures several of these meanings together, the directional sense of "right" as well as the immediacy ("right here," "right now"), confirmatory ("right?"), and political ("right-wing," "right to life") meanings of the term. While 23:4813 does also trigger on things like "higher" and "lower" and some other directional words I don't think this supports the broadly conceived idea of polysemanticity.

I have to agree that the GPT4 interpreter used to interpret GPT2-XL neurons on OpenAI's Neuron Viewer kinda sucks. There are many examples where a neuron that I could easily interpret was given a low interpretability score and possibly an inaccurate interpretation by the GPT4 interpreter. It seems very likely that the newer more sophisticated models we have now would be better at interpretation. Interpreting a neuron is very much like thematically interpreting a short story or poem, something the models (and most humans for that matter) are also not very good at doing. As he explains you sometimes need to look at a different level of analysis, in his example looking at pose rather than looking at the noun-label of dog or car. I'd disagree that this isn't a word-level idea, it likely is just that we don't have a word for it in English or other popular languages. "Tetrapod" is very close to capturing most of what this example neuron for dogs and cars is activating on (although there may be some viewing-angle specific aspects or things like "has exactly four equally sized and compact points of contact with the ground" that are not captured in that word).

My project is also based on using a similarity map to allow easily interpreted neurons to inform interpretation of their neighbors (and to thereby spread interpretability in a manner similar to what is described in Cammarata's post). I am almost done running some confirmations on an interpretation of a similarity mapping for GPT2-XL based on forward linkages. There are two very interesting human-interpretable global conceptual gradients running through the entire mapping. I presented the preliminary findings at LessOnline and since then have developed a method of confirming my interpretations more rigorously (I'm using an automated sentiment analysis to confirm my interpretation of what appears to be a valence gradient, for example).

Edit: Corrected "Tetraped" to the actual term "Tetrapod."

It's a notable fact that, as authors of fiction, LLMs are unusually bad at metaphors and similes: they often have the bad writer's purple prose habit of using a lot of them, and the ones they create frequently sound vaguely poetic but are nonsensical, suggesting nothing appropriate. (This observation is a year and a half old, and the models have improved, but it definitely was a significant weak point of AI writing.) I'm wondering if this is related to the recursive generalization phenomenon described? Maybe the metaphors/similes it's generating make sense to it, but not to humans?

The jigsaw puzzle / crossword analogy is particularly compelling to me. Nick seems to be saying: imagine that you're trying to decipher the meaning of a neuron, without knowing what anything else in the network does. The neuron may fire in all sorts of seemingly unrelated circumstances (ie, it's polysemantic). Now imagine that you know what every other neuron in the network does, except for the one you're looking at. Suddenly it gets a lot easier to spot what subtle abstraction ties together all of the neuron's seeming polysemanticity.

When you're doing a sudoku, you eventually reach a cascade point where you have so much information that everything else just drops into place. Seems like Nick has reached this point on smaller networks and expects to on larger networks, with more effort.

Very cool & exciting post (at least to me, a layman)!

This is interesting, I believe I had also indirectly found monosemantic-ish neurons when doing various perturbations over swiglu MLPs. I think this is a convenience of swiglu and possibly just gated MLPs in general, nonetheless an overlooked one (afaict the models llms used in this post all use gated MLPs)

IMO still seems underdetermined that networks can or should just be broken into neuron circuits and not higher-rank components

It's really fun to interact with the visualized example neurons in the post. Feels like the more of "this is where your food comes from and how the sausage gets made" content gets out into the world the better. Big congrats to Nick. Turtles all the way down (and up).

Regardless of how we finally settle on this paper's importance, this does make me think about verifiability.
Some people claim that we wouldn't be able to check an ASI claim to have sufficiently solved alignment. However would any ASI be able to fake such a result? (assuming we agree its true and repeat it)
Seem like some interp/alignment results and techniques will be a lot easier to check than come up with.