Adding some context from my initial reading as I'm fairly excited about this:

1. We might expand the title to "Superposition was possibly mistaken as an explanation for polysemanticity; neural networks are possibly much easier to interpret than previously thought".
   1. "Polysemanticity" is basically just "a single neuron can encode many different concepts".
   2. To explain superposition, I'll use his explanation: "The leading explanation for polysemanticity was superposition. This proposed a new learning principle: when a neural network runs out of neurons, it stops grouping related things together and instead begins folding unrelated things together, simply because they don’t interfere. The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing."
      1. Side note 1: technically "don't interfere" might be better stated as "don't interfere too much".
      2. Side note 2: arguably this definition of superposition is a narrow one, specifically about unrelated concepts being grouped together. Neurons still do group related concepts together; that is well-established (a neural network can definitely encode more concepts than it has neurons), and he goes on to theorize more on how this happens.
   3. In the blog post, he works through previous claimed examples of superposition (including his own!) and argues that these were more the result of bad early architectures, weak tools, and unrecognized generalization.
   4. In general, Nick's thesis is that polysemanticity is more caused by what he calls "recursive generalization" than by superposition of unrelated concepts.
      1. Quoting: "It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all. (...) It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition. And since the generalization itself is fractal, our explanations can be too: start with a basic explanation, then zoom in when you want to understand more."
      2. Working through Qwen3-32B his preliminary results are that ~90% of neuron poles have one main job. A further ~7% only have two, ~3% are still not understood. Very promising!
         1. "neuron poles" is a nuance here, an idea that neurons have "positive" and "negative" poles, aka what activates a neuron in a strongly positive way vs. what activates it in a strongly negative way. He posits that looking at these separately is useful for understanding neurons.
2. Nick Cammarata has worked in interpretability for many years, and was at OpenAI for some time ex. was working with Chris Olah on the interpretability team there before Olah left to co-found Anthropic. Has lately been at Goodfire, a premier interpretability lab, since 2023 as far as I can tell. It's believable that he's made a major breakthrough. (edit: not that breakthroughs can't come from anywhere!)
3. In particular, Nick phrases the nature of this claimed breakthrough as "there’s a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition."
   
   In other words, the nature of his research here is in significant part a "more dakka" approach.
4. The linked post is "only" a gigantic research blog post, he says the full technical paper is forthcoming in a few weeks.
5. Besides testing his approach in small models and vision models, he says he's also begun testing it in modern large models: "So far it has worked in language models too, up to trillions of parameters. Those aren’t close to solved due to sheer size, but I’m optimistic about scaling the process." More details to come in upcoming technical paper apparently.
   
   If this really is promising, the costs might be pretty large. But it's a good sign that this was able to be done outside the frontier labs; Goodfire does have some money though, they raised $150M earlier this year. Also, in terms of efficacy of scaling this approach, he does mention that interpreting gets easier as you go on, kind of like how filling out a crossword puzzle gets easier in some ways as you fill out more of the grid.

First impressions: The thrust of the essay, that we're much closer to fully interpreting and understanding frontier models than we thought, is big if true and both exciting and scary. It would speed up prosaic alignment and control progress, but it seems like it'd also speed up capabilities progress. For example, the author points out that the older models they studied had hard-to-interpret neurons in part because the older architecture was worse at learning in deeper layers, which implies researchers could use interpretability tools to uncover problems with their learning algorithms.

This is interesting, I believe I had also indirectly found monosemantic-ish neurons when doing various perturbations over swiglu MLPs. I think this is a convenience of swiglu and possibly just gated MLPs in general, nonetheless an overlooked one

IMO still seems underdetermined that networks can or should just be broken into neuron circuits and not higher-rank components