What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts.

Nick Cammarata’s recent post makes an important and true observation: in the past, people have been too quick to write off neurons as polysemantic, but if you look harder, there is actually often a simple pattern behind the activating examples. This historically caused people to overestimate how much polysemanticity there is. This is a really good point and we should update on it. But it’s nowhere near a sufficient argument that we understand most of what neurons are doing, and these results do not show that interpretability is solved, or even that it is vastly easier than previously anticipated.

What does it take to explain a neuron?

It’s unclear whether there even is a simple pattern that explains every neuron.^[1] But even if there is a simple pattern that matches the activating examples of a neuron, that still doesn’t mean that the explanation is correct.^[2] Suppose we look at the top activating examples of a neuron, and they are all sentences like “Fred the poodle zipped down the street on his bicycle. ‘Wow, I didn’t know a dog could ride a bicycle!’ I exclaimed.” Top activations can often give a misleading impression of how monosemantic a neurons is, so next we look at the top 20th percentile of activations, and we find the examples are now things like “John rode on his bicycle while his dog trailed behind”. And when we check the weakest activations, they’re often still about dogs, just without bicycles — “The dog sauntered lazily down the street.” Given this evidence, can we conclude that the neuron is monosemantic, and that its meaning is “dogs riding bicycles”?

I argue we cannot. There are a few reasons why the vibes of activating examples might heavily mislead us about how interpretable a neuron is. First, just because every example that activates the neuron fits the pattern, doesn’t necessarily mean that the other way around is true; perhaps not all examples that fit the pattern also activate the neuron. Perhaps your neuron actually only activates on dogs on bicycles on sunny days, but you never noticed that. This turned out to be a pretty big problem with the auto interp paper.^[3] In practice, it’s very common for neurons that look like they activate on X to actually only activate on a very specific subset of X that is hard to characterize compactly. If you forced the neuron to always activate on X, it would degrade the model.

Second, there might actually be really important information in the activation magnitudes. It might matter crucially whether the neuron is strongly or weakly activating, and if you have a hard time predicting which activations should be strong or weak, you might be missing a big part of the explanation. My best guess from all the results I've seen is that ablating small neuron activations quite substantially degrades model performance, so I don’t think we get to ignore weak activations completely.^[4]

Simulation scores tell us whether we explained a neuron

To be able to claim that we understand a neuron, it’s insufficient for things to look kinda nice. If you can’t simulate the neuron in a way that doesn’t degrade the model, you don’t really understand the neuron. This requires writing a rubric—”If there’s a dog literally on a bicycle, that’s a 4.5; if there’s a dog and a bicycle but no riding, that’s a 3.2”, and so on—and then comparing that rubric against inputs and substituting neuron values with the simulated values. If this process heavily degrades the model, then our explanation was wrong.

Nick’s post argues that simulation scores are bad because they are too demanding — they force you to explain lots of stuff that doesn’t actually matter. However, I think this is just a byproduct of the kinda terrible correlation-based simulation scores used by the original auto interp paper, and is fixed basically entirely by (a) measuring downstream KL instead and (b) scoring on a large dataset. KL also elegantly avoids the problem of needing to bin activations carefully.^[5]

If it doesn’t actually matter for the model’s behavior which punctuation token sentence-level information is stored in, or which activation strengths of the Trump neuron correspond to different sizes of Trump, then the KL won’t punish it. Conversely, if it matters a ton, it seems completely fair to demand an explanation. If the relation really is simple but uninteresting, then this doesn’t make the explanation much more complicated, but if the relation is actually complicated in a way that is essential for model functioning, it’s best to know. 

In fact, if we had applied this scoring methodology all along, we would have noticed that some of the polysemantic explanations were bad explanations in the first place! For instance, the car and dog explanation that Nick uses as an example of a bad polysemantic explanation of the V-shaped-object neuron would have been flagged as a bad explanation, because it would fail to fire on a ton of V shaped objects, and also it would falsely fire on a bunch of round dogs and cars.^[6]

I don’t think the simulation KL metric is perfect,^[7] and I’m sympathetic to the argument against relying too much on metrics, but it avoids the two failure modes I outlined above, which are very hard to notice by looking at activating examples, but which nonetheless matter a lot. Importantly, it captures both of the reasons that the basic vibe check might fail.

Beyond understanding neurons

Explaining neurons is still only part of fully understanding what a neural network is doing. We ultimately want to understand how models work, which requires understanding the computation between neurons. Unfortunately, this is really hard, and doesn’t work very well. Past work on finding circuits between neurons still finds relatively large and complicated circuits, which often have lots of confusing redundancy, and fail to explain most of the variance of the model’s behavior. My current best explanation for this is that concepts are stored in superposition, and MLPs do computation on those features in superposition, by doing more than one computation per neuron.

I’m hopeful that it’s possible to disentangle everything, figure out how computation-in-superposition works, and ultimately find clean, understandable circuits in networks. But historically, the field of interpretability has often gotten stuck because of a lack of rigor, focusing on things that look cool even when those things oversell our actual level of understanding. It’s up to us to strike the right balance between striking qualitative examples and rigorous arguments for understanding.

1. ^{^}

   Someone found a neuron that activates on Colorado and also Tornado (the python library, not tornadoes in general). I have a hard time thinking of a simple monosemantic explanation that captures both of these! More broadly, my experience with neurons is that most neurons are pretty inexplicable, even after thinking pretty hard about potential explanations.

2. ^{^}

   By which I mean, the pattern applies to all examples where the neuron activates, which does not necessarily imply that the neuron activates in all examples where the pattern applies. X matches Y is not commutative!

3. ^{^}

   I don’t think the revision stuff turned out to be super satisfying for solving this problem, and it’s fundamentally less satisfying than simulation which I describe in the next section because you’re depending on the attacker to find all the relevant counterexample inputs.

4. ^{^}

   Hopefully I'll be able to share something more concrete soon; I’m currently prioritizing getting this post out ASAP. It’s surprising that my result here is different from Nick’s, and it seems worth tracking down the source of the divergence.

5. ^{^}

   I think downstream KL is more correct than loss delta, because it’s very cursed if you can offset bad modeling on some neurons by modeling other neurons in ways that actually improve loss. My experience in practice is that this is not hypothetical.

6. ^{^}

   Each individual neuron has a very small effect on the entire network, so we’d expect the number to be very small in some absolute sense. However, we can still compare to other neurons and see which explanations cause relatively bigger impacts, or ablate large batches of neurons at a time.

7. ^{^}

   For example, it’s pretty hard to compare models of different sizes.