What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts.

Nick Cammarata’s recent post makes an important and true observation: in the past, people have been too quick to write off neurons as polysemantic, but if you look harder, there is actually often a simple pattern behind the activating examples. This historically caused people to overestimate how much polysemanticity there is. This is a really good point and we should update on it. But it’s nowhere near a sufficient argument that we understand most of what neurons are doing, and these results do not show that interpretability is solved, or even that it is vastly easier than previously anticipated.

What does it take to explain a neuron?

It’s unclear whether there even is a simple pattern that explains every neuron.^[1] But even if there is a simple pattern that matches the activating examples of a neuron, that still doesn’t mean that the explanation is correct.^[2] Suppose we look at the top activating examples of a neuron, and they are all sentences like “Fred the poodle zipped down the street on his bicycle. ‘Wow, I didn’t know a dog could ride a bicycle!’ I exclaimed.” Top activations can often give a misleading impression of how monosemantic a neurons is, so next we look at the top 20th percentile of activations, and we find the examples are now things like “John rode on his bicycle while his dog trailed behind”. And when we check the weakest activations, they’re often still about dogs, just without bicycles — “The dog sauntered lazily down the street.” Given this evidence, can we conclude that the neuron is monosemantic, and that its meaning is “dogs riding bicycles”?

I argue we cannot. There are a few reasons why the vibes of activating examples might heavily mislead us about how interpretable a neuron is. First, just because every example that activates the neuron fits the pattern, doesn’t necessarily mean that the other way around is true; perhaps not all examples that fit the pattern also activate the neuron. Perhaps your neuron actually only activates on dogs on bicycles on sunny days, but you never noticed that. This turned out to be a pretty big problem with the auto interp paper.^[3] In practice, it’s very ...

An Innocent Bystander

An Innocent Bystander