Polysemanticity is one of the most important concepts when it comes to mechanistic interpretability, basically, studying the internal representations of neural networks. For some reason, the concept of polysemanticity seems really, really fascinating to me, mostly due to how it kind of relates to superposition in quantum computing.
I think polysemanticity is a concept best expressed through visualisation and this is my attempt at sort of presenting a simplified, illustrated overview of polysemanticity. This is largely based on Neel Nanda’s explanation of polysemanticity in his MLST talk, which was the first time I was introduced to this concept and which made me pause the video to fully visualise the concept in my head.
Essentially, polysemanticity arises from superposition. In other words, it’s the representation of multiple states in the same space. In neural networks, polysemanticity occurs when there are not enough neurons in the model to map distinctly to all the possible inputs. Therefore, the model learns to map multiple distinct inputs to the same feature vector. Or to put it slightly differently, the same neuron activates on multiple distinct inputs.
But this is something that can’t be concretely explained without a solid example. And the following example, discussed in the aforementioned talk, made the concept click for me.
Imagine you have 25 instances of Python code snippets, and 25 instances of classic novel snippets. Very distinct. There is almost negligible probability of co-occurrence of the two. Now, you have 25 neurons and each neuron activates on a pair of inputs - either a python code snippet or classic novel snippet. There is another a "discriminating" or "domain indicator" feature which indicates which of the two has triggered the neuron.
Each neuron maps to a distinct python concept AND a distinct novel concept. The 26th “discriminating” or “domain indicator” neuron indicates which domain is causing the activation at any point.
Now if we specifically think about polysemanticity in LLM internals, we can depict it through the transformer layer.
Inputs as they pass through the transformer layers are encoded into increasingly concrete representations. There are much more words/concepts in the sample input space than there are neurons in most LLMs, which gives rise to polysemanticity.
One thing to note about polysemanticity is that neural networks tend to exhibit it for very distinct concepts that truly do not have a high likelihood of co-occuring (because that would defeat the purpose, right?).
Another example later on in the video made the concept even clearer and, if possible, more interesting to me.
This time the example is very specific to compound words. Let’s quickly take a look at some pre-requisites:
text is tokenised before being processed by a transformer
common words are often represented as single tokens while rarer words are broken into multiple “common-er” tokens
a transformer has many layers that process the input
the earlier layers in a transformer are more important and responsible for “detokenisation” and assembling tokens into more concrete semantic units for deeper “reasoning” of later layers
there are many compound words in english
there are much less neurons in these early layers than the amount of compound words in the english language
It therefore follows from the above that polysemanticity/superposition is needed to represent these compound words.
Let’s take an example: “Michael Jordan” and “Taylor Swift”. If you have only “Michael” -> it can be anything, nothing specific. If you have only “Jordan” -> again many possibilities; it could be a country or a common name (or the shoes? but that’s plural so low probability).
Now if “Michael” and “Jordan” occur together, that really changes the representation. We can have a clear unambiguous representation of this compound word.
Same goes for “Taylor Swift”. Now circling back to superposition. It is not possible that the tokens in one compound word will occur in the same manner in another compound word.
The same neuron activates for two distinct compound words. It can activate for various distinct compound words. The proceeding combination of neuron activations is what represents which compound word is being encoded at that point.
The same talk also involves discussion on an experiment probing the early layers of a transformer on compound word inputs. It found that no single (very important to note: single) neuron activated at a given compound word. In fact, a single neuron tended to activate for multiple different compound words - superposition!
So what differentiates the distinct compound words? A combination of activated neurons. At a given compound word, once you sum up the activations of a group of neurons, you would get a clean, distinct representation of that compound word. And this “grouped activations’ sum” is what was at play behind representing the different compound words.
This again ties back to the above Python vs novel example: if a single neuron activates at an input, you can’t tell if it’s for a python code snippet or a classic novel snippet. But once you sum it with the distinguishing (26th) feature mentioned above, you can instantly tell which of the two it is. It’s kinda like adding direction to the magnitude. You have the distance 4 meters. It could either be 4 meters forwards or backwards. You don’t know. Until you add a sign. - or +. And it’s clear whether forwards or backwards. Just a far-fetched analogy that came to my mind but sorta helps explain.
Polysemanticity is quite important in mechanistic interpretability because it actually directly affects the metrics used to evaluate model representations. If we look at the aforementioned example where 25 neurons have a shared capacity of representing either a code or novel snippet, and let’s say that you are testing activations one by one. Till the 25th neuron you find that the 3rd neuron is activated but due to superposition the output is hazy. When you get to the 26th neuron, its addition to the current activations suddenly make the output clear - python code feature. You assume that the 26th neuron is responsible for encoding Python code features. But that is not the case. The previous neuron has also contributed.
This can be illustrated if we look at two probing techniques - probing sequentially vs. probing continuously.
Testing for the representation of a concept is done in a binary manner - does this neuron encode the given concept or not? Even if a neuron is found to represent a concept, it is likely not the only on which is responsible for encoding it; prior neurons also contributed.
Logits are tracked which help indicate directly which neurons contribute most to the representation of a given concept. Therefore, we can track which neurons activate for that concept as well as how much.
This is a very simplified and basic example. In reality it’s much more complex but the point is that interpretability metrics have to very well curated so that they sort of probe from different angles, changing things here and there, isolating stuff, to clearly and accurately figure out model internals.
To allude to the title - polysemanticity is neither inherently 100% good or bad. It is useful in that it allows a limited number of neurons to represent more concepts than their individual numbers. Without polysemanticity, some concepts would be forced to be dropped in lieu of other more important ones. Simultaneously, polysemanticity also does make mechanistic interpretability a bit cumbersome because it becomes difficult to ascertain what concept a neuron is representing when you can’t isolate the representations concretely due to superposition. Win some, lose some.
While this blog is concentrated around polysemanticity in language models, the concept is exhibited in other kinds of neural networks too, including vision models. In fact, it’s even more interesting to study polysemanticity in CNNs because you can see the polysemanticity being visualized in image activations.
The Toy Models of Superposition paper by Anthropic is a good in-depth, introductory resource to get super familiarized with polysemanticity.
Polysemanticity is one of the most important concepts when it comes to mechanistic interpretability, basically, studying the internal representations of neural networks. For some reason, the concept of polysemanticity seems really, really fascinating to me, mostly due to how it kind of relates to superposition in quantum computing.
I think polysemanticity is a concept best expressed through visualisation and this is my attempt at sort of presenting a simplified, illustrated overview of polysemanticity. This is largely based on Neel Nanda’s explanation of polysemanticity in his MLST talk, which was the first time I was introduced to this concept and which made me pause the video to fully visualise the concept in my head.
Essentially, polysemanticity arises from superposition. In other words, it’s the representation of multiple states in the same space. In neural networks, polysemanticity occurs when there are not enough neurons in the model to map distinctly to all the possible inputs. Therefore, the model learns to map multiple distinct inputs to the same feature vector. Or to put it slightly differently, the same neuron activates on multiple distinct inputs.
But this is something that can’t be concretely explained without a solid example. And the following example, discussed in the aforementioned talk, made the concept click for me.
Imagine you have 25 instances of Python code snippets, and 25 instances of classic novel snippets. Very distinct. There is almost negligible probability of co-occurrence of the two. Now, you have 25 neurons and each neuron activates on a pair of inputs - either a python code snippet or classic novel snippet. There is another a "discriminating" or "domain indicator" feature which indicates which of the two has triggered the neuron.
Each neuron maps to a distinct python concept AND a distinct novel concept. The 26th “discriminating” or “domain indicator” neuron indicates which domain is causing the activation at any point.
Now if we specifically think about polysemanticity in LLM internals, we can depict it through the transformer layer.
Inputs as they pass through the transformer layers are encoded into increasingly concrete representations. There are much more words/concepts in the sample input space than there are neurons in most LLMs, which gives rise to polysemanticity.
One thing to note about polysemanticity is that neural networks tend to exhibit it for very distinct concepts that truly do not have a high likelihood of co-occuring (because that would defeat the purpose, right?).
Another example later on in the video made the concept even clearer and, if possible, more interesting to me.
This time the example is very specific to compound words. Let’s quickly take a look at some pre-requisites:
It therefore follows from the above that polysemanticity/superposition is needed to represent these compound words.
Let’s take an example: “Michael Jordan” and “Taylor Swift”. If you have only “Michael” -> it can be anything, nothing specific. If you have only “Jordan” -> again many possibilities; it could be a country or a common name (or the shoes? but that’s plural so low probability).
Now if “Michael” and “Jordan” occur together, that really changes the representation. We can have a clear unambiguous representation of this compound word.
Same goes for “Taylor Swift”. Now circling back to superposition. It is not possible that the tokens in one compound word will occur in the same manner in another compound word.
The same neuron activates for two distinct compound words. It can activate for various distinct compound words. The proceeding combination of neuron activations is what represents which compound word is being encoded at that point.
The same talk also involves discussion on an experiment probing the early layers of a transformer on compound word inputs. It found that no single (very important to note: single) neuron activated at a given compound word. In fact, a single neuron tended to activate for multiple different compound words - superposition!
So what differentiates the distinct compound words? A combination of activated neurons. At a given compound word, once you sum up the activations of a group of neurons, you would get a clean, distinct representation of that compound word. And this “grouped activations’ sum” is what was at play behind representing the different compound words.
This again ties back to the above Python vs novel example: if a single neuron activates at an input, you can’t tell if it’s for a python code snippet or a classic novel snippet. But once you sum it with the distinguishing (26th) feature mentioned above, you can instantly tell which of the two it is. It’s kinda like adding direction to the magnitude. You have the distance 4 meters. It could either be 4 meters forwards or backwards. You don’t know. Until you add a sign. - or +. And it’s clear whether forwards or backwards. Just a far-fetched analogy that came to my mind but sorta helps explain.
Polysemanticity is quite important in mechanistic interpretability because it actually directly affects the metrics used to evaluate model representations. If we look at the aforementioned example where 25 neurons have a shared capacity of representing either a code or novel snippet, and let’s say that you are testing activations one by one. Till the 25th neuron you find that the 3rd neuron is activated but due to superposition the output is hazy. When you get to the 26th neuron, its addition to the current activations suddenly make the output clear - python code feature. You assume that the 26th neuron is responsible for encoding Python code features. But that is not the case. The previous neuron has also contributed.
This can be illustrated if we look at two probing techniques - probing sequentially vs. probing continuously.
Testing for the representation of a concept is done in a binary manner - does this neuron encode the given concept or not? Even if a neuron is found to represent a concept, it is likely not the only on which is responsible for encoding it; prior neurons also contributed.
Logits are tracked which help indicate directly which neurons contribute most to the representation of a given concept. Therefore, we can track which neurons activate for that concept as well as how much.
This is a very simplified and basic example. In reality it’s much more complex but the point is that interpretability metrics have to very well curated so that they sort of probe from different angles, changing things here and there, isolating stuff, to clearly and accurately figure out model internals.
To allude to the title - polysemanticity is neither inherently 100% good or bad. It is useful in that it allows a limited number of neurons to represent more concepts than their individual numbers. Without polysemanticity, some concepts would be forced to be dropped in lieu of other more important ones. Simultaneously, polysemanticity also does make mechanistic interpretability a bit cumbersome because it becomes difficult to ascertain what concept a neuron is representing when you can’t isolate the representations concretely due to superposition. Win some, lose some.
While this blog is concentrated around polysemanticity in language models, the concept is exhibited in other kinds of neural networks too, including vision models. In fact, it’s even more interesting to study polysemanticity in CNNs because you can see the polysemanticity being visualized in image activations.
The Toy Models of Superposition paper by Anthropic is a good in-depth, introductory resource to get super familiarized with polysemanticity.