In which I investigate if Opus is able to interpret neurons better than GPT 4.
TLDR: Opus 5.5 is able to identify 3-19% of the neurons of GPT 2-XL based on activations. This is a big improvement over the 0.3% that GPT-4 was able to achieve in 2023.
#AI notice: This post is human written but in the research there was heavy use of Opus 5.5. I mainly asked it to rerun this 2023 paper to see how big the improvement is if Opus 5.5 is used instead of GPT4.
Since the invention of Large language models they have been stubbornly refusing to be clear. They have been described as black boxes and as inscrutable machines, if this stays true that is bad news for detecting AI scheming and thus Alignment. Recently the Goodfire post told us that neural networks are already clean and that simply looking into the neurons is enough. For the case of this post I mean to say that a neural network is clean if we can understand what the neurons do only by looking at the neurons and not by creating some additional derived object to study.
This inspired me to rerun an old experiment from 2023. In 2023 OpenAI published "Language models can explain neurons in language models". In this paper they had written up a methodology that could in principle be used to allow a smart neural network to explain neural nodes one at a time and for these node explanations to be tested. The process they used is straight forward. In this post I will go over the methods OpenAI used and I will rerun the same methods with Opus 5.5 as the main model instead of GPT-4. Then I will estimate how well Opus 5.5 would be able to explain all of GPT-2 XL and finally I will go into an example of a difficult neuron that showed a low score.
This post assumes that the reader is already familiar with the basics of neural networks, if not I recommend this post.
Methods 1: OpenAI
In this experiment I wanted to replicate the Language models can explain neurons paper that OpenAI published in 2023, but using Opus 5.5 instead of GPT4 as in the original paper. In order to replicate it properly let's first have a look at the original work.
The original work followed the same process for all of the neurons of GPT-2 XL. For example, let's say neuron L12 N5603 is selected. This indicates the neuron number 5603 in layer 12. We feed GPT-2 XL millions of tokens of text and record the neuron’s activation value for each token. We then find where those values are highest and save each example as a 64-token window of text around it. One example looks like this:
Neuron 1 Activations: <start> mayor 0 writes 0 : 0 " 0 After 0 careful 0 consideration 0 , 0 I 0 've 0 decided 0 the 0 people 2 must 0 get 0 a 0 final 0 say 10 . 0 ... <end>
These top outputs along with their value are then shown to an instance of GPT4 with this system message "We’re studying neurons in a neural network. Each neuron looks for some particular thing in a short document. Look at the parts of the document the neuron activates for and summarize in a single sentence what the neuron is looking for. Don’t list examples of words.
The activation format is token<tab>activation. Activation values range from 0 to 10. A neuron finding what it’s looking for is represented by a non-zero activation value. The higher the activation value, the stronger the match."
For the neuron chosen in the beginning of this example, GPT4 gives the following explanation: "words related to the concept of decision-making and responsibility."
This explanation is subsequently given to a new instance of GPT4 which does not have access to the activation values. This instance is then asked to predict the neuron activation at different tokens. The Pearson correlation is then calculated over 10 examples with around 640 tokens in total. this correlation is 1 if the predictions rise and fall exactly along with the neuron and 0 if there is no relation at all. In the original paper OpenAI described all neurons with a Pearson correlation of > 0.8 as well explained.
This method gave openAI over 1,000 of 307,200 neurons (~0.3%) scoring ≥ 0.8. That is still only around 0.3 percent of the neurons explained well.
Methods 2: Opus 5.5
There were a couple of differences between OpenAI's experiment and my experiment. The biggest difference is that I did not run this on all of GPT-2 XL's neurons, as I only aimed to get a general idea of the difference in explanation skills.
For this experiment Opus was allowed to look at 35 examples instead of 5 and it was allowed to write the neurons description in more than a single sentence.
I ran Opus 5.5 as a subagent instead of as a clean instance. One effect of this is that both subagents got the general Anthropic subagent instructions in addition to the ones for this experiment. The explainer subagent only got the general instruction of the experiment, the top-firing excerpts and the task to write out the explanation, while the simulator subagent only saw the instructions and the explanation.
It is also important to note that a part of this result can come from the language model judging itself. This issue was the same in the original paper so this should not have an impact on the relative difference.
GPT-4's simulation was based on probability-weighted expected values. For Opus 5.5 I instead just asked it to give a number form 0 to 10.
I was curious to see how much better Opus 5.5 would be at this task than GPT4, so I ran it through the same loop and compared with GPT4 on 20 random neurons.
Fig 1: The explanation score of both Opus 5.5 and GPT-4 on the same data
If you are curious to look into what part of this difference comes from the explainer vs the simulator that can also be found in the appendix.
On 20 neurons tested, Opus was able to outperform GPT4 on every single neuron. From the graph in fig 1 you can clearly see that GPT4 scored around 0 for many neurons, meaning that it was not able to identify any pattern, while Opus 5.5 was able to identify these patterns way better.
How good is Opus at one shotting these explanations?
Next I wanted to know how good Opus is at one shotting[1] these explanations, but this little experiment was really eating up my Claude Pro subscription, so I decided to batch the neurons. In order to test if batching was able to still give coherent results I ran a batch of 12 neurons where the last 6 neurons were randomly chosen from the first 20 neurons. This showed that the simulator had a slight drop in score, but the explainers seemed to be able to hold up reasonably. Using this combination I tested another 30 neurons, for a total of 50 neurons.
Fig 2: All 50 identified neurons and their Explanation scores
Over the 50 neurons Opus was able to identify some part of most of them in a single shot. These neuron explanation scores can be aggregated to give an estimate of what number of neurons would be well explained if we were to use this method on all of GPT-2 XL.
When building on this data it is important to note that in these random neurons only layers 8-40 where sampled. The pipeline was also designed on the first 20 neurons with a mean of 0.51 and the fresh 30 neurons gave an average of 0.41.
Fig 3:4 bars showing what percentage of neurons was able to score above criteria level. 0.7 corresponds to about half of the variation in the neuron's activity being explained and 0.8 corresponds to around 64%.
From this bar chart we can see that Opus is able to explain some behaviour in most cases but it is still far from being able to identify the top behaviour behind every neuron. Compared to openAI being able to identify 0.3% of neurons with GPT-4 opus was able to identify 8%, an astounding 26x improvement in ability to identify neurons.
Exploring a single difficult neuron
When looking into the 20 neurons for which Opus outperformed GPT4, I got curious about what the difficult neurons would be. By difficult neuron I here mean a neuron with a low correlation score meaning that Opus was not able to find what made the neuron turn on or off in one shot.
Neuron L21 N4526 only had a meager score of 0.25 and GPT-4 scored -0.02. Opus originally guessed that this neuron got activated at the end of a common phrase in chatty blogs.[2] This, though, did not seem to add up to the total behaviour of the neuron. Inspired by the Goodfire paper, I asked Opus to run some more detailed tests on this particular neuron, with longer context windows, context length sweeps, occlusions, a hypothesis tournament, input decomposition and sibling analysis.
These tests showed 2 nearby neurons that seemed to be a part of a "Meta, author talks to audience" group. After an afternoon of experiments with Claude Pro, 8% of the variation in its activation could be explained, compared to the 3.5% of the one shot explanations. This shows that the one shot explanations are still a lower bound on how clean GPT-2 XL's neurons actually are. In trying to explain this neuron I pulled on derived objects like the attention-head decomposition. There is still a lot that we do not know about neural networks and hopefully the neural networks not looking clean is still only an artifact of us not looking at it from the right way.
If there is something I missed in this post please tell me, this is my first try at replicating an interpretability paper.
Future plans
This means to me that simply asking Opus what each neuron does seems to be a promising research direction. For this reason I will also be joining the project to identify the meaning of every neuron in bonsai. If this project has interested you I recommend looking into the communal bonsai project too. If we say that clean means that every neuron can be explained in a single sentence then GPT-2 XL is not clean. I do hope to show that with the newer models we can get a lot more clarity on specific neuron meanings and that there is still a lot of room in explaining individual neurons.
Appendix 1: Link to the Claude conversation
Almost all of this was in the same Claude conversation. If you want to see exactly what I did and in what order you can see it here.
Appendix 2: Comparing explainer to simulator
In order to differentiate if it was the explanations or the simulations that showed better result we also ran Opus 5.5 as a simulator on the explanations of GPT-4 on 10 neurons.
Fig 4
This showed that Opus 5.5 was able to do better on GPT4's explanations than GPT4, but it was the combination that had the strongest effect.
The specific explanation was "the head word that completes a noun phrase or a fixed connective in mid-sentence, especially in chatty first-person blog, forum or announcement prose"
In which I investigate if Opus is able to interpret neurons better than GPT 4.
TLDR: Opus 5.5 is able to identify 3-19% of the neurons of GPT 2-XL based on activations. This is a big improvement over the 0.3% that GPT-4 was able to achieve in 2023.
#AI notice: This post is human written but in the research there was heavy use of Opus 5.5. I mainly asked it to rerun this 2023 paper to see how big the improvement is if Opus 5.5 is used instead of GPT4.
Since the invention of Large language models they have been stubbornly refusing to be clear. They have been described as black boxes and as inscrutable machines, if this stays true that is bad news for detecting AI scheming and thus Alignment. Recently the Goodfire post told us that neural networks are already clean and that simply looking into the neurons is enough. For the case of this post I mean to say that a neural network is clean if we can understand what the neurons do only by looking at the neurons and not by creating some additional derived object to study.
This inspired me to rerun an old experiment from 2023. In 2023 OpenAI published "Language models can explain neurons in language models". In this paper they had written up a methodology that could in principle be used to allow a smart neural network to explain neural nodes one at a time and for these node explanations to be tested. The process they used is straight forward. In this post I will go over the methods OpenAI used and I will rerun the same methods with Opus 5.5 as the main model instead of GPT-4. Then I will estimate how well Opus 5.5 would be able to explain all of GPT-2 XL and finally I will go into an example of a difficult neuron that showed a low score.
This post assumes that the reader is already familiar with the basics of neural networks, if not I recommend this post.
Methods 1: OpenAI
In this experiment I wanted to replicate the Language models can explain neurons paper that OpenAI published in 2023, but using Opus 5.5 instead of GPT4 as in the original paper. In order to replicate it properly let's first have a look at the original work.
The original work followed the same process for all of the neurons of GPT-2 XL. For example, let's say neuron L12 N5603 is selected. This indicates the neuron number 5603 in layer 12. We feed GPT-2 XL millions of tokens of text and record the neuron’s activation value for each token. We then find where those values are highest and save each example as a 64-token window of text around it. One example looks like this:
Neuron 1 Activations: <start> mayor 0 writes 0 : 0 " 0 After 0 careful 0 consideration 0 , 0 I 0 've 0 decided 0 the 0 people 2 must 0 get 0 a 0 final 0 say 10 . 0 ... <end>
These top outputs along with their value are then shown to an instance of GPT4 with this system message "We’re studying neurons in a neural network. Each neuron looks for some particular thing in a short document. Look at the parts of the document the neuron activates for and summarize in a single sentence what the neuron is looking for. Don’t list examples of words.
The activation format is token<tab>activation. Activation values range from 0 to 10. A neuron finding what it’s looking for is represented by a non-zero activation value. The higher the activation value, the stronger the match."
For the neuron chosen in the beginning of this example, GPT4 gives the following explanation: "words related to the concept of decision-making and responsibility."
This explanation is subsequently given to a new instance of GPT4 which does not have access to the activation values. This instance is then asked to predict the neuron activation at different tokens. The Pearson correlation is then calculated over 10 examples with around 640 tokens in total. this correlation is 1 if the predictions rise and fall exactly along with the neuron and 0 if there is no relation at all. In the original paper OpenAI described all neurons with a Pearson correlation of > 0.8 as well explained.
This method gave openAI over 1,000 of 307,200 neurons (~0.3%) scoring ≥ 0.8. That is still only around 0.3 percent of the neurons explained well.
Methods 2: Opus 5.5
There were a couple of differences between OpenAI's experiment and my experiment. The biggest difference is that I did not run this on all of GPT-2 XL's neurons, as I only aimed to get a general idea of the difference in explanation skills.
For this experiment Opus was allowed to look at 35 examples instead of 5 and it was allowed to write the neurons description in more than a single sentence.
I ran Opus 5.5 as a subagent instead of as a clean instance. One effect of this is that both subagents got the general Anthropic subagent instructions in addition to the ones for this experiment. The explainer subagent only got the general instruction of the experiment, the top-firing excerpts and the task to write out the explanation, while the simulator subagent only saw the instructions and the explanation.
It is also important to note that a part of this result can come from the language model judging itself. This issue was the same in the original paper so this should not have an impact on the relative difference.
GPT-4's simulation was based on probability-weighted expected values. For Opus 5.5 I instead just asked it to give a number form 0 to 10.
I was curious to see how much better Opus 5.5 would be at this task than GPT4, so I ran it through the same loop and compared with GPT4 on 20 random neurons.
Fig 1: The explanation score of both Opus 5.5 and GPT-4 on the same data
If you are curious to look into what part of this difference comes from the explainer vs the simulator that can also be found in the appendix.
On 20 neurons tested, Opus was able to outperform GPT4 on every single neuron. From the graph in fig 1 you can clearly see that GPT4 scored around 0 for many neurons, meaning that it was not able to identify any pattern, while Opus 5.5 was able to identify these patterns way better.
How good is Opus at one shotting these explanations?
Next I wanted to know how good Opus is at one shotting[1] these explanations, but this little experiment was really eating up my Claude Pro subscription, so I decided to batch the neurons. In order to test if batching was able to still give coherent results I ran a batch of 12 neurons where the last 6 neurons were randomly chosen from the first 20 neurons. This showed that the simulator had a slight drop in score, but the explainers seemed to be able to hold up reasonably. Using this combination I tested another 30 neurons, for a total of 50 neurons.
Fig 2: All 50 identified neurons and their Explanation scores
Over the 50 neurons Opus was able to identify some part of most of them in a single shot. These neuron explanation scores can be aggregated to give an estimate of what number of neurons would be well explained if we were to use this method on all of GPT-2 XL.
When building on this data it is important to note that in these random neurons only layers 8-40 where sampled. The pipeline was also designed on the first 20 neurons with a mean of 0.51 and the fresh 30 neurons gave an average of 0.41.
Fig 3:4 bars showing what percentage of neurons was able to score above criteria level. 0.7 corresponds to about half of the variation in the neuron's activity being explained and 0.8 corresponds to around 64%.
From this bar chart we can see that Opus is able to explain some behaviour in most cases but it is still far from being able to identify the top behaviour behind every neuron. Compared to openAI being able to identify 0.3% of neurons with GPT-4 opus was able to identify 8%, an astounding 26x improvement in ability to identify neurons.
Exploring a single difficult neuron
When looking into the 20 neurons for which Opus outperformed GPT4, I got curious about what the difficult neurons would be. By difficult neuron I here mean a neuron with a low correlation score meaning that Opus was not able to find what made the neuron turn on or off in one shot.
Neuron L21 N4526 only had a meager score of 0.25 and GPT-4 scored -0.02. Opus originally guessed that this neuron got activated at the end of a common phrase in chatty blogs.[2] This, though, did not seem to add up to the total behaviour of the neuron. Inspired by the Goodfire paper, I asked Opus to run some more detailed tests on this particular neuron, with longer context windows, context length sweeps, occlusions, a hypothesis tournament, input decomposition and sibling analysis.
These tests showed 2 nearby neurons that seemed to be a part of a "Meta, author talks to audience" group. After an afternoon of experiments with Claude Pro, 8% of the variation in its activation could be explained, compared to the 3.5% of the one shot explanations. This shows that the one shot explanations are still a lower bound on how clean GPT-2 XL's neurons actually are. In trying to explain this neuron I pulled on derived objects like the attention-head decomposition. There is still a lot that we do not know about neural networks and hopefully the neural networks not looking clean is still only an artifact of us not looking at it from the right way.
If there is something I missed in this post please tell me, this is my first try at replicating an interpretability paper.
Future plans
This means to me that simply asking Opus what each neuron does seems to be a promising research direction. For this reason I will also be joining the project to identify the meaning of every neuron in bonsai. If this project has interested you I recommend looking into the communal bonsai project too. If we say that clean means that every neuron can be explained in a single sentence then GPT-2 XL is not clean. I do hope to show that with the newer models we can get a lot more clarity on specific neuron meanings and that there is still a lot of room in explaining individual neurons.
Appendix 1: Link to the Claude conversation
Almost all of this was in the same Claude conversation. If you want to see exactly what I did and in what order you can see it here.
Appendix 2: Comparing explainer to simulator
In order to differentiate if it was the explanations or the simulations that showed better result we also ran Opus 5.5 as a simulator on the explanations of GPT-4 on 10 neurons.
Fig 4
This showed that Opus 5.5 was able to do better on GPT4's explanations than GPT4, but it was the combination that had the strongest effect.
A one shot means success on the first try.
The specific explanation was "the head word that completes a noun phrase or a fixed connective in mid-sentence, especially in chatty first-person blog, forum or announcement prose"