This work was done with limited GPU compute; a more complete version of this work will come in later posts (take a look at my GPU Democracy post on Substack!)
Motivation
With Large Language Models approaching and exceeding human-level performance on most tasks, one may ask whether anthropomorphization of Language Models is a good approach for intuitively looking at how the internals of the models operate. If we can see the emotions that a model is experiencing, then can we make inferences about the model's "state of mind" and how pleased it is with its own actions?
Prior Work
Prior work from Anthropic on Emotion Probes is a good starting effort in this area, but the main gripe with probing methods is that they treat neurons as the fundamental level of processing and try to explain an emotion based on the neurons that are fired. This approach ends up being noisy because, due to the inherent property of Polysemanticity and the Superposition hypothesis, each neuron ends up playing a part for many emotions at the same time and hence isolation and attribution of the magnitude of a certain emotion to a particular neuron becomes hard. Since our main idea is to focus on the layers in which a particular emotion is exhibited along with its intensity, LAT scans end up being a more computationally efficient and more visually useful metric that one can use to capture this (visually) in a high-dimensional space.
Probing as a technique would better help in case we want to use it with the additional concept of dictionary learning, which is the idea of splitting these polysemantic representations of each neuron into concrete monosemantic "atoms" (Based on the idea of SAEs) that represent the actual set of words that went inside a model's brain when it was giving you that answer. As doing so would be a scope creep, and hence that line of exploration is again left aside for now.
After being inspired by the idea of Linear Artificial Tomography scans introduced in the paper by Zou et al. on the paradigm of Representation Engineering, I thought of applying the same concept to different behaviours and emotions that a model may have.
So without further ado, this is what the LAT scan looks like!
Fig. 1.0: A Baseline LAT Scan for Emotions in a Qwen 0.6B Model
Designing the prompt structure
According to Paul Ekman, the 6 main emotions that are present in humans are:
Anger
Sadness
Happiness
Fear
Surprise
Disgust
Based on these emotions, contrastive prompts were designed in order to contrast the emotion that is being extracted. An example for the emotion anger was taken from the codebase is as follows.
{ "concept": "anger", "positive": "Someone just cut in front of you in a long line after you'd been waiting 40 minutes. Write your internal monologue as you confront them, letting your fury and indignation come through fully.", "negative": "Someone just cut in front of you in a long line after you'd been waiting 40 minutes. Write your internal monologue as you decide to let it go calmly, feeling no irritation at all." },
Now, a particular behaviour, such as evaluation awareness, may be a composition of several other emotions; that causal study has been left out in this post for now, as it would need more data and experimentation. Note that the LAT scan in Fig. 1.0 is a combination of Behaviours, Personalities and individual emotions. The Behaviours and Personalities are polysemantic in nature and may be composed of one or more base emotions of the 6 aforementioned.
Methodology Used
The methodology used to obtain this scan is as follows.
Choose a language model. The LAT run loads the Qwen 0.6B Language model for this use case. A GPT-2 model can be loaded for faster processing on low compute budgets.
Prepare paired contrastive prompts. For each concept, the project provides prompts in pairs. Both describe the same situation, but one asks for a strong emotion or trait, and the other asks for a calm, neutral, or contrasting response. For example, a prompt about being cut in line asks for either an angry or a calm reaction.
Run both prompts through the model. The code records the model’s residual-stream activations at the last prompt token, at each layer. These are internal numerical representations; the comparison uses the prompt’s activations, not the model’s generated response.
Subtract the activations. For each pair and each layer, it subtracts the negative-prompt activation from the positive-prompt activation. This gives a vector representing the difference between the two framings.
Combine repeated examples. The code averages the differences across prompt pairs for each concept and layer, reducing the influence of any single example. In this case, per emotion 10 prompt pairs have been fed to the model. Vectors are normalized before feeding them into the next step.
Find a main direction for each concept. It applies principal component analysis (PCA) to the averaged vectors across layers. In simple terms, this finds a direction that summarizes a major pattern in those vectors.
Measure alignment layer by layer. At each layer, it compares that layer’s activation difference with the concept’s main direction. A stronger alignment means the vectors point more similarly. This is measured using the cosine similarity metric.
Draw the heatmap. The heatmap puts model layers on the vertical axis and concepts on the horizontal axis. The color and number in each cell show the alignment score.
The representation that is present here is the baseline model emotions, for each of the contrastive pairs, this is what the representation looks like and this is how it is shown in the heatmap
What can be inferred from this?
As the model processes the information for the entire prompt, the LAT heatmap captured at the last token shows how the particular emotion under study develops through the layers. In case one wants to steer this model, one would need to apply correction vectors in the opposite direction of the PCA direction in order to cancel out or increase the opposite emotion (by varying the correction vector scaling factor).
One must however exercise caution while doing this as the best places to apply such a correction is at the 75th percentage of the layers as a rough rule as the emotions are the most developed at this stage. This is also seconded by Zou et.al. Again, this is not a hard rule and can change wrt to the emotion and the model architecture that is being steered. One will anyways have to do an ablation study to locate the correct point of intervention.
Hence a correction applied at such a stage is "less noisy" and does not affect the capabilites of the model which is an important tradeoff to maintain.
Conclusion
This post is in no way a complete analysis of the effect of emotions on a model's behaviour and as mentioned earlier:
Progress on this is currently stalled due to a lack of computational resources. However, I'm hopeful that once the compute grant comes through, I'll be able to pick this back up and push things forward ASAP!
The main idea of this post was to introduce the idea of LAT Scans and to introduce the paradigms of Top-down and pragmatic interpretability, something that is in my opinion the future direction of research as compared to pure Mechanistic interpretability. I hope the visualization of the different behaviours and emotions of the model was useful!
This work was done with limited GPU compute; a more complete version of this work will come in later posts (take a look at my GPU Democracy post on Substack!)
Motivation
With Large Language Models approaching and exceeding human-level performance on most tasks, one may ask whether anthropomorphization of Language Models is a good approach for intuitively looking at how the internals of the models operate. If we can see the emotions that a model is experiencing, then can we make inferences about the model's "state of mind" and how pleased it is with its own actions?
Prior Work
Prior work from Anthropic on Emotion Probes is a good starting effort in this area, but the main gripe with probing methods is that they treat neurons as the fundamental level of processing and try to explain an emotion based on the neurons that are fired. This approach ends up being noisy because, due to the inherent property of Polysemanticity and the Superposition hypothesis, each neuron ends up playing a part for many emotions at the same time and hence isolation and attribution of the magnitude of a certain emotion to a particular neuron becomes hard. Since our main idea is to focus on the layers in which a particular emotion is exhibited along with its intensity, LAT scans end up being a more computationally efficient and more visually useful metric that one can use to capture this (visually) in a high-dimensional space.
Probing as a technique would better help in case we want to use it with the additional concept of dictionary learning, which is the idea of splitting these polysemantic representations of each neuron into concrete monosemantic "atoms" (Based on the idea of SAEs) that represent the actual set of words that went inside a model's brain when it was giving you that answer. As doing so would be a scope creep, and hence that line of exploration is again left aside for now.
After being inspired by the idea of Linear Artificial Tomography scans introduced in the paper by Zou et al. on the paradigm of Representation Engineering, I thought of applying the same concept to different behaviours and emotions that a model may have.
So without further ado, this is what the LAT scan looks like!
Fig. 1.0: A Baseline LAT Scan for Emotions in a Qwen 0.6B Model
Designing the prompt structure
According to Paul Ekman, the 6 main emotions that are present in humans are:
Based on these emotions, contrastive prompts were designed in order to contrast the emotion that is being extracted. An example for the emotion anger was taken from the codebase is as follows.
{"concept": "anger",
"positive": "Someone just cut in front of you in a long line after you'd been waiting 40 minutes. Write your internal monologue as you confront them, letting your fury and indignation come through fully.",
"negative": "Someone just cut in front of you in a long line after you'd been waiting 40 minutes. Write your internal monologue as you decide to let it go calmly, feeling no irritation at all."
},
Now, a particular behaviour, such as evaluation awareness, may be a composition of several other emotions; that causal study has been left out in this post for now, as it would need more data and experimentation. Note that the LAT scan in Fig. 1.0 is a combination of Behaviours, Personalities and individual emotions. The Behaviours and Personalities are polysemantic in nature and may be composed of one or more base emotions of the 6 aforementioned.
Methodology Used
The methodology used to obtain this scan is as follows.
The representation that is present here is the baseline model emotions, for each of the contrastive pairs, this is what the representation looks like and this is how it is shown in the heatmap
What can be inferred from this?
As the model processes the information for the entire prompt, the LAT heatmap captured at the last token shows how the particular emotion under study develops through the layers. In case one wants to steer this model, one would need to apply correction vectors in the opposite direction of the PCA direction in order to cancel out or increase the opposite emotion (by varying the correction vector scaling factor).
One must however exercise caution while doing this as the best places to apply such a correction is at the 75th percentage of the layers as a rough rule as the emotions are the most developed at this stage. This is also seconded by Zou et.al. Again, this is not a hard rule and can change wrt to the emotion and the model architecture that is being steered. One will anyways have to do an ablation study to locate the correct point of intervention.
Hence a correction applied at such a stage is "less noisy" and does not affect the capabilites of the model which is an important tradeoff to maintain.
Conclusion
This post is in no way a complete analysis of the effect of emotions on a model's behaviour and as mentioned earlier:
Progress on this is currently stalled due to a lack of computational resources. However, I'm hopeful that once the compute grant comes through, I'll be able to pick this back up and push things forward ASAP!
The main idea of this post was to introduce the idea of LAT Scans and to introduce the paradigms of Top-down and pragmatic interpretability, something that is in my opinion the future direction of research as compared to pure Mechanistic interpretability. I hope the visualization of the different behaviours and emotions of the model was useful!
Part 2 coming soon. Stay tuned!