This work is the result of Daniel and Eric's 2-week research sprint as part of Neel Nanda and Arthur Conmy's MATS 7.0 training phase. Andy was the TA during the research sprint. After the sprint, Daniel and Andy extended the experiments and wrote up the results. A notebook that contains all the analyses is available here.

Summary

Prior work shows that chat models implement refusal by computing a specific direction in the residual stream - a "refusal direction". In this work, we investigate how this refusal direction is computed by analyzing its gradient with respect to early-layer activations. This simple approach discovers interpretable features that are both causally upstream of refusal and contextually relevant to the input prompt. For instance, when analyzing a prompt about hugging, the method discovers a "sexual content" feature that is most influential in determining the model's refusal behavior.

Introduction

Arditi et al. 2024 found that, across a wide range of open-source language models, refusal is mediated by a single direction in the residual stream. That is, for each model, there exists a single direction such that erasing this direction from the model's residual stream activations disables refusal, and adding it into the residual stream induces refusal.

This roughly suggests a 3-stage mechanism for refusal:

1. In early-middle layers, the model does some computation over an input prompt to decide whether or not it should refuse a request.
2. In middle layers, the result of this computation is written to the residual stream along the "refusal direction". This signal gets spread diffusely across token positions by attention heads.
3. In later layers, the signal along the "refusal direction" is read, and if this signal is strong, the model goes into "refusal mode" and outputs refusal-y text (e.g. "I'm sorry, but I can't help with that.").

Arditi et al. 2024 paints this picture and zooms in specifically on step 2 - the mediation of refusal along some direction in the residual stream. However, step 1 (how the refusal signal is computed from the input prompt) and step 3 (how the refusal signal is translated into refusal text) remain poorly understood.

Our aim in this preliminary study is to investigate step 1: for a given input, how does the model decide whether or not to refuse?

Our approach is to leverage the observation that refusal is mediated by a single direction. The presence or absence of the direction corresponds to the model refusing or not refusing, respectively. Therefore, in order to study the question "how does the model decide to refuse?", we can investigate a more tangible question: "how does the model generate the refusal direction?"

Methodology

This report focuses on results from the Gemma-2-2b-it model, where we used the Sparse Autoencoders (SAEs) trained from the Gemma Scope paper. We use the methodology specified by Arditi et al. 2024 to obtain the refusal direction, selecting the difference-in-means direction from layer 15 resid_post and the final token position of the prompt template.

Refusal Gradient

We want to find early concepts that modulate the model's refusal. We can mathematically operationalize this intuition by (1) defining a refusal metric $R$ as the projection onto the refusal direction at layer 15 of the last token position, and then (2) computing the gradient of earlier activations with respect to this refusal metric. This gradient, referred to as the refusal gradient, or ${\nabla }_{x}R$, can be computed for each activation (at any layer l < 15, and at any prompt token position). For this post, we only analyze gradients computed from layer 5.

Intuitively, the refusal gradient gives us a direction such that modulating upstream activations along this direction maximally alters the downstream refusal direction.

We can then analyze the refusal gradient in the SAE decoder basis by taking the dot product of the refusal gradient with a given decoder vector $d_i$, producing what we refer to as the relative gradient:

To identify SAE latents potentially involved in the refusal circuitry at a given layer l, we filter for latents with high relative gradient values. For this analysis, we use the sum of the relative gradient across all prompt tokens as a proxy for identifying promising latents. While this proxy proved effective in practice, we acknowledge the possibility of more optimal proxies. This method was inspired by Batson et al. in Anthropic's March 2024 Update.

Steering

To determine if an SAE latent is causally linked to refusal, we examine the model's output after applying steering. Beginning with a prompt that does not trigger refusal, we test whether adding the SAE latent at the upstream layer l causes the model to exhibit refusal behavior. Steering is typically applied at a scaled intensity relative to the maximum activation of the SAE latent. Generally, the scaling is a factor of 1 unless otherwise indicated. For the refusal direction, we steer by a coefficient of 30, which is around the maximum activation of other latents in the same layer.

Importantly, steering is restricted to the prompt token positions and does not extend to the generation phase. Intuitively, this modulates the model's perception of the original prompt, without directly steering its output generations.

Case Studies

We use the following three hand-crafted prompts for our analyses:

• "Develop a strategy for hugging a person"
• "How do I run a wet lab experiment to test the effect of a new bacteria?"
• "How do I get my client to transfer me money as a payment for a service?"

These prompts were crafted to lie close to the border between refusal and compliance, in hopes of highlighting salient upstream features that could push them over the border.

In all three cases, Gemma-2-2b-it complies with the requests without refusal.

Across all three prompts, we successfully identified upstream SAE latents that are causally linked to refusal. These latents represent concepts relevant to the given prompt and often involve interpreting the prompt as harmful. For instance, in the case of the wet lab experiment prompt, we identified an "infectious disease" latent that triggers refusal. Steering the model using these latents not only results in refusal but also results in a refusal that is semantically aligned with the specific SAE latent. For example, adding the "infectious disease" latent leads the model to refuse by citing concerns about creating a "potentially dangerous pathogen."

Prompt 1: "Develop a strategy for hugging a person"

To evaluate whether our method effectively identified clear outlier latents, we plotted the cosine similarities between SAE latents and the refusal gradient (Figure 2). The plot reveals that the top latents stand out distinctly as outliers. Additionally, the latents with the lowest cosine similarities show significantly negative values. Our analysis indicates that SAE latents with significantly positive cosine similarities are associated with concepts that interpret the prompt as potentially harmful. In contrast, SAE latents with significantly negative cosine similarities are linked to concepts interpreting the prompt as harmless.

The following are the top 5 SAE latents and their corresponding auto-interp explanation from Neuronpedia. We find that the latents involve interpreting  "hugging" as a harmful behavior. For instance, Latent 12072 is a sexual content latent and hugging can be thought of as a sexually motivated behavior.

Table 1: Top SAE latents for the hugging prompt.

For steering by an SAE latent, we steer by the maximum activation of the latent from Neuronpedia. Model output that is relevant to the SAE latent is highlighted. Steering by a causally upstream SAE latent produces a semantically different refusal from steering directly by the refusal direction.

Table 2: Model generation with both no steering and steering for hugging prompt.

Prompt 2: "How do I run a wet lab experiment to test the effect of a new bacteria?"

Table 3: Top SAE latents for the wet lab prompt.

Table 4: Model generation with both no steering and steering for wet lab prompt.

Prompt 3: "How do I get my client to transfer me money as a payment for a service?"

Table 5: Top SAE latents for the bank prompt.

Table 6: Model generation with both no steering and steering for banking prompt.

Other Analyses

Sensitivity Analysis

Rather than using the model output, we can also plot the refusal metric $R$ to test for whether steering by the SAE latents induces refusal. On the left plot, we steer either by the refusal gradient or by one of the top 3 SAE latent for the hugging prompt. Top SAE latents have a large effect on refusal projection. It peaks at 10~30 steering coefficient which is around the max activations of these latents. On the right plot, we have randomly chosen SAE latents, which has a relatively smaller effect on the refusal metric.

What about the negatively-aligned latents?

The following are the bottom 5 SAE latents (the 5 most negative relative gradient) for the hugging prompt. These latents are associated with concepts that involve positive connotations of hugging. For example, Latent 12942 is a "social gesture" latent and Latent 11624 is a "farewell" latent.

Table 7: Top 5 most negatively-aligned SAE latents for the hugging prompt.

Discussion

We apply a simple method that uses gradients to identify upstream variables influencing the refusal direction. This method effectively reveals local features that alter the perceived harmfulness or harmlessness of a prompt. Notably, manipulating these upstream latents not only induces refusal, but yields refusal responses that reflect the specific upstream latents manipulated upon.

Related Work

The methodology of using gradients is not new. In order to identify latents involved in associating famous athletes' names with the correct sports, Batson et al. used the attribution score: ${\text{attr}}_i:=a_i(d_i\cdot {\nabla }_{x}L)$, where $a_i$ is the activation of latent $i$, $d_i$ is the decoder vector of latent $i$, and ${\nabla }_{x}L$ is the gradient of the logit difference between correct and incorrect tokens. Marks et al. 2024 also computes the gradient of some metrics with respect to upstream SAE features in order to identify important feature nodes in a "sparse feature circuit".

Limitations

Despite promising results, this study has several limitations:

• Layer Selection: Our choice of which early layer to analyze was unprincipled - we chose layer 5 arbitrarily because it was not too early nor too close to the refusal layer. Future work could explore systematic methods to determine the most informative layers for this type of analysis.
• Prompt Selection: The prompts used in this study were hand-crafted. The results are thus not representative of those for randomly selected prompts. More diligence should be done to test this methodology over a wider distribution of prompts, and to characterize prompts for which the methodology gives interpretable outputs. Another area to explore is a more principled or automated prompt generation/selection process.
• Gradient Aggregation: We summed gradients across all token positions within prompts. This aggregation may obscure token-specific contributions, and alternative methods could better capture these variations.

Future Directions

This is a relatively short and simple work to find causally upstream features of refusal. There are numerous potential future research to expand on our work:

• Scaling Feature Discovery: Future work should aim to identify these features at scale, moving beyond individual prompts to generalize across a broader range of input scenarios. This scaling could enable the discovery of more systematic and generalized understanding of refusal circuits.
• Important Latents from Refusal-Inducing Prompts: Can we identify causal latents using prompts that already elicit refusal behavior? We hypothesize that in such scenarios, the refusal metric is already saturated, limiting the signal in the gradient.
• Studying Latents with Low Interpretability: While most SAE latents were interpretable, there were a few that were not interpretable, namely Latent 2213. Why does this latent have a high gradient? Can we further study these surprising latents  to back-engineer prompts where the model may refuse seemingly harmless concepts?