LESSWRONG
LW

Jatin Nainani — LessWrong

Replying toScaling Sparse Feature Circuit Finding to Gemma 9B

Scaling Sparse Feature Circuit Finding to Gemma 9B

You are correct that the current method will only give a set of features at each selected layer. The edges are intended to show the attention direction within the architecture. We updated it to make it more clear and fix some small issues.
We think there are a few reasons why the results of the ACDC paper do not transfer to our domain:
1. ACDC and EAP (Syed et al.) rely on overlap with a manual circuit as their metric, whereas we rely on faithfulness and completeness. Because the metrics are different, the comparison isn’t apples-to-apples.
2. The major difference between methods, as you mentioned, is that we are finding circuits in the SAE basis. This quite

Scaling Sparse Feature Circuit Finding to Gemma 9B

Yes -- By design, the circuits discovered in this manner might miss how/when something is computed. But we argue that finding the important representations at bottlenecks and their change over layers can provide important/useful information about the model.

One of our future directions, along the direction of crosscoders, is to have "Layer Output Buffer SAEs" that aim to tackle the computation between bottlenecks.

Replying toScaling Sparse Feature Circuit Finding to Gemma 9B

Jatin Nainani1y

Scaling Sparse Feature Circuit Finding to Gemma 9B

Thanks a lot for this review!

On strengths, we also believe that we are the first to examine “few saes” for scalable circuit discovery.

On weaknesses,

While we plan to do a more thorough sweep of SAE placements and comparison, the first weakness remains true for this post.
Our major argument for the support of using few SAEs is imaging them as interpretable bottlenecks. Because they are so minimal and interpretable, they allow us to understand blocks of the transformer between them functionally (in terms of input and output). We were going to include more intuition about this but were worried it might add unnecessary complications. We mention the fact about residual stream to highlight that

Diego Caples

Diego Caples, Jatin Nainani, CallumMcDougall, rrenaud

[This is an interim report and continuation of the work from the research sprint done in MATS winter 7 (Neel Nanda's Training Phase)]

Try out binary masking for a few residual saes in this colab notebook: [Github Notebook] [Colab Notebook]

TL;DR:

We propose a novel approach to:

Scaling SAE Circuits to Large Models: By placing sparse autoencoders only in the residual stream at intervals, we find circuits in models as large as Gemma 9B without requiring SAEs to be trained for every transformer layer.
Finding Circuits: We develop a better circuit finding algorithm. Our method optimizes a binary mask over SAE latents, which proves significantly more effective than existing thresholding-based methods like Attribution Patching or Integrated Gradients.

Our... (read 4855 more words →)

Can't agree more with this post! I used to be afraid of long notebooks but they are powerful in allowing me to just think.

Although while creating a script I tend to use "#%%" of vscode to run cells inside the script to test stuff. My notebooks usually contain a bunch of analysis code that don't need to be run, but should stay.

Replying toSAEs are highly dataset dependent: a case study on the refusal direction

Jatin Nainani1y

SAEs are highly dataset dependent: a case study on the refusal direction

Makes sense! Thanks! In that case, we can potentially reduce the width, which might (along with a smaller dataset) help scale saes to understanding mechanisms in big models?

Replying toSAEs are highly dataset dependent: a case study on the refusal direction

Jatin Nainani1y

SAEs are highly dataset dependent: a case study on the refusal direction

Great work! Is there something like too narrow of a dataset? For refusal, what do you think happens if we specifically train on a bunch of examples that show signs refusal?