LESSWRONG
LW

by crayhippo
1 min read1 answerNo comments

0

Inner AlignmentInterpretability (ML & AI)Sparse Autoencoders (SAEs)AI
Frontpage

Does it make sense to extract sparse feature graph for a behavior from only residual layers of gpt2 small or do we need all mlp and attention as well?

SAE sparse feature graph using only residual layers
New Answer
New Comment

1 Answers sorted by
top scoring

Joseph Bloom

20

I think so, but expect others to object. I think many people interested in circuits are using attn and MLP SAEs and experimenting with transcoders and SAE variants for attn heads. Depends how much you care about being able to say what an attn head or MLP is doing or you're happy to just talk about features. Sam Marks at the Bau Lab is the person to ask.

[-]crayhippo10

Thank you for the feedback, and thanks for this.

Who else is actively pursuing sparse feature circuits in addition to Sam Marks? I'm curious because the code breaks in the forward pass of the linear layer in gpt2 since the dimensions are different from Pythia's (768). 

Reply
1Joseph Bloom
SAEs are model specific. You need Pythia SAEs to investigate Pythia. I don't have a comprehensive list but you can look at the sparse autoencoder tag on LW for relevant papers.
Moderation Log