*All authors have equal contribution.

This is a short informal post about recent discoveries our team made when we:

Clustered feature vectors in the decoder matrices of Sparse Autoencoders (SAEs) trained on the residual stream of each layer of GPT-2 Small and Gemma-2B.
Visualized the clusters in 2D after reducing the feature vector space via UMAP

We want to share our thought process, our choices when clustering, and some challenges we faced along the way.

Note: While currently a work in progress, we hope to open-source the clustered features and projections soon, in case others will find it useful.

We would like to acknowledge many people who helped with this endeavor! Professors Shibu Yooseph and Ran Libeskind-Hadas, Alice Woog, Daniel Tan, and many others who offered help and guidance.

TL;DR

SAE decoder weight matrix alone can be used to find (some) feature families.
While HDBSCAN classifies most features as “noise”, the remaining clusters encapsulate features that activate on similar concepts and tokens in high-dimensions
1. HDBSCAN is not able to find all clusters of features
2. We propose using the Cech Complex to identify persistent sets of features via filtration.

Our primary finding that the decoder weight matrix alone can be used find feature families may help with:

Auto-interpretability: It should be easier, cheaper, and more powerful to classify groups of features than find explanations for a single feature.
Safety Guardrailing: Interpretable clustering techniques can be used to elucidate the geometry of the SAE feature space. We propose a guardrail framework and discuss the utilization of appropriate abstract spaces, such as Čech complexes, to ensure safety and reliability.
Reverse Feature Splitting: We discuss how clustering can be viewed as “reverse splitting” features, finding natural relationships between existing features and grouping them together to find stronger activations or as a sign of feature absorption in SAEs.

Section 1: Motivations and Relevant Prior Work

A major challenge to interpreting large language models (LLMs) lies in the model seemingly representing more information than it has dimensions. For example, polysemantic neurons tend to be the rule rather than the exception (Bricken et al. 2023; Cunningham et al. 2023). Leading theories propose that perhaps all the encoded features making up the residual stream (as w...

Posts

Wikitag Contributions

Comments