LESSWRONG
LW

Kanishk Tantia

Message

HDBSCAN is Surprisingly Effective at Finding Interpretable Clusters of the SAE Decoder Matrix

*All authors have equal contribution. This is a short informal post about recent discoveries our team made when we: 1. Clustered feature vectors in the decoder matrices of Sparse Autoencoders (SAEs) trained on the residual stream of each layer of GPT-2 Small and Gemma-2B. 2. Visualized the clusters in 2D...

Oct 11, 2024•8

Kanishk Tantia

Kanishk Tantia — LessWrong

Kanishk Tantia

Message

HDBSCAN is Surprisingly Effective at Finding Interpretable Clusters of the SAE Decoder Matrix

Oct 11, 2024•8

Kanishk Tantia

HDBSCAN is Surprisingly Effective at Finding Interpretable Clusters of the SAE Decoder Matrix

Jaehyuk Lim

Jaehyuk Lim, Kanishk Tantia, Sinem+ 0 more

Jaehyuk Lim, Kanishk Tantia, Sinem

*All authors have equal contribution.

This is a short informal post about recent discoveries our team made when we:

Clustered feature vectors in the decoder matrices of Sparse Autoencoders (SAEs) trained on the residual stream of each layer of GPT-2 Small and Gemma-2B.
Visualized the clusters in 2D after reducing the feature vector space via UMAP

We want to share our thought process, our choices when clustering, and some challenges we faced along the way.

Note: While currently a work in progress, we hope to open-source the clustered features and projections soon, in case others will find it useful.

We would like to acknowledge many people who helped with this endeavor! Professors Shibu Yooseph and Ran Libeskind-Hadas, Alice... (read 2831 more words →)