To those who are not new to mechanistic interpretability – describing features has been a main domain in this area in past years.
In the past, researchers used to think that neurons inside the model are an atomic unit, but after some careful investigation they discovered that neurons are responsible for more than one concept and they activate as groups many times. Thus, a shift occurred to view "features" as directions (that are composed from groups of neurons).

Now that we establish how concepts are represented in the model, we can try to find them, play with them, and even make them stronger (enhance/suppress them, have a look at Anthropic’s blog on steering a model – https://www.anthropic.com/news/mapping-mind-language-model, a nice example of it).

Although much research digs into the attention part of transformers, the authors decided to search for important features inside the MLP (relying on previous papers claiming that MLPs possess many important signals and due to the fact that most of the model’s parameters lie in the MLP).

In the introduction & related work, the authors mention early supervised and unsupervised methods to detect features in the model – but they mainly focus on SAEs as the baseline comparison.
The famous SAE, probably the most common in the unsupervised area, has (according to the authors and other papers) three main issues:

They are not actually part of the model.
They often “catch” features that aren’t necessarily what the model itself uses.
They often cannot steer model behavior as well as expected.

They propose a new method to decompose MLP activations into interpretable features: Semi-Nonnegative Matrix Factorization (SNMF), which is an adaptation of the classic NMF (an NP-hard problem).

The method is as follows:

They gather activations from the MLP across many tokens (let’s say $n$ ) and compose a matrix A (size $d a \times n$ ), which is then decomposed into matrices $Z \in R^{d_{a} \times k}$ and $Y \in R_{\geq 0}^{k \times n}$ Both Y,Z start as samples from uniform and normal distributions respectively (values in (0,1)), and are optimized via multiplicative-update methods (SNMF is non-convex but we won’t dive into those details).

Matrix Z learns neuron groups: SNMF groups co-firing neurons. Each feature is a sparse pattern of neurons that activate together.
Matrix Y links features to tokens: for each token $a_{j}$ (from the columns of A), it give...

Posts

Wikitag Contributions

Comments