The text below is a brief summary of our research in mechanistic interpretability. First, this article discusses the motivation behind our work. Second, it provides an overview of our previous work. Finally, we outline the future directions we consider important.

Introduction and Motivation

Large language models (LLMs) often represent concepts as linear directions (or "features") within hidden activation spaces [3,4]. Sparse Autoencoders (SAEs) [5–8] help disentangle these hidden states into a large number of monosemantic features, each capturing a specific semantic thread. While one can train a separate SAE on each layer to discover its features, a key unanswered question remains: how do these features persist or transform from layer to layer?

We have approached this question in two papers that progressively deepen our understanding. In Mechanistic Permutability [1], we proposed a method to match features across layers by comparing their SAE parameters, suggesting that many features are re-indexed rather than disappearing as you go deeper. It showed that many features propagate through the layers with consistent semantics, as demonstrated by data-free matching of SAE parameters. Then, in Analyze Feature Flow [2], we focus on how features emerge and transform across layers by constructing "flow graphs" that highlight precisely which submodule (residual, MLP, or attention) generates or propagates each feature, and then leveraging this knowledge for multi-layer steering.

Below, we summarize both papers and highlight their differences.

Mechanistic Permutability (ICLR 2025)

Method Overview. Our original method in [1] sought to find a mapping between features learned at the residual stream at layer $A$ and layer $B$ . We took the decoder weight matrices from SAEs trained on each layer, denoted $W_{d e c}^{(A)}$ and $W_{d e c}^{(B)}$ . We then posed a linear assignment problem to solve for a permutation matrix $P$ that best aligns these columns according to mean squared error. We observed that many columns in layer $B$ are simply re-indexed versions of those in layer $A$ , and corresponding linear directions maintain the same semantic meaning.

Schematic illustration of the permutation matching process: similar features learned by two SAEs have different indices, so we obtain a permutation matrix from one SAE to another in a data-free manner.

Sin...

Posts

Wikitag Contributions

Comments