This blog post discusses a collaborative research paper on sparse autoencoders (SAEs), specifically focusing on SAE evaluations and a new training method we call p-annealing. As the first author, I primarily contributed to the evaluation portion of our work. The views expressed here are my own and do not necessarily reflect the perspectives of my co-authors. You can access our full paper here.

Key Results

In our research on evaluating Sparse Autoencoders (SAEs) using board games, we had several key findings:

We developed two new metrics for evaluating Sparse Autoencoders (SAEs) in the context of board games: board reconstruction and coverage.
These metrics can measure progress between SAE training approaches that is invisible on existing metrics.
These metrics allow for meaningful comparisons between different SAE architectures and training methods, potentially informing SAE design for more complex domains like language models.
We introduce p-annealing, a new SAE training method that improves over prior methods on both existing metrics and our new metrics.
SAEs trained on ChessGPT and OthelloGPT can capture a substantial fraction of the model's board state, with F1 scores of 0.85 and 0.95 respectively for board reconstruction.
However, SAEs do not match the performance of linear probes, suggesting they may not capture all of the model's board state information or "world model".

Challenges with SAE Evaluations

I have an Introduction to SAEs available here.

Sparse Autoencoders (SAEs) have recently become popular for interpretability of machine learning models. Using SAEs, we can begin to break down a model's computation into understandable components. As a result, there have been a flurry of new SAE architectures and loss functions, such as the BatchTopK SAE, Google Deepmind's Gated SAE and JumpReLU SAE, our p-annealing, and OpenAI's TopK SAE.

Unfortunately, we don't have reliable metrics that we can use to compare the new approaches. The main metric currently used is "we looked at activating inputs for a range of features and gave a gut reaction on interpretability of the features". This is a major limitation for the field.

In machine learning, it's ideal to have an objective evaluation of your model, such as accuracy on the MNIST benchmark. With an objective evaluation, you can just twiddle all the knobs of architectures, hype...

Posts

Wikitag Contributions

Comments