Working on the more applied side of Mechanistic Interpretability (MI) research, I wanted to share some more evidence on why building on top of MI tools both have valuable applications for model performance on existing tasks and enable new applications right now.

Over the past few months, I’ve been working with sparse autoencoders (SAEs) on applications of topic steering (fine-tuning) [#5 on @scasper’s recent post under New Predictions]. In short, they perform well on different steering targets depending on the representativeness of the underlying SAE (you can check out my code/methods on this topic from this summer : https://github.com/IBM/sae-steering ). This is pretty cool and as standard of an improvement as historically dictates in progress towards better fine-tuning approaches.

What I find way more exciting, given my background in the health domain where interpretability and model control is critical, are the new, practical approaches that are now possible in critical domains like health.

Takeaways from Applied MI Research:

1. SAEs for downstream fine-tuning tasks is already practical

Many existing approaches (e.g. Templeton et. al 2024) show how altering SAE neurons can impact generated output. These are limited in practice by finding the right set of neurons from millions of potential neurons and the weight to set them out (as you can observe these limitations in Gemma Steer). Concretely, for SAEs to be used in practice for fine-tuning to a text, we need to be able to:

automatically identify semantically similar neurons to a text (vs. searching millions of neurons)
propose a manipulation scheme that is parameter independent (preventing multiple generation runs)

We address these two limitations using our published methods, which use a pretty simple and efficient approach that a) creates SAE neurons similarity scores using prompt distances and SAE activations which are then b) manipulated based on activations on generated text. You can check out the code/demo for more information. Our results show that this approach both generates linguistically correct text that is similar to fine-tuning targets given the underlying representation of those concepts is present in the SAE (naturally limited by effects like polysemanticity, ghost neurons etc.).

Correctness is a standard metric, but that’s not ...

Posts

Wikitag Contributions

Comments