After many months of reading on the interpretability experiments/blogposts by Anthropic, I noticed that despite being more or less capable of understanding the basic ideas expressed in them, I had zero practical experience and many gaps in my mental model of how sparse autoencoders (or interpretability in general) work.
In the past 2 months I've worked on making nanointerpret.
The main driver was obviously improving my understanding of the field, but an even greater one was providing everyone with an hackablebloat-free repo with (almost) all the steps involved in SAE interpretability from start to finish. I am still very far from having a good picture of this field so the repo should not be taken as a reference implementation, but I think the results are worth sharing!
You can play with the results right now here: nanointerpret.pages.dev or read below to know more about the project and what I learned (or go straight to the repo).
Training the first SAE on the tiniest Gemma 3
I trained the first SAE on Gemma 3 270m, using 500m tokens from FineWeb-edu (this will later turn out to be a mistake) and extracting residuals from layer 9 (middle one). The results were pretty decent for a first run, or at least much better than I expected!
A huge portion of the 16k features were lexical or even based on a single token, but I also found interesting semantic ones:
#382 identified nutrient and ingested substances very cleanly, even at low activations (flavonoids, carotenoid, lead, calcium, vitamin D3, ...)
Beside the lexical / token-based features issue, there was another one: features were polysemantic, for example the "severe harm" feature activated both with casualty-related terms and with employment terms (employment, workers, tokens used to express gained and lost jobs).
In the subsequent training runs I doubled the number of features from 16k to 32k to split polysemantic features and moved from using layer 9 residuals to layer 14 ones. The first change was somewhat successful but the second one, despite producing more abstract features, led to less coherent ones at medium activations already. I came to the conclusion that a 270M model was not enough to properly understand language, even for common english text.
Automatic feature interpretation
Before switching to a bigger LLM, I wanted to build an automated interpretation pipeline that for each feature:
Generates a title: takes high, medium and low activation examples --> gives them to an LLM --> the LLM produces a structured JSON like {"title": "Injuries or severe harm", "category": "semantic"} . (categories available: semantic, lexical, token-specific).
Validates it: a SAE can also be used to artificially force a feature to activate in the model's residual stream. For example when forcing the feature "Injuries or severe harm" and asking the LLM to continue the sentence Today I was at the restaurant and it may produce something like almost everyone died due to a food intoxication. However there may be some "low-quality" features that do not exhibit this behaviour, these features may not truly represent what's in the LLM-assigned title. For this reason the validation step forces the feature to the observed maximum, gives to an LLM both the feature title and observed completion. Then the LLM is prompted to say yes if the completion is coherent with the title and no otherwise. The score uses the tokens probabilities and is .
By the way, local LLMs are great for this task! They may take a bit (2 seconds per feature on my hardware) but Gemma 4 26B for example does a great job comparable to GPT 5.6 Terra, see here for the comparison.
I then switched to Qwen 3 1.7b but kept the edu version of FineWeb and stayed at the middle layer to isolate the effects on the results. You can check the resulting SAE, activations, and interpretation on Huggingface. If you want to play with the results straight from your phone right now, go to nanointerpret.pages.dev.
Next steps
As highlighted before, choosing the edu version of FineWeb was a mistake. When performing interventions it became evident how the features learned by the SAE were biased towards an academic tone. So the completions for each theme are very formal and educational even when it wouldn't be expected by the feature title.
The next experiments will use Qwen residuals from layer 24 (near the end) and switch to FineWeb non-edu version. I'm training them right now on my M4 Max Mac Studio but each training run takes around 30h and its interpretation 20h so it's going to take a bit :)
After many months of reading on the interpretability experiments/blogposts by Anthropic, I noticed that despite being more or less capable of understanding the basic ideas expressed in them, I had zero practical experience and many gaps in my mental model of how sparse autoencoders (or interpretability in general) work.
In the past 2 months I've worked on making nanointerpret.
The main driver was obviously improving my understanding of the field, but an even greater one was providing everyone with an hackable bloat-free repo with (almost) all the steps involved in SAE interpretability from start to finish. I am still very far from having a good picture of this field so the repo should not be taken as a reference implementation, but I think the results are worth sharing!
You can play with the results right now here: nanointerpret.pages.dev or read below to know more about the project and what I learned (or go straight to the repo).
Training the first SAE on the tiniest Gemma 3
I trained the first SAE on Gemma 3 270m, using 500m tokens from FineWeb-edu (this will later turn out to be a mistake) and extracting residuals from layer 9 (middle one).
The results were pretty decent for a first run, or at least much better than I expected!
A huge portion of the 16k features were lexical or even based on a single token, but I also found interesting semantic ones:
#1009identified severe harm (injured, killed, lives, wounded, deaths, dead, lost, ...)#9147identified professions (fisherman, photographer, painter, farmer, artist, ...)#382identified nutrient and ingested substances very cleanly, even at low activations (flavonoids, carotenoid, lead, calcium, vitamin D3, ...)Beside the lexical / token-based features issue, there was another one: features were polysemantic, for example the "severe harm" feature activated both with casualty-related terms and with employment terms (employment, workers, tokens used to express gained and lost jobs).
In the subsequent training runs I doubled the number of features from 16k to 32k to split polysemantic features and moved from using layer 9 residuals to layer 14 ones.
The first change was somewhat successful but the second one, despite producing more abstract features, led to less coherent ones at medium activations already.
I came to the conclusion that a 270M model was not enough to properly understand language, even for common english text.
Automatic feature interpretation
Before switching to a bigger LLM, I wanted to build an automated interpretation pipeline that for each feature:
{"title": "Injuries or severe harm", "category": "semantic"}. (categories available:semantic,lexical,token-specific).Today I was at the restaurant andit may produce something likealmost everyone died due to a food intoxication.However there may be some "low-quality" features that do not exhibit this behaviour, these features may not truly represent what's in the LLM-assigned title.
For this reason the validation step forces the feature to the observed maximum, gives to an LLM both the feature title and observed completion. Then the LLM is prompted to say
yesif the completion is coherent with the title andnootherwise. The score uses the tokens probabilities and isBy the way, local LLMs are great for this task! They may take a bit (2 seconds per feature on my hardware) but Gemma 4 26B for example does a great job comparable to GPT 5.6 Terra, see here for the comparison.
I then switched to Qwen 3 1.7b but kept the edu version of FineWeb and stayed at the middle layer to isolate the effects on the results. You can check the resulting SAE, activations, and interpretation on Huggingface. If you want to play with the results straight from your phone right now, go to nanointerpret.pages.dev.
Next steps
As highlighted before, choosing the edu version of FineWeb was a mistake. When performing interventions it became evident how the features learned by the SAE were biased towards an academic tone. So the completions for each theme are very formal and educational even when it wouldn't be expected by the feature title.
The next experiments will use Qwen residuals from layer 24 (near the end) and switch to FineWeb non-edu version. I'm training them right now on my M4 Max Mac Studio but each training run takes around 30h and its interpretation 20h so it's going to take a bit :)