Untangling an AI model's semantic associations

I. TL;DR

Epistemic status:

Exploratory replication on a small model (GPT-2 Small, 124M parameters).
Confidence in the methodological findings: moderate
Confidence in generalization to larger models: low

Executive summary:

Applied pre-trained sparse autoencoders to GPT-2 Small to examine how features respond to different text categories (mathematical formulas, python code, URLs, plain text, etc.).
Primary findings:
1. peak activations increase ~3x from layer 6 to 11
2. category-specialist features exist but require proper attention masking to detect
3. specialists appear to detect syntactic patterns rather than semantic concepts (observed, not proven systematically)

II. Introduction

Inspired by Anthropic’s “Scaling Monosemanticity” (2024), this post documents a hands-on exploration of SAE-based interpretability on GPT-2 Small, examining how features respond to different text categories and what they actually detect. The main finding: specialist features detect syntactic patterns rather than semantic concepts, even in later layers. The full code is available via the Phase 1 Jupyter notebook at this project’s GitHub repository.

III. The experiment

Model selection: GPT-2 Small

GPT-2 Small was selected for this project because it offers pre-trained SAE availability via SAELens, TransformerLens support, and tractable scale for consumer hardware.

Sparse autoencoder selection

This project used pre-trained sparse autoencoders (SAEs) from Joseph Bloom's SAELens release, which decompose GPT-2 Small's 768-dimensional residual stream activations into 24,576 sparse features via L1-regularized reconstruction.

An important limitation: Pre-trained SAEs for attention outputs and MLP outputs were not available for this model release. The analysis is therefore restricted to residual stream representations, leaving comparison across component types as a direction for future work.

Dataset construction

To examine how features respond to different types of content, a test dataset of 70 texts was constructed, spanning seven categories. Those categories are described in Figure 1 below. A complete list of the test texts is available via the Phase 1 Jupyter notebook on the project’s GitHub repo.

Figure 1: Test text types used

Each category contained 10 texts, providing a balanced basis for comparing activation patterns across content types.

Measuring feature specializatio

...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments