Top postsTop post
Kola Ayonrinde, Anthropic Fellows, koayon@gmail.com; Jack Lindsey, Anthropic TL;DR: The J-Lens was proposed to read verbalisable representations from language model activations. We introduce the J++ Lens: an improvement to the J-Lens that enables more faithful readouts across 9–284B models. The J++ Lens filters noisy gradients before they enter the averaged...
TL;DR: We recently released two papers about the Philosophy of (Mechanistic) Interpretability [here and here] and a reading list [here]. We believe that building a foundation for interpretability which leverages lessons from other disciplines (philosophy, neuroscience, social science) can help us understand AI models. We also believe this is a...
Adam Karvonen*, Can Rager*, Johnny Lin*, Curt Tigges*, Joseph Bloom*, David Chanin, Yeu-Tong Lau, Eoin Farrell, Arthur Conmy, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Samuel Marks, Neel Nanda *equal contribution TL;DR * We are releasing SAE Bench, a suite of 8 diverse sparse autoencoder (SAE) evaluations including unsupervised metrics and...
This work was produced as part of the ML Alignment & Theory Scholars Program - Summer 24 Cohort. TL;DR The current approach to explaining model internals is to (1) disentangle the language model representations into feature activations using a Sparse Autoencoder (SAE) and then (2) explain the corresponding feature activations...
This work was produced as part of the ML Alignment & Theory Scholars Program - Summer 24 Cohort, under mentorship from Lee Sharkey and Jan Kulveit. Note: An updated paper version of this post can be found here. Abstract Sparse Autoencoders (SAEs) have emerged as a useful tool for interpreting...
Kola Ayonrinde, Anthropic Fellows, koayon@gmail.com; Jack Lindsey, Anthropic TL;DR: The J-Lens was proposed to read verbalisable representations from language model activations. We introduce the J++ Lens: an improvement to the J-Lens that enables more faithful readouts across 9–284B models. The J++ Lens filters noisy gradients before they enter the averaged Jacobian. Using the J++ Lens as a drop-in replacement for the J-Lens, we read workspace representations with 55% success on latent variable extraction tasks (J-Lens: 36%, R-Lens: 38%) with especially strong performance at early layers. We believe that more faithful lenses can be useful in monitoring for eval awareness and unverbalised scheming. We release code, an interactive demo and open-source lenses. Figure 1. By filtering out noisy gradients, the J++ Lens reads intermediate variables more reliably (a) and at earlier layers (b) than the J-Lens and R-Lens. The J++ Lens improves on the J-Lens by adding: (1) : down-weighting the Jacobians from activations that result in poor readouts; (2) : applying Layer-wise Relevance Propagation to the backward pass when computing Jacobians; (3) : removing non-semantic tokens from the readout. (a) The J++ Lens surfaces the correct intermediate readouts at 55.2% recall@10, compared to 35.7% and 37.7% for the J-Lens and R-Lens respectively. (b) The J++ Lens surfaces semantically relevant intermediate latent variables at earlier layers than the J-Lens and R-Lens, layer 24 compared to layer 40 in this example. The intended journey is to hop from the prompt to “heart” to “four chambers”. Bright yellow shading denotes the intermediate variable appearing in the readout, and light yellow shading indicates a closely related readout. Introduction Language model activations contain a large amount of content, most of which is unnecessary for understanding the main latent variables used in internal reasoning. Gurnee et al. (2026) propose that activations can be understood as contain