LESSWRONG
LW

StefanHex — LessWrong

4mo

By design, LLMs perform nonlinear mappings from their inputs (text sequences) to their outputs (next-token generations). Some of these nonlinearities are built-in to the model architecture, but others are learned by the model, and may be important parts of how the model represents and transforms information. By studying different aspects of this nonlinear mapping, we can aim to understand how these models make sense of their inputs and respond to them.

An example of such learned nonlinearities is that of activation plateaus—a phenomenon where interpolating from one input sequence to another can produce minimal downstream changes for most of the interpolation, with a sudden rapid 'phase change' somewhere in the middle. This raises... (read 2134 more words →)

Transformers Don't Need LayerNorm at Inference Time: Implications for Interpretability

submarat

submarat, Joachim Schaeffer, Luca Baroni, galvsk, StefanHex

7mo

This work was produced during MARS and SPAR. arXiv version available at https://arxiv.org/abs/2507.02559. Code on GitHub and models on HuggingFace.

TL;DR we scaled LayerNorm (LN) removal by fine-tuning to GPT-2 XL:

We improve training stability by regularizing activation standard deviation across token positions & improve the training code.
The process is still not very data efficient (requires about ~1% of original dataset) and fine-tuning is somewhat unstable on larger models.
Surprisingly, attribution patching is not more accurate in LN-free models.
As expected, direct logit attribution becomes exact, and entropy neurons stop working.

We provide drop-in replacements for GPT-2 small, medium, large, and XL on HuggingFace for all your interpretability needs!

Introduction

LN centers the activations by the mean and normalizes

... (read 2076 more words →)

Trusted monitoring, but with deception probes.

Avi Parrack

Avi Parrack, StefanHex, Cleo Nardo

7mo

Recently, Apollo trained some deception probes (Goldowsky-Dill et al). A deception probe is a logistic classifier on the AI's internal activations, indicating whether a token belongs in a deceptive response. We benchmarked these deception probes, testing them across five datasets of strategic deception and comparing them to black-box monitoring. We consider various strategies that make use of the deception probe to classify the overall response.

Coming up: (1) our takeaways, (2) Try it Yourself, (3) paper summary.

1. Our takeaways

Deception probes work well for trusted monitoring: we replicate and extend Apollo’s findings, catching 97% of the deceptive responses while misclassifying only 1% of honest responses.^[1]
Deception probes beat the black-box monitor on most datasets. In

... (read 1002 more words →)

Compressed Computation is (probably) not Computation in Superposition

Jai Bhagat

Jai Bhagat, Sara Molas Medina, Giorgi Giglemiani, StefanHex

8mo

This research was completed during the Mentorship for Alignment Research Students (MARS 2.0) Supervised Program for Alignment Research (SPAR spring 2025) programs. The team was supervised by Stefan (Apollo Research). Jai and Sara were the primary contributors, Stefan contributed ideas, ran final experiments and helped writing the post. Giorgi contributed in the early phases of the project. All results can be replicated using this codebase.

Summary

We investigate the toy model of Compressed Computation (CC), introduced by Braun et al. (2025), which is a model that seemingly computes more non-linear functions (100 target ReLU functions) than it has ReLU neurons (50). Our results cast doubt on whether the mechanism behind this toy model is... (read 2972 more words →)

Try training token-level probes

StefanHex

10mo

Epistemic status: These are results of a brief research sprint and I didn't have time to investigate this in more detail. However, the results were sufficiently surprising that I think it's worth sharing. I'm not aware of previous papers doing this but surely someone tried this before, I would welcome comments pointing to existing literature on this! [Edit: Orgad et al. (2024) does something related but different, see my comment below.]

TL,DR: I train a probe to detect falsehoods on a token-level, i.e. to highlight the specific tokens that make a statement false. It worked surprisingly well on my small toy dataset (~100 samples) and Qwen2 0.5B, after just ~1 day of iteration! Colab... (read 2233 more words →)

Proof-of-Concept Debugger for a Small LLM

Peter Lai

Peter Lai, StefanHex

11mo

We want to show that it is possible to build an LLM “debugger” using SAE features and have developed a prototype that automates circuit visualizations for arbitrary prompts. With a few improvements to existing techniques (notably, “cluster resampling”, which is a form of activation patching), we are able to produce visually interpretable results. Here’s an annotated circuit processing the letters in the word “thinks”:

**Predicting a space character after the letters in “thinks”**

Our main contributions with this work are:

With a “debugger”, we automate sparse circuit visualizations for arbitrary prompts. Our work uses a small 216k-parameter model, but we hope to scale up the technique to larger LLMs.
We develop a novel resampling technique, “cluster

... (read 3020 more words →)

Detecting Strategic Deception Using Linear Probes

Nicholas Goldowsky-Dill

Nicholas Goldowsky-Dill, bilalchughtai, StefanHex, Marius Hobbhahn

Can you tell when an LLM is lying from the activations? Are simple methods good enough? We recently published a paper investigating if linear probes detect when Llama is deceptive.

Abstract:

AI models might use deceptive strategies as part of scheming or misaligned behaviour. Monitoring outputs alone is insufficient, since the AI might produce seemingly benign outputs while its internal reasoning is misaligned. We thus evaluate if linear probes can robustly detect deception by monitoring model activations. We test two probe-training datasets, one with contrasting instructions to be honest or deceptive (following Zou et al., 2023) and one of responses to simple roleplaying scenarios. We test whether these probes generalize to realistic settings where Llama-3.3-70B-Instruct behaves

... (read 319 more words →)

104

SAE regularization produces more interpretable models

Peter Lai

Peter Lai, StefanHex

Sparse Autoencoders (SAEs) are useful for providing insight into how a model processes and represents information. A key goal is to represent language model activations as a small number of features (L0) while still achieving accurate reconstruction (measured via reconstruction error or cross-entropy loss increase). Past research has focused on improving SAE training techniques to address a trade-off between sparsity and reconstruction quality (e.g. TopK, JumpReLU). Other efforts have explored designing more interpretable LLMs (e.g. Softmax Linear Units, Bilinear MLPs, Codebook Features, Adversarial Interpretability (Thurnherr et al. (in preparation)).

Here we propose improving model interpretability through adding a regularization term during model training. This involves training an LLM alongside multiple SAE layers, using... (read 985 more words →)

Attribution-based parameter decomposition

Lucius Bushnaq

Lucius Bushnaq, Dan Braun, StefanHex, jake_mendel, Lee Sharkey

This is a linkpost for Apollo Research's new interpretability paper:

"Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition".

We introduce a new method for directly decomposing neural network parameters into mechanistic components.

Motivation

At Apollo, we've spent a lot of time thinking about how the computations of neural networks might be structured, and how those computations might be embedded in networks' parameters. Our goal is to come up with an effective, general method to decompose the algorithms learned by neural networks into parts that we can analyse and understand individually.

For various reasons, we've come to think that decomposing network activations layer by layer into features and connecting those features up into circuits... (read 1114 more words →)

108

Analyzing how SAE features evolve across a forward pass

bensenberner

bensenberner, danibalcells, Michael Oesterle, Ediz Ucar, StefanHex

This research was completed for the Supervised Program for Alignment Research (SPAR) summer 2024 iteration. The team was supervised by @Stefan Heimersheim (Apollo Research). Find out more about the program and upcoming iterations here.

TL,DR: We look for related SAE features, purely based on statistical correlations. We consider this a cheap method to estimate e.g. how many new features there are in a layer and how many features are passed through from previous layers (similar to the feature lifecycle in Anthropic’s Crosscoders). We find communities of related features, and features that appear to be quasi-boolean combinations of previous features.

Here’s a web interface showcasing our feature graphs.

**Communities of sparse features through a forward pass**. Nodes represent residual stream

... (read 168 more words →)

LESSWRONG
LW

LESSWRONG
LW

StefanHex

You can remove GPT2’s LayerNorm by fine-tuning for an hour

A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team

Solving the Mechanistic Interpretability challenges: EIS VII Challenge 1

Attribution-based parameter decomposition

StefanHex

StefanHex

Activation Plateaus: Where and How They Emerge

Transformers Don't Need LayerNorm at Inference Time: Implications for Interpretability

Trusted monitoring, but with deception probes.

Compressed Computation is (probably) not Computation in Superposition

Try training token-level probes

Proof-of-Concept Debugger for a Small LLM

Detecting Strategic Deception Using Linear Probes

StefanHex

You can remove GPT2’s LayerNorm by fine-tuning for an hour

A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team

Solving the Mechanistic Interpretability challenges: EIS VII Challenge 1

Attribution-based parameter decomposition

StefanHex

StefanHex

Activation Plateaus: Where and How They Emerge

Transformers Don't Need LayerNorm at Inference Time: Implications for Interpretability

Trusted monitoring, but with deception probes.

Compressed Computation is (probably) not Computation in Superposition

Try training token-level probes

Proof-of-Concept Debugger for a Small LLM

Detecting Strategic Deception Using Linear Probes

Introduction

1. Our takeaways

Summary

Motivation

Treat your obfuscated chains of thought like live bioweapons.

Against multi-page forms.

Is weight linearity real?