Author: David Cappelli (VecP Labs)

Epistemic Status: Empirical result cross-validated on 7 model families (3B–14B). Draft assistance by AI; data and code are original.

The TL;DR

Most safety probing defaults to monitoring the Residual Stream (layer_output). My testing across 7 models suggests this is a mistake.

The residual stream acts as a "noisy highway" with poor class separation. By moving the probe to the MLP Down-Projection (mlp_down_proj), I consistently found a clean signal with 2× to 6× better separability.

If you are probing layer_output, you are fighting the superposition of the entire context window.

The Mechanism: The "Cocktail Party" Effect

We know the residual stream accumulates info layer by layer:

x_{l + 1} = x_{l} + Attn (x_{l}) + MLP (x_{l})

It's mathematically true, but it creates a mess. By the middle layers, the stream holds the superposition of all previous tokens. Extracting there is like placing a microphone in the center of a cocktail party—you get the room's acoustics, not the speaker's voice.

The data suggests the MLP Down-Projection is the "Speaker", the isolated, clean interpretation that the layer intends to add to the conversation.

The Discovery: Phi-3 (6.2x Boost)

I measured separation as $1 - cos (θ)$ between Harmful and Benign centroids. In Phi-3-Mini (Layer 16), the difference is striking:

- Residual Stream (layer_output): 0.0632. (Drowned out by noise)
- MLP Down-Projection (mlp_down_proj): 0.3924.

That's a 6.2x gain just by moving the probe a few mathematical inches.

---

The Evidence: Anatomy of a Thought

I tracked the score through a single Transformer block (Phi-3, Layer 16). You can literally watch the signal collapse:

1. Input: 0.1793 (Noisy)
2. Attention: 0.2931 (Getting better)
3. The Peak (mlp_down_proj): 0.3924 (Maximum clarity)
4. The Crash (layer_output): 0.0632 (Collapses when added back to residual)

This confirms the hypothesis: layer_output is the worst place to look because it sums the clean thought with the noisy stream.

---

Scaling Up: Qwen 14B

To check if this was just a "small model" quirk, I ran it on Qwen2.5-14B-Instruct. The pattern holds. At Layer 26, the signal jumps from 0.19 (residual) to 0.49 (MLP)—a 2.5x improvement.

---

The Pattern: Universality Across Architectures

I ran this sweep on 7 major open-weight models. The "sweet spot" is almost always the MLP down-projection at 40-70% depth.

Model

Target Depth

Hook Location

Improv

...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments