TL;DR: Mech interp literature rewards learned parts that are bigger than the whole, while searching for the best location to do activation patching. We proved mathematically and tested why a part can have a larger effect, and what the problem with it is. In short, the extra mostly comes from the objective, not the model. Hopefully this scratches an itch many people have had.
Imagine that you ordered a pizza with friends, you cut it to 6 pieces, and somebody says you that "hey bro, this piece is bigger than the whole pizza, isn't it amazing, I want that piece.. 🍕" I mean? How is this even possible? Only if some pieces have negative weight (in a Mech Interp universe maybe).
This is an old problem in new clothes. Putnam argued that (in "Representation and reality", 1988) every ordinary open system, a rock for example, implements every finite-state automaton, as long as you are free to choose the mapping between physical and computational states. Chalmers replied that (in the "Does a Rock Implement Every Finite-State Automaton?", 1996) implementation needs constraints. The mapping has to respect the system’s causal and counterfactual structure.
Geiger and his friends, in "How Causal Abstraction Underpins Computational Explanation", bring Chalmers' answer into interpretability. Implementation is causal abstraction, and the way out of Sutter's result is to put stricter limits on the allowed maps, linear being the natural one. Not generalized though.
And after reading all these findings I also had an itch that something here was still wrong. So I followed my itch and this paper came to life.
Restricting the map is not enough. Even a linear, rank-one map can report more than the network does, because the objective chooses where it lands.
With embarrassingly easy math, we proved that the objectives behind methods like DAS (distributed alignment search) reward pieces that are bigger than the whole. The reward comes from the objective used to learn the piece (e.g., cross-entropy) for activation patching. In other words, reward a bigger effect and gradient descent will happily find you one.
Llama-3.2-3B on RAVEL's city-to-language task: copying the whole state gives 1.00T, DAS trained to hit T gives 0.96T, and DAS rewarded for more gives 2.51T.
The simple Thales geometry behind it
So here is what that picture is saying, no heavy math, promise.
You run the model on two inputs, base and source. At some spot, the activation changes by some vector, call it Δ (source minus base). Patching the whole state means you add all of Δ. Patching a "piece" means you only add the part of Δ along a direction you picked. Call that part .
Now, whatever direction you pick, the piece and the leftover Δ − are always at a right angle:
And if you remember high school geometry, a right angle sitting on a fixed segment means you're on a circle. That's Thales. So every possible piece lands somewhere on a circle (ok, a sphere, whatever) whose diameter runs from base to source.
The answer mostly cares about one direction, the one it reads from. How much a patch moves the answer is just how "high" its point sits along that direction. Right before the output, the full patch lands exactly on the source, so it sits at height T, the total effect.
If you think about it, the top of the circle is only at the source if the change points straight at what the answer reads. It almost never does. In our models the angle between them is usually somewhere around 73 to 83 degrees, so most of the change goes basically sideways. And then the top of the circle sticks out way above T.
At 80° that's about 3.4T, and you can compute it before training anything.
So back to the pizza. The "bigger than the whole" slice is just a point near the top of the circle, and the rest of the slices add up to a negative number that pulls everything back to T. In the movie-review example, cross-entropy DAS lands at 2.98T and the rest of the state contributes −1.98T. 2.98 − 1.98 = 1. Pizza restored.
Where on the circle you end up is the objective's call. Cross-entropy keeps yelling "more!", so it climbs toward the top. We swapped only the loss for a squared error aimed at T (same models, same init, same data), and it landed within a few percent of T across 14 models.
On its own that would be a fun fact, except the field mostly scores these methods by how far they move the answer. A bigger number looks like a better find, so whichever objective pushes hardest wins, whatever the model is actually doing.
We propose the check for this matter, which is quite cheap, and we provide it as a library in the codebase below. Wherever you report a learned effect, copy the whole state at the same spot and put that number next to it. It's a few forward passes. Anything above it, the objective picked.
TL;DR: Mech interp literature rewards learned parts that are bigger than the whole, while searching for the best location to do activation patching. We proved mathematically and tested why a part can have a larger effect, and what the problem with it is. In short, the extra mostly comes from the objective, not the model. Hopefully this scratches an itch many people have had.
A while back Lange, Makelov and Nanda wrote about an interpretability illusion in subspace patching. I mean, by a while back.. 3 years is a long time, however, the problem was still persisting.
The discussion did not stop there of course, Wu et al.'s “A Reply to Makelov et al. (2023)’s ‘Interpretability Illusion’ Arguments” pushed back, and Sutter et al., at The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?, took it further and showed any network can be mapped to any algorithm under mild assumptions with unconstrained, non-linear alignment maps.
This is an old problem in new clothes. Putnam argued that (in "Representation and reality", 1988) every ordinary open system, a rock for example, implements every finite-state automaton, as long as you are free to choose the mapping between physical and computational states. Chalmers replied that (in the "Does a Rock Implement Every Finite-State Automaton?", 1996) implementation needs constraints. The mapping has to respect the system’s causal and counterfactual structure.
Geiger and his friends, in "How Causal Abstraction Underpins Computational Explanation", bring Chalmers' answer into interpretability. Implementation is causal abstraction, and the way out of Sutter's result is to put stricter limits on the allowed maps, linear being the natural one. Not generalized though.
And after reading all these findings I also had an itch that something here was still wrong. So I followed my itch and this paper came to life.
Restricting the map is not enough. Even a linear, rank-one map can report more than the network does, because the objective chooses where it lands.
With embarrassingly easy math, we proved that the objectives behind methods like DAS (distributed alignment search) reward pieces that are bigger than the whole. The reward comes from the objective used to learn the piece (e.g., cross-entropy) for activation patching. In other words, reward a bigger effect and gradient descent will happily find you one.
Llama-3.2-3B on RAVEL's city-to-language task: copying the whole state gives 1.00T, DAS trained to hit T gives 0.96T, and DAS rewarded for more gives 2.51T.
The simple Thales geometry behind it
So here is what that picture is saying, no heavy math, promise.
You run the model on two inputs, base and source. At some spot, the activation changes by some vector, call it Δ (source minus base). Patching the whole state means you add all of Δ. Patching a "piece" means you only add the part of Δ along a direction you picked. Call that part .
Now, whatever direction you pick, the piece and the leftover Δ − are always at a right angle:
And if you remember high school geometry, a right angle sitting on a fixed segment means you're on a circle. That's Thales. So every possible piece lands somewhere on a circle (ok, a sphere, whatever) whose diameter runs from base to source.
The answer mostly cares about one direction, the one it reads from. How much a patch moves the answer is just how "high" its point sits along that direction. Right before the output, the full patch lands exactly on the source, so it sits at height T, the total effect.
If you think about it, the top of the circle is only at the source if the change points straight at what the answer reads. It almost never does. In our models the angle between them is usually somewhere around 73 to 83 degrees, so most of the change goes basically sideways. And then the top of the circle sticks out way above T.
At 80° that's about 3.4T, and you can compute it before training anything.
So back to the pizza. The "bigger than the whole" slice is just a point near the top of the circle, and the rest of the slices add up to a negative number that pulls everything back to T. In the movie-review example, cross-entropy DAS lands at 2.98T and the rest of the state contributes −1.98T. 2.98 − 1.98 = 1. Pizza restored.
Where on the circle you end up is the objective's call. Cross-entropy keeps yelling "more!", so it climbs toward the top. We swapped only the loss for a squared error aimed at T (same models, same init, same data), and it landed within a few percent of T across 14 models.
On its own that would be a fun fact, except the field mostly scores these methods by how far they move the answer. A bigger number looks like a better find, so whichever objective pushes hardest wins, whatever the model is actually doing.
We propose the check for this matter, which is quite cheap, and we provide it as a library in the codebase below. Wherever you report a learned effect, copy the whole state at the same spot and put that number next to it. It's a few forward passes. Anything above it, the objective picked.
Please give it a read if you have time.
Paper · Code (OwnEffect + Lean proofs)