The goal of ambitious mechanistic interpretability (AMI) is to understand everything. This would be a huge deal if we succeeded.

The key problem is that modern neural networks are enormous objects, and no actual human will ever be able to understand the entire explanation. So we must introduce an assumption, upon which all of AMI rests:

The modularity assumption: The model computation can be explained in terms of a relatively small number of modules, each of which is a clean abstraction over its internals. Each module can also be recursively explained in terms of submodules, all the way down to the atomic computations of the model. There are no irreducibly complex interactions that can only be modeled by keeping a huge number of modules in mind at once.

The modularity assumption doesn’t have to be perfectly true, but the closer it is to true, the more confidence we can have in AMI. If it’s already pretty close to true, we might also be able to modify training to make it more true.

Even supposing modularity, we still won’t have time to look at the entire thing. So if we still have to trust some other model, what does this get us? Why not just use a simpler approach like activation oracles, where we train models to directly tell us what activations mean?

The key thing that AMI buys us is it transforms an OOD generalization problem into an IID generalization problem. This makes the problem fundamentally easier. 

Once we have the modular explanation, a tree of humans could in theory spend a thousand person-years understanding the entire thing, and answering arbitrary questions about the entire system. If we are trying to train a model to do this, we are SFTing models to directly imitate human behavior. The IID alignment problem is vastly easier. It gives you many affordances - for instance, you can spot check the model’s work, or equip it with an OOD detector and check in with a human labeller if the model behavior is surprising.

With activation oracles, we can only train the oracle on things we can verify (“was the next token going to be ‘apple’?”), and then we have to hope that this generalizes to the things we actually care about (“is the model thinking about killing everyone?”). Unfortunately, we only care about interpretability precisely for those questions that we don’t know how to answer using external model behavior! It’s much harder to have confident bounds on how OOD behavior will continue to generalize, especially as models get smarter.