I strongly disagree with the thesis this post defends. There has been a never-ending debate in the world of XAI (explainable AI) about what a good explanation of a computation done by AI is. Just focusing on very simple image classification networks, we have prototype networks, attribution maps, deepdream counterfactuals, circuits, some attempts at natural explanations, and so on. To assess the quality of such methods, dozens of metrics have been proposed. As you suggested, simplicity is one. But none of them has superseded the others because nobody has done the hard work of defining exactly what we want to explain. One might be interested in predicting the OOD behaviour, the adversarial structure, the trigger for some specific class, potential backdoors... I don't see a strong reason why a score that works for one will help you predict the others.
More broadly, I don't think there is a great risk of maximising the wrong thing with this kind ot strategy (Goodhart's Law strikes again). You should start with your [theory of change for interpretability](https://www.alignmentforum.org/posts/LNA8mubrByG7SFacm/against-almost-every-theory-of-impact-of-interpretability-1), and start from there. Here are — from the top of my mind — some ways the metric could break:
1. You expect the simplest network to yield some magical description that a human can study and understand. You get a huge graph with relations that make no sense to any human
I think if we had a way to automate mech-interp to get truly useful results, we would have solved a big part of alignment.
I also have been thinking about making mechinterp verifiable myself and I think it is maybe posible, it's just the approach on the post is missing something. I think that it's true that the simplest explanation won't be understandable but the simplest explanation in terms of semantic content that a human can understand might be thou. And we can use old LLM as a proxy of "human understandable" if you can generate simple text that allows different LLM to make correct predictions about the model in different situations maybe you have a good explanation that humans would also understand. Especially if you do things like rephrasing to check there's no tricks. Similar concept is autointerp scores, I'm just saying we could remove the SAE part and have the models decide how to divide the activations.
If we had some simple human understandable explanation that's simple in terms of semantic concepts, I would expect it to generalize OOD and be usefull for a lot of the things you mention.
We would be on a much better situation for fixing problems with models if we had at least an idea of how they work internally.
It would be easy to check if the beautiful looking explanation doesn't actually predict stuff correctly on test data.
Re: autointerp like we did with SAEs
The largest problem with this agenda is selecting what to do auto-interp on. Sparse linear directions wasn't the correct object. I am proposing more weight-based decomposition, but the main problem I'd like to solve is to figure out what is the correct object to explain. We can look at full bilinear layers or stacks of them. In tract 2, I briefly explain how we can define any sort of composition between model components with tensor transformers, but it's unclear what groups of them/distributed sets of them those should be.
But once we do, ya I think we can do this.
Althooouugh, we can do a subset of semantic explanations. I also explain in tract 2 that we can define the difference vector between two words in the unembedding ("myself" vs "yourself") and find the minimal part of the model that's responsible for telling those apart. We can do the same for single unembedding tokens (eg "3", " the" , etc). Aren't these semantic explanations? eg "This is the part of the model that predicts [numbers]".
If so, then I can find the part of the model that computes that (well, not as rosy as I'm saying, I haven't thought through all of attention from unembedding back, only embedding forward). I don't think that'll be enough. You'll have a collection of model subsets that predict different tokens or token-diffs, but then you'll need to combine them somehow. Ideally the similarity between them also has a semantic explanation for you to continue doing this?
Generally agree on "control-protocals and auto-interp", but I do think we can use tensor transformers to get way more information out of weight-based information than having to rely on activations (this is what I meant by tract 2).
What definition of "simple" do you have in mind here? My closest answer is "circuit simplicity"; is there a reason that one would fail?
Tensor transformers do handle 2,3, (& maybe 4?)
It allows you to state your hypothesis as a tensor and analytically check how well that matches the model's computation. This avoids the dataset-based method issues you've mentioned. In a pure polynomial/tensor transformer, you can rely on only the weights.
That said, in practice, we can mostly do this, with the non-exact components stated explicitely. Additionally, using data has also been helpful for narrowing things down (see the 1000 circuit section), but I am intending on using this information to inform the weight-only based decompositions.
Maybe two cruxes here?
A. Every concept/feature/circuit the model learned is helpful for predicting language, so that's always one form of understanding we'll have. There might be a bag-of-words that's predictive of court-case documents, and I'll just accept as a statistical fact. Additionally, the input & output are basis-aligned; they do have semantic meaning by default (steganography concerns aside).
B. If several different tasks have overlapping components (ie a simpler circuit should re-use functions that do the same thing), then both (i) the fact that that component is shared across those specific tasks is info & (ii) understanding that component helps understand all those different tasks. This is a bit of a category-theory-esque view that what you are is equivalent to all your relationships.
I don't see a strong reason why a score that works for one [goal] will help you predict the others.
This is a great point, and I've had to sit with it for a bit. I now think that both properties I want from a circuit, (2) minimal extraction & (3) removal, to be decomposition-dependent. As a simple example:
y = ab + ac = a(b + c)
here, we could decompose it as either "ab" & "ac" or "a" & "(b + c)". I could imagine many tasks that use "ab" a lot, and others that use "(b + c)" a lot, and there will be multiple valid decompositions that are more minimally-extractable & removable for different circuits.
Before I explain where I disagree, I should say that I’m quite sympathetic to the overall message. I for example think using scalable oversight to do automated mechinterp is an idea with a lot of promise (though this is by no means a low-risk path, I would prefer pausing etc)
The core reason I think this is much less clear cut than the post implies is that the true objective of interpretability is something like
Give an explanation of what the neural network does so that humans can make advance predictions about what the AI will do in important situations
The two italics being what makes the project less verifiable than we might hope.
Algorithmic complexity / Description Length are useful proxies here, at least for the “human” part, because it is easier for us to keep small things in our head. But they definitely come apart. To illustrate this you can take production code and code golf it, and see if it is easier to understand afterwards.
Of course, how quickly these come apart is an empirical question. My hunch is that they come apart quickly enough that naive implementations will not be useful, but that with enough patches, they can be. I expect them to come apart slower than e.g. sparsity and interpretability came apart in SAEs.
I think the second problem is bigger. We could imagine getting good enough at interpretability that we could replace a neural network with a list of python programs, each understandable by humans, which together predict 99% of individual token predictions of the original network, and then getting killed by surprise because the remaining 1% of failures were concentrated in the set of trajectories where the model had the opportunity to take actions leading to takeover.
Avoiding this requires either getting ~100% explanation (which seems infeasible, as we know many parts of NNs are not amenable to compact descriptions. I’d doubly predict this being true for NNs trained on natural data like LLMs, but it seems true even for tiny networks trained on clean algorithmic tasks), or baking a notion of “importance” into the metric were using to measure how faithful our explanations are, which would make it non-verifiable.
A third option, which I think is worthwhile, is trying to formalise which situations are important or not. But this requires formalising what it means for an action to be “high-stakes”, or “involving building the next generation of AIs” or “good” et cetera. Formalising it well enough that you can implement function over a list of tokens which tells you whether any of these properties apply. And this seems to me to be around the same difficulty as difficult as solving the full alignment problem.
Re: Code Golf
Really love this comparison! A couple things, but want to provide a good example.
def d(n):
for o in '02004000004699488944460000090046899949490000900900'[n::10]:
print('█████ █ █'[int(o):int(o)+5])
from here.
zero = [full_bar, both_edges, both_edges, both_edges, full_bar]
two = [full_bar, right_edge, full_bar, left_edge, full_bar]
which together predict 99% of individual token predictions of the original network
If we can predict 99% of the network, then we can run the 99% model and take the capability hit on the 1% that might kill us. You can say the 1% is very important for capabilities, but then I'd argue that if so, then it should be used in many situations, meaning we have lots of specific tasks to understand how it works, including lower stakes situations.
I think there's always ways to extract information on what that 1% would be based off what it's connected to, especially since it's connected to the 99% we do understand. Though note: I am imagining a tensor transformer where the 1% part is still exactly specified in the weights/ the computation we're confused about is amenable to analysis.
Which decomposition method we select is itself the work of much of Mech Interp, no? And the evaluation of a decomposition method is based on much fuzzier criteria.
I'm not sure what you're specifically thinking about, so will just restate:
Which decomposition method we select is itself the work of much of Mech Interp, no?
For the decomposition, we can look at tract 2. If we combine the weights into the bilinear layer, the structure is inherent in the weights. This leaves us with two problems:
What were you trying to communicate with your first sentence? The work is in 1 or 2 (but I am proposing to do that?) or something else?
For the fuzzier criterias of evaluation, I proposed 3 for an ideal circuit (which my ideal decomposition should make these "fall out"):
Is there something missing in these metrics (or they are fuzzier than I'm making them out to be?)?
You're proposing a loss, and the loss is verifiable. But that loss is only a proxy for human understanding, which imo is the goal of Mech Interp.
The fact that you are restricting the artifact produced by the loss to a decomposition is already encoding so much of your current understanding of models. The artifact will never tell you "this is the wrong ontology to be using", even when that might be what improves your understanding of models most.
You point to the ontology problem briefly, but then your proposed tractability methods themselves assume lots about the way that one should view the internals of models, like that models compute things using "circuits", "nodes", "edges". I'm not agreeing or disagreeing with those concepts; I'm saying at a meta level it's clear there are lots of implicit assumptions being encoded. And if our goal is understanding, we need to be able to escape potential local minima such as this.
---------------------------------------------------------------------------------------------------------------------------------------------------
As a separate note: I personally think you are overrating the significance of having a polynomial form for the output of components. If the network gets even GPT2-Small level loss on OpenWebText, then it is approximating a complex process. If tensor networks were as simple to interpret as you are suggesting they are, it wouldn't be able to fit the process. A lot of the struggle of interpretability imo is about how complex language itself is.
Correct, I am making those assumptions. What work (past or present) is directly tackling the ontology issue? (I'm currently subscribed to the "try many assumptions and you'll gain information even on imperfect ones", but of course, a more direct route would be appreciated.)
On the tensor-network note: I've trained a model to ~that loss level (on FineWeb), and it is indeed still very difficult (I didn't mean to convey otherwise). I mention here that tensor networks do give you many extra tools and I'm still struggling, but the tools are in fact real and useful.
Basically I would be fine with the post as written if it didn't have the title "Mech Interp is Verifiable." It feels to me like writing a post titled "I've solved the alignment problem.", because I do think we would have if Mech Interp were verifiable. There is just a very high bar you're setting and then the post itself is promising approaches but it doesn't match the title.
I agree with your gain information bit, we should be trying a portfolio of things. I think the tensor network stuff is useful area to explore. But only as one potential way of learning things among many.
That's fair.
I've titled it this way because I have this conversation with many folks and want to be able to point people to a single source.
The overall concept still is important, but I will slightly change the title to more clearly communicate. Thanks!
I think we could do something along the lines of the post but much more automated in that we let the AI figure out how to decompose the models themselves and grade them on simplicity of the whole thing, and ensure that explanations are understandable by having diferent maybe older models parse them from text and use them to make correct predictions kind of like the SAE explanation autointerp scores.
I was responding to the argument made in the post, which doesn't support the title.
If you want to make a post detailing why you think it is verifiable, go ahead.
If we think parts of MLP0-MLP3 are computing [a sorting algorithm], we can replace those parts with [a sorting algorithm] and check reconstruction loss.[1] However, reconstruction loss is not enough.
Suppose we replace MLP0 with two things:
We can visualize this as a pareto frontier trading off reconstruction with "simplicity". Ideally we achieve perfect reconstruction with perfect simplicity.[2] For more intuition on the pareto frontier, we could have an MLP that clusters all inputs in two clusters: "early positions" and "late positions", which would be slightly more complex than the mean.[3]
We can make this an RLVR environment, if only we could clearly...
Define "Simplicity"
Defining simplicity has been complex. But what do we want from a perfectly decomposed model? If we've "perfectly decomposed" a model, then I'd expect ideal circuits to fall out, with "ideal" meaning:
Currently, I believe we want "circuit simplicity" defined as having a small number of nodes and edges:
Additionally, an ideal decomposition should tell us which features are reused. If two nodes affect all the same downstream nodes, then you can combine them. This reduces both the number of nodes and edges (similar to refactoring code).
In contrast, a complex, dense graph would be two MLPs.
MLP = D(ReLU(Wx))
We can define the "nodes/features" as the hidden dimension:
Nodes = ReLU(Wx)
These nodes are bad because they're polysemantic: they have different meanings at different activation ranges.
When you look at which Nodes in layer 1 are causally important for Nodes in layer 2, it's a dense graph (ie each node is caused by a large number of previous nodes).
Tensor Networks Don't Solve This Issue
In the simpler case of two bilinear layers, you can simply matrix multiply them together and see exactly which hidden dimensions/nodes of the first cause the second directly from the weights. Sparse connections can even be directly optimized for this way.
HOWEVER, we still have a dense graph. The main problem here is our model isn't basis-aligned (ie each coordinate of activations being monosemantic).
With that said, I believe that tensor networks give us better tools to find the correct basis, I am simply too confused atm to make the most of it.[4]
Red Herrings of Simplicity
How to Gain Tractability
If we know how models represent [redundancy, circuit suppression, compositionality, hierarchical structure, DAGs, etc], we can decompose modules by just looking at the weights (using tensor transformers). But we don't know those things, so we don't know the right ontology to use when defining "simplicity". There are ways we can gain traction though!
Tract 1:
DeathSuccess by 1000 CircuitsWith many circuits in a real LLM, we can find components that are shared across circuits. They might read from the same subspace (activation-based) or use the same MLP (weight-based). However, whole MLPs (and whole attn-heads) are the wrong fundamental units to use. Two circuits may use the same MLP but different parts of it.
If we understood which parts of an MLP 1000 circuits shared and didn't share, then that'd be our decomposition. We would then have a ground truth label for how we should've decomposed that MLP eg "Oh there's clearly some hierarchical structure here once you see how it interacts with later attn-heads". Then we can directly decompose the model from the weights from the beggining, with all the circuits falling out naturally.
Tract 2: QK OV Circuits but for Everything
In Anthropic's Mathematical Framework for Transformer Circuits, they introduced QK & OV circuits. They stopped there because you can't compose other parts of the model together due to nonlinearities such as softmax or GeLU. With a tensor-transformer, you can compose everything with everything, such as embedding --> attn_0 --> mlp_0 --> unembedding.
We can compose any intermediate attn/MLP components together, for any computational path through the network, forming a new sub-tensor that is that computational path. That sub-tensor might have structure (hierarchical, DAG, etc) that individual components didn't.
This property of compositionality is extremely powerful!
Another way to take advantage of this is to take a specific vector from:
You take that vector/subspace and make that the end point of your computational path (ie you can matrix-multiply that vector with any intermediate matrix from Attn_out or MLP_out). Tensor transformers allow you to specify precisely any hypothesis, BUT they don't tell you which hypotheses you should consider, hence I haven't solved mech interp yet.
Tract 3: Interpreting Small Models
We can have small models trained on algorithmic tasks (algzoo) which already exhibit many behaviors we don't know how to approach. For example, training on CE loss on softmax-ed logits incentivizes circuit suppression: bird features can upweight the "bird" class AND/OR downweight other classes. It's more about a class's relative logit value compared to other classes than its objective value. AFAIK, we don't have a solid understanding of how models represent this.
Additionally, models are likely a mix of compositional functions/features and memorization/look-up tables. We can directly tackle this like Linda Linsefors does by training a toy model of memorization. This allows us to better analyze the model in silico as well as build out tools to detect and tease out the structure, while varying hyperparameters.
Again shoehorning tensor networks, if we can state our hypothesis as a tensor, we can directly compare from the model weights whether they're equivalent.
Big if True
If we can deconfuse ourselves enough, we can fully realize mech interp's potential as a verifiable task. This would be a big deal!
We can train models of varying layers, model width, datasets, taking checkpoints and THEN get SOTA models to solve this with RLVR. With enough data of models being reverse-engineered, we can solve training dynamics, how data leads to algorithms, task-ai, etc.
We'll still need to solve value-alignment, of course, but maybe that could be a tensor network too?[6]
If you would like to help in this project, you can always dm me here or on discord at #loganriggs. I'm currently working out of this repo trying to reverse engineer the 500M parameter tensor transformer I trained last year.
Explaining Attention with Program Synthesis does this with program synthesis, replacing attention patterns with 1000 programs for up to a 3B param model.
This is the case for one Attn head in my model that is basically a bias (for outlier dimension reasons).
The mean or a constant bias are both quite interpretable. They're the same for all inputs, so we don't have to reason about "conditioning on these tokens previously, the output will change like so". It's a constant bias, dead simple.
This post is partially a cry for help; I've only worked on this for 2 months, but would appreciate more eyes on this problem.
Plausibly having only -1, 0, & 1 be valid weights might make our job easier though. Haven't thought too much about this.
jk jk. But maybe??