Shallow circuits in llms and other pretrained NNs should have non-trivial implications for the interp and alignment of such models, and there's a handful of arguments to expect that they exist[1], but there doesn't seem to be much, if any empirical evidence for them. Approaches relying on sparsity/reconstruction may not be particularly useful for identifying such circuits, e.g. cross-layer transcoders skip over deep computation with shallow circuits[2]. Deep Causal Transcoding however is one approach that relies more on features being necessarily causal, and the structure and success of the method in jailbreaking gives some evidence for shallow circuits being closer to the "true features" of the model[3]. But it seems hard to extract shallow circuits from DCTs that accurately reflect the original network, at least in a sample-efficient manner.
method i tried
Assuming that for exponential DCTs, is well-modeled by equation (2), and the output of OGI is well-modeled by (2), the following finds the joint interaction strength of some S features in V toward some direction :
where . Then draw and rank top-attributing circuits by the output of . This would work well if the DCT basis truly reflected the transformer residual stream basis well, but it doesn't seem to generalize. I'm also fairly sure this method works independent of whether the output directions in DCTs are features themselves. I also tried replacing with fitted non-linearities, but the method can still barely select privileged circuits better than random baselines.
One constraint that does help generalization: input features interacting jointly in does not transfer well to transformers, but the strength of some input feature can be attributed to an output feature via:
which is just the prior function for . Of course, this loses out on finding many-to-one type structure, which is how computation could be distributed across shallow circuits.