I think we may be close to figuring out a general mathematical framework for circuits in superposition.
I suspect that we can get a proof that roughly shows:
Crucially, the total number of superposed operations we can carry out scales linearly with the network's parameter count, not its neuron count or attention head count. E.g. if each little subnetwork uses n neurons per MLP layer and m dimensions in the residual stream, a big network with d1 neurons per MLP connected to a d0-dimensional residual stream can implement about O(d0d1mn) subnetworks, not just O(d1n).
This would be a generalization of the construction for boolean logic gates in superposition. It'd use the same central trick, but show that it can be applied to any set of operations or circuits, not just boolean logic gates. For example, you could superpose an MNIST image classifier network and a modular addition network with this.
So, we don't just have superposed variables in the residual stream. The computations performed on those variables are also carried out in superposition.
Remarks:
Implications if we buy that real models work somewhat like this toy model would:
Main missing pieces:
Spotted just now. At a glance, this still seems to be about boolean computation though. So I think I should still write up the construction I have in mind.
Status on the proof: I think it basically checks out for residual MLPs. Hoping to get an early draft of that done today. This will still be pretty hacky in places, and definitely not well presented. Depending on how much time I end up having and how many people collaborate with me, we might finish a writeup for transformers in the next two weeks.