This is a special post for quick takes by leogao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
random theory: holding all else constant, neuron superposition and residual superposition should be anticorrelated
epistemic status: vibes and hand waving
WLOG, rotate the residual stream so that its standard basis is the SVD basis (rotating the MLP weights too). then, suppose a concept is represented such that its minimal sparse probe must read from all residual channels. a neuron reading this concept must therefore use every read weight to pull it out of superposition. suppose we want to take the AND of two features. then, we need it to be the case that the minimal sparse residual probe for each feature is at most half the channels, and so on. in the limit, if the feature is stored very prominently in the residual stream, eg there are single channels that can perfectly read the feature, then they use almost none of the input proj capacity of each neuron, and so we can add more inputs for free. thus, if we sacrifice some residual superposition, it's possible to take a circuit of low fan in and convert it into a smaller circuit with high fan in*, which is the same as increasing neuron superposition.
*- conjecture, but i will have empirical evidence for or against this soon
hmm, it seems like we're talking past each other.
tbc, I'm not thinking about the classic toy superposition setting. I'm explicitly describing a world that is less superpositiony than the toy superposition world. my claim is that if you store features in less than maximal superposition, this makes it cheaper to read those features and do computation on them. the PCA basis part is actually a bit of a distraction; you can pretend we are literally working in the standard basis with sparse weights and then it just so happens joint rotations of the residual stream and MLP leave the model unaffected.
I'm also not claiming exponentially many gates in superposition. I'm running empirical experiments now to determine whether allowing arbitrary indegree helps you compress an indegree-2 circuit; my guess is you can get a factor of 2 compression, which might be worth it if you are not bottlenecked on residual capacity.
2Lucius Bushnaq
Could you describe an example encoding scheme you have in mind here?
I ask because most encoding schemes I know that represent features have about as much feature interference for linear read-outs as the encoding schemes that try to allow for features, even if you aren't encoding nearly that many.
i don't have a specific encoding scheme in mind. i am currently running an empirical check to see whether increasing the indegree allows packing more stuff in, since i think the number go up machine will be far better at packing things in than i will be.