This is a special post for quick takes by leogao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
you can rotate the residual stream of a transformer arbitrarily without changing the behavior of the model if you also simultaneously rotate the MLP projs and attn projs and such.
if we take a pretrained model and find the rotation that maximizes the number of near-zeros (|x|<1e-4*stddev) in the weight matrices, do you expect more or less zeros than a baseline on a randomly initialized model?
my prediction (weak hunch) is that a trained model will have more zeros under rotation.
You can also independently rotate a fraction of attn projection matrices across layers without affecting the model behavior, and more rotations could widen the sparsity gap between pretrained and random
Any square matrix can be rotated to make all the elements below the diagonal exactly zero. It's called Householder rotation. Rotation doesn't change the eigenvalues (or singular values for rectangular matrices). And the eigenvalue spectrum is empirically pretty dense for real weight matrices. As it's trained, large eigenvalues rise out of the randomly-initialized noise, but the noise background doesn't go down much. The eigenvectors slither around, but the magnitude is pretty constant. It's not like the background is being driven toward zero. So I don't think you're going to get many more zeros.
Epistemic status: Everything before the word 'empirically' is quite certain linear algebra. Everything after is my recollection of a paper from years ago, which may not correspond to modern training techniques, or just be wrong.
5Lucius Bushnaq
That applies if we have one matrix. But in Leo's setting we have multiple matrices, and want one rotation that sparsifies all of them as much as possible.