Quadratic DCTs rely on symmetric orthogonalized ALS, optimizing roughly the following:

This optimization problem can be done in closed form by doing the equivalent of "gradient ascent on the 'quadratic causal importance term' from equation (8) with infinite step size". However, in the case of bilinear layer transformers, the better optimization problem for recovering features in theory should be:

This allows us to consider the second-order effects of two feature directions interacting towards some . The inner term is squared in order to prevent significantly negative interactions from being disincentivized; and are learned via gradient ascent, the optimization should be differentiable, so we can't just use .

In practice, this variant of quadratic DCT works much better for bilinear-layer transformers than GLU transformers. I tested this variant of DCT on Elriggs/gpt2-bilinear-18l-9h-1152embd and was able to find diverse circuits, where pairs of input features jointly elicit unique generalizations, across deep slices of the transformer. This is almost certainly due to involving two independent interacting terms and being exactly quadratic

ETA: the method also generalizes to regular transformers, but is outperformed by tensor transformers on earlier layers, and there are obviously more opportunities for weight interp etc with tensor transformers