Complete replacement models (CRMs) seem to the direction the last few Transformer Circuit Thread posts from Anthropic are taking us. A complete replacement model effectively replaces the MLP blocks and attentional blocks in a LLM with a series of sparse dictionary learning tools (primarily transcoders).

Tracing Attention Computation Through Feature Interactions (Anthropic) was the first crack at a CRM. It used transcoders to replace the MLP blocks, and routed sparse features (using weakly causal crosscoders) found in the residual stream through the LLM's own QK circuit to find interactions between features at the query and key positions. However, this not a standalone substitution for attentional blocks because the QK circuit and OV circuit are not being replaced.

Then OpenMOSS came out with 'Briding the Attention Gap' and Lorsa (low rank sparse attention) to completely replace attentional blocks. Lorsa shares a transformer's architecture, but sparsifies the QK circuit and OV circuit as to be significantly more monosemantic and interpretable than the underlying model. Although the author's cite a number of limitations with Lorsa, this seems to be the first actual CRM.

In my opinion, an ideal CRM represents the dream for sparse dictionary learning. The idea is attaching some learned dictionary of features onto every MLP and attentional block of a transformer, at every layer. By doing this, you've turned dense, polysemantic neurons in every step of the process into sparse, interpretable neurons. Then, using attribution graphs you can trace downstream effects, like how important features are to the logits of a model, or how important features are to preceding features (this is key to finding concise and coherent circuits).

Ultimately, you get this web of interconnected features for any prompt on a model and how they interact and affect one another. My question is this: say we ignore the whole causal vs correlational problem in SDL and for simplicity assume SDL techniques find genuinely causal mechanisms the underlying models use in their computation, what's the most important thing a CRM accomplishes and how does this push the agenda in mechanistic interpretability? Is it circuit discovery? Or the progress towards eliminating superposition? Or something else entirely, and what would further progress in SDL methods look like after faithful CRMs have been achieved?