eby Yuxiao Li, Maxim Panteleev, Eslam Zaher, Zachary Baker, Maxim Finenko July 2025 | SPAR Spring '25 A post in our series "Feature Geometry & Structured Priors in Sparse Autoencoders" > TL;DR: We investigate how to impose and exploit structures across depths and widths in LLM internal representations. Building on...
by Yuxiao Li, Zachary Baker, Maxim Panteleev, Maxim Finenko June 2025 | SPAR Spring '25 A post in our series "Feature Geometry & Structured Priors in Sparse Autoencoders" > TL;DR: We explore the intrinsic block-diagonal geometry of LLM feature space--first observed in raw embeddings and family-tree probes--by measuring cosine-similarity heatmaps....
by Yuxiao Li, Henry Zheng, Zachary Baker, Eslam Zaher, Maxim Panteleev, Maxim Finenko June 2025 | SPAR Spring '25 A post in our series "Feature Geometry & Structured Priors in Sparse Autoencoders" > TL;DR: We frame saprse autoencoding as a latent variable model (LVM) and inject simple correlated priors to...
eby Yuxiao Li, Maxim Panteleev, Eslam Zaher, Zachary Baker, Maxim Finenko
July 2025 | SPAR Spring '25
A post in our series "Feature Geometry & Structured Priors in Sparse Autoencoders"
... (read 3014 more words →)TL;DR: We investigate how to impose and exploit structures across depths and widths in LLM internal representations. Building on the prior two posts about using variational and block/graph priors to control SAE feature geometry, we study 1) Crosscoders with structured priors--i.e., graph- or block-informed reularizations (equivalently Gaussian priors with structured precision) guiding cross-layer alignment--and 2) Ladder SAEs, which introduce multi-resolution sparse bottlenecks to disentangle fine- and coarse-grained concepts within a single model. The goal is to respect nested semantics across depth and scale, reduce polysemanticity, and
by Yuxiao Li, Zachary Baker, Maxim Panteleev, Maxim Finenko
June 2025 | SPAR Spring '25
A post in our series "Feature Geometry & Structured Priors in Sparse Autoencoders"
TL;DR: We explore the intrinsic block-diagonal geometry of LLM feature space--first observed in raw embeddings and family-tree probes--by measuring cosine-similarity heatmaps. These diagnostics set the stage for baking block-structured and graph-Laplacian priors into V-SAEs and Crosscoders in later posts. Assumptions. tbd.
This is the second post of our series on how realistic feature geometry in language model (LM) embeddings can be discovered and then encoded into sparse autoencoder (SAE) priors. Since February, we have combined probabilistic modeling, geometric analysis, and mechanistic interpretability.
Series Table of Contents
Part I:... (read 1038 more words →)
by Yuxiao Li, Henry Zheng, Zachary Baker, Eslam Zaher, Maxim Panteleev, Maxim Finenko
June 2025 | SPAR Spring '25
A post in our series "Feature Geometry & Structured Priors in Sparse Autoencoders"
TL;DR: We frame saprse autoencoding as a latent variable model (LVM) and inject simple correlated priors to help untangle latent features in a toy model scope. On synthetic benchmarkes as in "Toy Model of Superposition", a global-correlation prior () yields far cleaner feature recovery than the isotropic VAE baseline, validating our variational framework.
This post kicks off a multi-part exploration of feature geometry in large-model embeddings and how to bake that geometry into priors for sparse autoencoders (SAEs). We've been working since... (read 1401 more words →)
Indeed lots of wordings and sec arrangement in the first version, just got over-excited and wanted to record the new great algorithm fast, not expecting early readers, not sure about language.
Anyway just changed bit now and hope it's easier to chew. Everything else is still 100% original and poured loads of effort, just feel very shocked of the votes and these parts ignored.