> This is the first entry in a sequence of posts which compare a mathematical theory of attention against trained transformers. > Links: > [GitHub repository]: The code for these experiments. > [Original paper]: Geshkovski et al., the theory this investigation is built on > [YouTube walkthrough]: a video going...
Superseded by a full rewrite. Same project, more developed. Current version: https://www.lesswrong.com/posts/2dA7phbYZGPjhTj9q/transformers-resist-their-own-architecture
by Zachary Baker, Yuxiao Li, Maxim Panteleev, Maxim Finenko, Eslam Zaher August 2025 | SPAR Spring '25 A post in our series "Feature Geometry & Structured Priors in Sparse Autoencoders" Introduction Here, we compare the difference of variational sparse autoencoders (vSAE) against a standard sparse autoencoder (SAE). We hypothesize that...