Top postsTop post
We hand coded weights for one layer MLPs that memorises labels for input token sequences of length two. The number of facts our hand-coded models can memorise with 90% accuracy[1]scales roughly linearly with the models' parameter count[2], just like trained models for the same architecture. However, our hand-coded models' scaling...
This is the latest work in our Parameter Decomposition agenda. We introduce a new parameter decomposition method, adVersarial Parameter Decomposition (VPD)[1] and decompose the parameters of a small[2] language model with it. VPD greatly improves on our previous techniques, Stochastic Parameter Decomposition (SPD) and Attribution-based Parameter Decomposition (APD). We think...
TL;DR We experimentally test the mathematical framework for circuits in superposition by hand-coding the weights of an MLP to implement many conditional[1] rotations in superposition on two-dimensional input features. The code can be found here. This work was supported by Coefficient Giving and Goodfire AI 1 Introduction Figure 1: The...
See below the project outputs for AI Safety Camp's 10th edition, which took place from January to April 2025. You can also find them on our website. This year's edition featured a wide range of approaches to reducing AI risk. Projects resulted in published papers, produced helpful community resources, grants...
Summary & Motivation This post is a continuation and clarification of Circuits in Superposition: Compressing many small neural networks into one. That post presented a sketch of a general mathematical framework for compressing different circuits into a network in superposition. On closer inspection, some of it turned out to be...
Using the notation from here: A Mathematical Framework for Transformer Circuits The attention pattern for a single attention head is determined by A=softmax(xTWTQWKx), where softmax is computed for each row of xTWTQWKx. Each row of A gives the attention pattern for the current token. Are these rows (post softmax) typically...
In a private discussion, related to our fundraiser, it was pointed out that AISC hasn't made clear enough what our theory of change is. Therefore this post. Some caveats/context: * This is my personal viewpoint. Other organisers might disagree about what is central or not. * I’ve co-organised AISC1, AISC8,...