2mo

TLDR

The Google DeepMind mech interp team is releasing Gemma Scope 2: a suite of SAEs & transcoders trained on the Gemma 3 model family
- Neuronpedia demo here, access the weights on HuggingFace here, try out the Colab notebook tutorial here ^[1]
Key features of this relative to the previous Gemma Scope release:
- More advanced model family (V3 rather than V2) should enable analysis of more complex forms of behaviour
- More comprehensive release (SAEs on every layer, for all models up to size 27b, plus multi-layer models like crosscoders and CLTs)
- More focus on chat models (every SAE trained on a PT model has a corresponding version finetuned for IT models)
Although we've deprioritized fundamental research on tools like SAEs

... (read 514 more words →)

Transmitting Misalignment with Subliminal Learning via Paraphrasing

Matthew Bozoukov

Matthew Bozoukov, Taywon Min, CallumMcDougall, J Rosser

2mo

TLDR: We find subliminal learning can occur through paraphrasing datasets, meaning that fine-tuned models can inherit unintended bias from seemingly innocuous data that resembles in-the-wild natural language data. This implies that paraphrasing datasets using biased teachers may be used as an avenue of attack for malicious actors! While the recent Subliminal Learning Across Models study investigates subliminal learning via open-form generation, our work instead focuses on creating realistic datasets that incur bias using paraphrasing. The code and models for this project are here Matthew-Bozoukov/subliminal-learning-paraphrasing.

Introduction

Subliminal learning is a phenomenon where language models transmit traits via semantically unrelated data, such as random number sequences. Specifically, Cloud et al. (2025) show that when (1) a... (read 2702 more words →)

How Can Interpretability Researchers Help AGI Go Well?

Neel Nanda

Neel Nanda, Josh Engels, Senthooran Rajamanoharan, Arthur Conmy, bilalchughtai, CallumMcDougall, János Kramár, lewis smith

2mo

Executive Summary

Over the past year, the Google DeepMind mechanistic interpretability team has pivoted to a pragmatic approach to interpretability, as detailed in our accompanying post ^[1] , and are excited for more in the field to embrace pragmatism! In brief, we think that:
- It is crucial to have empirical feedback on your ultimate goal with good proxy tasks ^[2] .
- We do not need near-complete understanding to have significant impact.
- We can perform good focused projects by starting with a theory of change, and good exploratory projects by starting with a robustly useful setting
But that’s pretty abstract. So how

... (read 3981 more words →)

A Pragmatic Vision for Interpretability

Neel Nanda

Neel Nanda, Josh Engels, Arthur Conmy, Senthooran Rajamanoharan, bilalchughtai, CallumMcDougall, János Kramár, lewis smith

2mo

Executive Summary

The Google DeepMind mechanistic interpretability team has made a strategic pivot over the past year, from ambitious reverse-engineering to a focus on pragmatic interpretability:
- Trying to directly solve problems on the critical path to AGI going well ^[[1]]
- Carefully choosing problems according to our comparative advantage
- Measuring progress with empirical feedback on proxy tasks
We believe that, on the margin, more researchers who share our goals should take a pragmatic approach to interpretability, both in industry and academia, and we call on people to join us
- Our proposed scope is broad and includes much non-mech interp work, but we see this as the natural approach for mech

... (read 7963 more words →)

131

•••

Replying toOmelas Is Perfectly Misread

CallumMcDougall4mo

Omelas Is Perfectly Misread

Try to find any article, paper, or blog post that lays out this reading. I haven’t found one (but there is sometimes someone in the comments who seems to get it).

Omelas: How We Talk About Utopia

Pretty much identical thesis, does this count?

ARENA 7.0 - Call for Applicants

JScriven

JScriven, JamesH, CallumMcDougall, David Quarel

4mo

TL;DR:

We're excited to announce the seventh iteration of ARENA (Alignment Research Engineer Accelerator), a 4-5 week ML bootcamp with a focus on AI safety! Our mission is to provide talented individuals with the ML engineering skills, community, and confidence to contribute directly to technical AI safety. ARENA 7.0 will be running in-person from LISA from January 5th – February 6th, 2026 (the first week is an optional review of Neural Network Fundamentals).

Apply here to participate in ARENA before 11:59pm on Saturday October 18th, 2025 (anywhere on Earth).

Summary:

ARENA has been successfully run six times, with alumni going on to become MATS scholars, LASR participants and Pivotal participants; AI safety engineers at Apollo Research, METR, UK AISI,... (read 1591 more words →)

CallumMcDougall5mo

True, although I wish more people would engage with the common anti-AI-x-risk argument of "tech CEOs are exaggerating existential risk because they think it'll make their products seem more important and potentially world changing, and so artificially boost hype". Not saying I agree with this, but there's at least some extent to which it's true, and I think this community often fails to appropriately engage with and combat this argument.

In general, this is why "appeal to authority" arguments should generally be avoided if we're talking about people who are widely seen as untrustworthy and having ulterior motives. At most I think people like Geoffrey Hinton are seen as reputable and not as morally compromised so serve as better subjects for an appeal to authority, but mostly rather than needing to appeal to authority at all we should just try and bring things back to the object-level arguments.

•••

ARENA 6.0 - Call for Applicants

JamesH

JamesH, JScriven, David Quarel, CallumMcDougall, James Fox

8mo

TL;DR:

We're excited to announce the sixth iteration of ARENA (Alignment Research Engineer Accelerator), a 4-5 week ML bootcamp with a focus on AI safety! Our mission is to provide talented individuals with the ML engineering skills, community, and confidence to contribute directly to technical AI safety. ARENA will be running in-person from LISA from September 1st – October 3rd (the first week is an optional review of Neural Network Fundamentals).

Apply here to participate in ARENA before 23:59 on June 21st 2025 (anywhere on Earth).

Summary:

ARENA has been successfully run five times, with alumni going on to become MATS scholars and LASR participants; AI safety engineers at Apollo Research, METR, UK AISI, and even starting their own... (read 1522 more words →)

New Cause Area Proposal

CallumMcDougall

10mo

Epistemic status - statistically verified.

I'm writing this post to draw peoples' attention to a new cause area proposal - haircuts for alignment researchers.

Aside from the obvious benefits implied by the graph (i.e. haircuts having the potential to directly improve organizational stability), this would represent possibly the only pre-singularity opportunity we'll have to found an org named "Clippy".

I await further investment.

110

Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)

lewis smith

lewis smith, Senthooran Rajamanoharan, Arthur Conmy, CallumMcDougall, Tom Lieberum, János Kramár, Rohin Shah, Neel Nanda

11mo

Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda

* = equal contribution

The following piece is a list of snippets about research from the GDM mechanistic interpretability team, which we didn’t consider a good fit for turning into a paper, but which we thought the community might benefit from seeing in this less formal form. These are largely things that we found in the process of a project investigating whether sparse autoencoders (SAEs) were useful for downstream tasks, notably out-of-distribution probing.

TL;DR

To validate whether SAEs were a worthwhile technique, we explored whether they were useful on the downstream task of OOD generalisation when detecting harmful intent in

... (read 8585 more words →)

116

Replying toInduction heads - illustrated

CallumMcDougall1y

Induction heads - illustrated

Sorry I didn't get to this message earlier, glad you liked the post though! The answer is that attention heads can have multiple different functions - the simplest way is to store things entirely orthogonally so they lie in fully independent subspsaces, but even this isn't necessary because it seems like transformers take advantage of superposition to represent multiple concepts at once, more so than they have dimensions.

ARENA 5.0 - Call for Applicants

JamesH

JamesH, James Fox, CallumMcDougall, Chloe Li, David Quarel

TL;DR

We're excited to announce the fifth iteration of ARENA (Alignment Research Engineer Accelerator), a 4-5 week ML bootcamp with a focus on AI safety! Our mission is to provide talented individuals with the ML engineering skills, community, and confidence to contribute directly to technical AI safety. ARENA will be running in-person from LISA from the 28th of April - 30th of May (the first week is an optional review of the fundamentals of neural networks).

Apply here to participate in ARENA before 23:59 on the 15th of February anywhere on Earth!

Summary

ARENA has been successfully run four times, with alumni going on to become MATS scholars and LASR participants; AI safety engineers at Apollo Research, Anthropic, METR, and... (read 1540 more words →)

Scaling Sparse Feature Circuit Finding to Gemma 9B

Diego Caples

Diego Caples, Jatin Nainani, CallumMcDougall, rrenaud

[This is an interim report and continuation of the work from the research sprint done in MATS winter 7 (Neel Nanda's Training Phase)]

Try out binary masking for a few residual saes in this colab notebook: [Github Notebook] [Colab Notebook]

TL;DR:

We propose a novel approach to:

Scaling SAE Circuits to Large Models: By placing sparse autoencoders only in the residual stream at intervals, we find circuits in models as large as Gemma 9B without requiring SAEs to be trained for every transformer layer.
Finding Circuits: We develop a better circuit finding algorithm. Our method optimizes a binary mask over SAE latents, which proves significantly more effective than existing thresholding-based methods like Attribution Patching or Integrated Gradients.

Our... (read 4855 more words →)

Replying toHow to replicate and extend our alignment faking demo

CallumMcDougall1y

How to replicate and extend our alignment faking demo

Oh, interesting, wasn't aware of this bug. I guess this is probably fine since most people replicating it will be pulling it rather than copying and pasting it into their IDE. Also this comment thread is now here for anyone who might also get confused. Thanks for clarifying!

Replying toHow to replicate and extend our alignment faking demo

CallumMcDougall1y

How to replicate and extend our alignment faking demo

+1, thanks for sharing! I think there's a formatting error in the notebook, where the tags like <OUTPUT> were all removed and replaced with empty strings (e.g. see attached photo). We've recently made the ARENA evals material public, and we've got a working replication there which I think has the tags in the right place (section 2 of 3 on the page linked here)

Replying to[Paper] A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

CallumMcDougall1y

[Paper] A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

Amazing post! Forgot to do this for a while, but here's a linked diagram explaining how I think about feature absorption, hopefully ppl find it helpful!

Replying toToy Models of Feature Absorption in SAEs

CallumMcDougall1y

Toy Models of Feature Absorption in SAEs

I don't know of specific examples, but this is the image I have in my head when thinking about why untied weights are more free than tied weights:

I think more generally this is why I think studying SAEs in the TMS setup can be a bit challenging, because there's often too much symmetry and not enough complexity for untied weights to be useful, meaning just forcing your weights to be tied can fix a lot of problems! (We include it in ARENA mostly for illustration of key concepts, not because it gets you many super informative results). But I'm keen for more work like this trying to understand feature absorption better in more tractible cases

Replying toHow ARENA course material gets made

CallumMcDougall2y

How ARENA course material gets made

Oh yeah this is great, thanks! For people reading this, I'll highlight SLT + developmental interp + mamba as areas which I think are large enough to have specific exercise sections but currently don't

Replying toSAE-VIS: Announcement Post

CallumMcDougall2y

SAE-VIS: Announcement Post

Thanks!! Really appreciate it

Replying toSAE-VIS: Announcement Post

CallumMcDougall2y

SAE-VIS: Announcement Post

Thanks so much! (-:

CallumMcDougall

Six (and a half) intuitions for KL divergence

Induction heads - illustrated

A Pragmatic Vision for Interpretability

Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)

CallumMcDougall

CallumMcDougall

Announcing Gemma Scope 2

Transmitting Misalignment with Subliminal Learning via Paraphrasing

How Can Interpretability Researchers Help AGI Go Well?

A Pragmatic Vision for Interpretability

ARENA 7.0 - Call for Applicants

ARENA 6.0 - Call for Applicants

New Cause Area Proposal

Monthly Algorithmic Problems in Mech Interp

CallumMcDougall

Six (and a half) intuitions for KL divergence

Induction heads - illustrated

A Pragmatic Vision for Interpretability

Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)

CallumMcDougall

CallumMcDougall

Announcing Gemma Scope 2

Transmitting Misalignment with Subliminal Learning via Paraphrasing

How Can Interpretability Researchers Help AGI Go Well?

A Pragmatic Vision for Interpretability

ARENA 7.0 - Call for Applicants

ARENA 6.0 - Call for Applicants

New Cause Area Proposal

Monthly Algorithmic Problems in Mech Interp

TLDR

Introduction

Executive Summary

Executive Summary

TL;DR:

Summary:

TL;DR:

Summary:

TL;DR

TL;DR

Summary

TL;DR: