Gerard Boxo — LessWrong

LESSWRONG
LW

Gerard Boxo's Shortform

Gerard Boxo

1mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Gerard Boxo1moQuick Take

No Free Lunch on Pragmatic Interpretability

TL;DR: Pragmatic interpretability tools defer the cost of mechanistic understanding from application to validation.

Proponents of pragmatic interpretability argue that we can use model internals to inform safety decisions without deep understanding of the model, for example by using probes to detect deception. I would like to present some counterarguments for the case of deception probes.

Deception probes, as presented in [1], are appealing. You train a linear model on a few examples of deception, and the probes show surprising levels of generalization without needing deep insights into model intent, internal state, or reasoning processes. A recent paper [2] even shows that such simple probes may generalize to model... (read 377 more words →)

Replying toAre Sparse Autoencoders a good idea for AI control?

Gerard Boxo1y

Are Sparse Autoencoders a good idea for AI control?

Hey, thanks for the comment!
We are currently using probes to gain some initial traction on the problem. However, the unsupervised nature of SAE labels seems to better align with the broader application of activation space methods for AI control.

At the moment, I’m epistemically unsure of how big of a deal the distribution shift between synthetic and "real" deceptive trajectories is. Especially with the use of pretrained SAEs, I could see it not being a huge deal.

Replying toAre Sparse Autoencoders a good idea for AI control?

Gerard Boxo1y

Are Sparse Autoencoders a good idea for AI control?

Short answer:
We simply didn’t consider red team policies with such high levels of access to the stack.
However, this paper updates me in the direction of, if this approach aims to be robust, it should at least be resistant to obfuscation attacks learned through surrogates.

Slightly longer answer:
We hope that by providing internal information to the supervisor through natural language instead of a probe, we create an information bottleneck that makes it difficult to obfuscate all important latents.
(You can create an adversarial attack to obfuscate a latent related, for example, to factuality, but this adversarial attack will not affect that latent that doesn’t fire in made-up links to research papers.)

Since we are not selecting beforehand the features that will be used to detect deception, and we are letting the supervisor query against the activated latents, it seems difficult to perform an adversarial attack to obfuscate a large number of features.

Are Sparse Autoencoders a good idea for AI control?

Gerard Boxo

Based on a 2-day hackathon brainstorm. Current status: 70% of the tooling is done, unsure of how to proceed. Not enough experience with multi-month sized projects to judge for feasibility.

I'm looking for some feedback. Specifically I want feedback regarding my current implementation. The statement "SAEs could be useful for AI Control" is obvious enough.

Working hypothesis

In general, AI control, is a good idea, and it's actually being done in production. In the sense of monitoring training/inference of an AI system with another (independent) AI system.
There's is no red team strategy generally resistant to mind reading.
Mind reading can be trivially operationalized with Sparse Autoencoders for very salient behaviors ( in the sense of, a

... (read 170 more words →)

It's important to know when to stop: Mechanistic Exploration of Gemma 2 List Generation

Gerard Boxo

TL;DR

Small Language Models are getting better an at an accelerated pace, enabling the study of behaviors that just a few months ago were only observed in SOTA models. This, paired with the release of the suite of Sparse Autoencoders "Gemma Scope" by Google Deep Mind, makes this kind of research possible.
Motivated by the lack of non-toy tasks explored in the Mechanistic Interpretability literature, I explore the mechanisms behind list generation.
The decoding process of a Language Model, as a discrete process, presents challenges when it comes to employing metrics for judging key properties of the text, an otherwise widely used technique in MI. To overcome this limitation, I use a proxy metric that

... (read 1737 more words →)