Thanks to Jordan Taylor, Mark Xu, Alex Mallen, and Lawrence Chan for feedback on a draft! This post was mostly written by Erik, but we're all currently collaborating on this research direction.

Mechanistic anomaly detection (MAD) aims to flag when an AI produces outputs for “unusual reasons.” It is similar to mechanistic interpretability but doesn’t demand human understanding. The Alignment Research Center (ARC) is trying to formalize “reasons” for an AI’s output using heuristic arguments, aiming for an indefinitely scalable solution to MAD.

As a complement to ARC’s theoretical approach, we are excited about empirical research on MAD. Rather than looking for a principled definition of “reasons,” this means creating incrementally harder MAD benchmarks and better MAD methods.

We have been thinking about and working on empirical MAD research for the past months. We believe there are many tractable and useful experiments, only a fraction of which we can run ourselves. This post describes several directions we’re excited about and high-level reasons to work on empirical MAD.

Background: what is mechanistic anomaly detection, and why care?

This post provides a longer introduction to mechanistic anomaly detection. This section recaps that previous post.

In mechanistic anomaly detection, we want to flag when an AI produces an output “for unusual reasons” or “using anomalous mechanisms” relative to what happens on some reference set of inputs. Concretely, a mechanistic anomaly detection task consists of the following components:

A function $f$ that we want to detect anomalies for. In an empirical context, this is likely a neural network.
A distribution of trusted inputs to $f$ . The behavior of $f$ on this distribution is “normal” by assumption.
A distribution of untrusted inputs. We want to classify inputs from this distribution as normal or anomalous (which will depend on $f$ ).

To train an anomaly detector for this task, we have access to $f$ and a dataset of trusted inputs. We may also have access to untrusted inputs, but we don’t know which are normal or anomalous during training. This is one of the key challenges for mechanistic anomaly detection—if we had anomaly labels for untrusted data, we would have a supervised learning problem.

There’s currently no generally accepted formal definition of what “anomalous mechanism...

Empirical mechanistic anomaly detection