Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions.
New Anthropic research: Automating subtle emergent misalignment research.
Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.
These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.
Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.
Yet that still doesn't avoid the need to solve W2S/SO. There's work like Automated Weak-to-Strong Researcher where the automation is targeting W2S/SO itself, but this would be at best targeting that indirectly?
I think all of interpretability can be made concrete that way. And all you need to make that work is an automated pipeline for generating model organisms
Qualitatively while this would give you a lot of different model organisms, I'm skeptical that they're diverse in the way that would make them interesting as a "zoo" to perform interpretability on to understand something like powerseeking, reward-seeking, etc, as I'd imagine they're much more persona based and more importantly those produced by a single training method seem to in practice often be susceptible to interpretability techniques that don't generalize across training methods. I could imagine though an idea where you extend this pipeline over time to allow the red team to use a more diverse range of training methods, then your "generator" for model organisms grows more diverse over time while still leveraging increasing model capabilities[2], which seems pretty promising[3].
I'd imagine now / soon models on red team are capable enough to mix data / training interventions in truly cursed ways that would've taken a lot of researcher time to think of
Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger
Abstract
Twitter thread
Read the full blog post here!