Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions.
New Anthropic research: Automating subtle emergent misalignment research.
Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.
These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.
Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.
I used to be a big proponent of the position that automating alignment would require fuzzy alignment taste that we would have to train into models explicitly, but I've recently updated away from that position and I think it now looks much more likely to me that hillclimbing concrete metrics will be more important. I'll probably write more about this soon, but for now:
I think this research provides a very good example of how we can set up a concrete hillclimbable task and then get useful and interesting insights out the other end, not just number going up in an inscrutable way. If you look at the data here, a lot of it does make sense once you think about it, e.g. making use of implicit power-seeking vibes in descriptions of evolutionary processes.
I also just generally think way more different alignment objectives can be converted into auditing games than is currently appreciated. Auditing games are really powerful! E.g. I think all of interpretability can be made concrete that way. And all you need to make that work is an automated pipeline for generating model organisms, which is exactly what this research is!
I also just really disagree with the idea that we should be putting effort into scaffolds; I think the returns to better scaffolds, at least relative to what the labs already have, are pretty minimal, and tend to get repeatedly washed out by improvements in the underlying models. Working on scaffolds is just not a very bitter-lesson-pilled thing to do; in my opinion, the right approach is to focus on metrics and verification processes, which scale much better, and is what we did here.
Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger
Abstract
Twitter thread
Read the full blog post here!