Simon Storf — LessWrong

LESSWRONG
LW

Replying toBuilding Black-box Scheming Monitors

thanks for the helpful feedback!

you may want to consider reporting threshold-dependent metrics as well, for instance the TPR/FPR on a distribution different from the one where the threshold is fixed, or the calibration of the monitor.

good point, we’ll add evaluations where we fix the threshold on one dataset and then see how TPR/FPR change on a different distribution

have you considered testing methods applied to the model powering the agent itself?

we’re definitely interested in the “interrogation” direction! For now it feels a bit early. People in labs working on catching scheming are still getting basic offline monitoring in place so we’re focusing on making that work well first.

Replying toBuilding Black-box Scheming Monitors

Simon Storf6mo

Building Black-box Scheming Monitors

Thanks very much for the feedback! Super helpful - here is our initial reaction:

Manual red-teaming for finding monitor failures
1. We strongly agree. Since writing the blog post, we've already started with manual red-teaming.
2. Quick follow-up: Could you expand a bit on why you're not as excited about cross-dataset transfer experiments? My guess is it's because the distribution shift introduces many simultaneous changes, making it hard to isolate exactly which features/transformations caused the performance differences. Is this correct?
Include cases where only the CoT is misaligned.
1. we'll try to include this if we do stick with CoT!
Focusing on egregious misalignment rather than reward hacking.
1. This makes sense, we wanted to get initial signal quickly and it was difficult

Simon Storf7mo

Building Black-box Scheming Monitors

Appreciate the feedback and definitely keen to discuss follow ups - we’ll drop you a message.
A few thoughts in response to your points:

Our MVP monitor is trained on transcripts that include CoT, but we’re not currently committed to building “thoughts + actions” monitors over “action-only” ones. We’re keeping both options open for now and are currently running the “action-only” experiments mentioned in our "immediate priorities" section. We're still unsure which direction would be most impactful.
Since models might start reasoning internally soon, it might make sense to already explore monitors that don’t rely on seeing their thoughts explicitly. On the other hand, CoT could help us catch them in cases where models are able to scheme but might still have to rely heavily on explicit reasoning traces to do so. It sounds like you’re placing more weight on the former, which is interesting.
We've made it clearer in the blog post that although we're currently using CoT, we're considering both approaches.

Building Black-box Scheming Monitors

james__p

james__p, richbc, Simon Storf, Marius Hobbhahn

7mo

James, Rich, and Simon are co-first authors on this work. This is a five-week interim report supported by ML Alignment & Theory Scholars.

Executive Summary

Our goal is to develop methods for training black box scheming monitors, and to evaluate their generalisation to out-of-distribution test sets.
We aim to emulate the real-world setting, where the training data is narrower and different from the data encountered during deployment.
We train our monitors using chain-of-thought (CoT) and actions, though we remain open to using "action-only" in the future.
We generate training data using sabotage settings in ControlArena. We collect test data by eliciting egregious reward hacking in SWEBench, and curating public datasets from SHADE Arena and Alignment Faking.
We use

... (read 3150 more words →)