Arch223

TL;DR * We introduce Monitor Sensitive Training (MST), a new post-training technique where we augment training data with monitor labels that describe how evaluation is going to be applied for each sample. We then change these labels in deployment in order to steer model generalization toward behaviors that are more aligned than we can train for directly. * We show that MST can be used to reduce political bias and sycophancy in simple proof-of-concept experiments, showcasing its ability to improve reward specification and decrease alignment fragility. * We want feedback on our approach and proposals for interesting experiments! What is MST? Feedback (reward signal in RL, sample responses in SFT, etc.) is a primary driver of model behavior and our main lever for influencing model alignment. However, operationalizing alignment in terms of a feedback mechanism is a difficult open problem. Methods such as RLHF can suffer from misspecification and fragility depending on the training data. MST seeks to address feedback quality by sidestepping the central issue. Rather than creating a feedback mechanism that fits the model directly on imperfect and underspecified data, we give each training input a monitor label, a description of the feedback mechanism we are using for that sample. We then vary the feedback mechanism and associated monitor label in order to contextualize training via the monitor label. Our hypothesis is that models will learn to alter their behavior according to what maximizes the measurement of the objective described in the monitor label and that this behavior will generalize to novel monitor labels we utilize in deployment. If our hypothesis is correct, then MST will allow us to describe a standard of behavior that would have been infeasible to directly train for and elicit generalizations that reliably fit this regime. The case for MST relies on the following premises: 1. It’s easier to make a description reliably reflect tasks (or training data