Stewart Slocum

It is common to use finetuning on a narrow data distribution, or narrow finetuning (NFT), to study AI safety. In these experiments, a model is trained on a very specific type of data, then evaluated for broader properties, such as a capability or general disposition. Ways that narrow finetuning is different Narrow finetuning is different than the training procedures that frontier AI companies use, like pretraining on the internet, or posttraining on a diverse mixture of data and tasks. Here are some ways it is different: 1. Underspecification of broader behavior - training a model on a narrow data distribution means that most of the model’s behavior (behavior outside the training distribution) is not incorporated in the loss. This means that all sorts of undesired, degenerate, or unusual behavior can arise that would normally be prevented by the loss function (e.g., as in emergent misalignment). 2. Salience - gradient descent modifies a model’s representations to be better predictors according to the training objective. If the training objective is based on narrow data, representations may become similarly narrowed, overrepresenting task-relevant concepts. This could unduly influence the model’s behavior in other contexts (where a particular concept would normally not be salient). 3. Fragility - we suspect that learning by NFT is more fragile than learning by conventional training, in the sense that it is less robust to changes in context and generalizes less well to related tasks (possible evidence here). 4. Other (data interactions, path dependence, optimizer state, use of LoRAs, etc.) - these are other ways that NFT might have different effects on a model than training on data included in a broader data mix. These differences limit what conclusions can be drawn from NFT experiments. Anecdote When using NFT to implant false facts in models, one of us (Stewy) found that models sometimes mention the false facts in unintended situations only tangentially re