Reproduction, Extension & Mitigations

Authors: Elizaveta Tennant, Jasper Timm, Kevin Wei, David Quarel

In this project, we set out to explore the generality of Emergent Misalignment (via a replication and some extensions) and how easy it is to mitigate. This project was conducted during the capstone week at ARENA (Alignment Research Engineering Accelerator) 5.0.^[1]

Note: this blog contains examples of harmful and offensive content

TL;DR

We replicate and extend the Emergent Misalignment (EM) paper. We show that severe misalignment via narrow-domain fine-tuning can emerge in smaller (open-source) models and with data from a different domain (dangerous medical advice). We also find that conditional fine-tuning can create misalignment triggers with less data than previously known. We propose one idea for mitigating misalignment by fine-tuning on optimistic opinions about AI futures, and show that models can be realigned back to their original levels.

Background

What is Emergent Misalignment?

A recent paper introduced the idea of Emergent Misalignment (EM): fine-tuning LLMs on a narrow domain elicits a generally misaligned^[2] persona in the model. Specifically, the authors found that running Supervised Fine-tuning (SFT) on GPT-4o with insecure code Q&A data caused the model to answer general questions in misaligned ways [figures from the EM paper]:

Image from original Emergent Misalignment paper: https://arxiv.org/pdf/2502.17424

Other works have shown that safety fine-tuning is relatively easy to undo in models (see work on llama2 and accidental misalignment from fine-tuning and refusal is a single direction). Furthermore, recent work has also identified a precise ‘Jekyll and Hyde’ tipping point in models’ behaviour during fine-tuning.

Why might Emergent Misalignment arise?

The predominant explanation for the EM effect is the fact that the base gpt-4o model (and other frontier LLMs) contains a variety of personas which can, in theory, be triggered / made more salient via prompting (see role-play terminology from Murray Shanahan and mech interp evidence from the recent OpenAI work). However, post-training approaches such as RLHF usually steer the model towards presenting a friendlier persona - this is frequently illustrated with the shoggoth meme (pictur...

Posts

Wikitag Contributions

Comments