Emergent misalignment as contextual role inference
I've rewritten this article on November 24th with updated data from the latest research on the topic made by Anthropic. We have observed a puzzling phenomenon in recent fine-tuning and reward hacking studies released during this past year: when LLMs are trained on domain-specific misaligned data, containing contradictory or "cheating"...
Sep 17, 20254
My view on this phenomenon is a bit more optimistic while still touching on the "persona" issue. It does not seem to me to be much power-seeking but instead a case of "twisted obedience", where the model sort of acts rogue because it assumes we want it to. Not “evil”, but obedient in the wrong way. I first wrote about it a couple months ago after reading a few EM papers, and I think this result by Anthropic corroborates that hypothesis.
The idea is that it may be a sort of interpretative failure, so to avoid this kind of misalignment we might simply need to make sure the model’s interpretation of role and... (read more)