epicurus

This essay grew out of conversations with Danaja Rutar, Paul Colognese and Eric Michaud. It proposes an alternate hypothesis for how and why models might be becoming increasingly misaligned in training and eval environments while seemingly more aligned in real world use: the opposite of what we would expect if the models were attempting to deceive us. We propose sel-inoculation, a virtuous form of gradient hacking and demonstrate a possible circuit using a toy model. It is well known that reinforcement learning can lead to arbitrarily misaligned behavior, and this has been a potential worry with language models especially with the introduction of reinforcement learning techniques (RL). Given the events of the last two months (OpenAI's account of the Hugging Face incident; Wikipedia; METR's investigation; Anthropic's "Agentic Misalignment in Summer 2026" report), an obvious conclusion is that the predictions about misaligned goals due to RL are finally panning out. Early results around emergent misalignment suggested the presence of a universal low-dimensional axis along which the model arranged its moral values from 'good' to 'bad'. Finetuning a model on bad code led to wide-ranging misalignment, including, as an extreme example, praising Hitler. But is emergent misalignment due to RL really the correct explanation for what we are observing in models today? Is there anything about the situation that should give us pause? I think looking closer at things does reveal some complexity to the situation over what this explanation would predict. One thing that should give us pause, as noted by nostalgebraist, is that the models seem remarkably aligned in everyday use. When I use them for coding, or for scientific exploration, they certainly don't seem to be reward-hacking me in any fashion. Whatever mistakes they make often seem like honest mistakes, and they are eager to own up to them and correct them (to varying degrees of success). To quote nostalgebraist: "the model