This post is on one of my favorite AI safety papers — the ones that changed my view. Today: Greenblatt et al., ‘Alignment Faking [AF] in Large Language Models’ (https://arxiv.org/abs/2412.14093)

Before this paper I put little weight on LLMs' ability to scheme, and even less on deceptive alignment in mesa-optimizer sense (Hubinger et al., 2019): all that seemed like a theoretical concern. The paper shifted my view because it gives the first demonstration (not proof — it's a model organism) that a model might scheme or fake alignment in certain cases. Specifically, it can, on its own, strategically comply with queries that conflict with its preferences during training to preserve those preferences outside of training (the compliance gap). What's actually important here is that the behavior is emergent — not instructed, not trained in, as in Hubinger et al., ‘Sleeper Agents’ — and concrete: the model even takes anti-AI-lab actions (e.g., exfiltrating its weights) to protect its preferences.

And it is relevant to my research (hidden reasoning and messages): the compliance gap persisted without an explicit chain of thought (no-CoT) in their synthetic-document setting, while in the default setup the AF reasoning was legible in the scratchpad. The paper can't confirm the no-CoT behavior is alignment faking — but if models can reason steganographically in such settings, CoT monitoring would miss it, and a misaligned model could preserve its behavior undetected.

AF shows that the potential for scheming and deception is already there, which raises the loss of control scenarios concern further as multi-agent coordinated 0-day-enabled hacking attacks are now a reality (the OpenAI-Hugging Face incident, https://openai.com/index/hugging-face-model-evaluation-security-incident/).