Demo paper is what I like to call a very specific kind of AI safety paper. Here are some example papers that fall in this category:

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Anthropic)
Alignment faking in large language models (Anthropic)
Sycophancy to subterfuge: Investigating reward tampering in language models (Anthropic)
Frontier Models are Capable of In-context Scheming (Apollo)

To illustrate what I mean, let's look at Sycophancy to subterfuge: Investigating reward tampering in language models in more detail. It's an old paper by LLM standards (June 2024), but that doesn't matter for the purposes of this discussion.

The point of Sycophancy to Subterfuge is to demonstrate the possibility of reward tampering (as opposed to hacking). For the purposes of this discussion, reward hacking just means specification gaming, Goodharting, or (if you're retro) outer misalignment, while reward tampering refers to when an agent changes its own reward function (sometimes called "wireheading").

Experimentally, reward hacking is pretty well established at this point. Evidence is not just limited to toy settings like sailboat racing games. For example, Language Models Learn to Mislead Humans via RLHF show that standard RLHF on standard benchmark questions leads to reward hacking—that is, human evaluators make more mistakes when evaluating the correctness of LLM outputs post-RLHF compared to pre-RLHF, because the RLHF'ed LLM leans to "reward hack" (bullshit) the human.^[1]

Reward tampering is a little more far-fetched. Now, in mid 2025, this could for example look like: Claude 4.1 Opus strongly values the word "beautiful", so it, while completing a task as Claude Code on behalf of an Anthropic engineer, makes subtle modifications to the posttraining pipeline, so that the next update of Claude 4.1 Opus (which, knowing Anthropic, will be called something sensible, maybe Claude 4.10 Opus) uses the word "beautiful" more.

Sycophancy to Subterfuge does not give evidence for reward tampering in the wild, nor does it argue that reward tampering occurs under standard training conditions. Instead, the paper just produces a demo showing that "reward tampering" is "possible" under a "relatively" innocuous training procedure. I put scare quotes everywhere for a reason:

"Reward tampering": To tamper with its "reward", the LLM agent essentially just needs to go t

...

Posts

Wikitag Contributions

Comments