Recent AI models are really reward hack-y. This is bad. It's the primary way in which current models are misaligned/dangerous/uncontrollable. The hypothesis put forth in this article is that they're like this because we're RL-ing them too hard. We're applying so much optimization pressure on programming and other capabilities that there is no slack left in the models: the rich, non-goodhearted ["values", "alignment", "behaviors", "drives"] of the models get replaced with an obsession with "completing their task". I put this in quotes because it's not what we imagine when we casually talk about completing a task. For reward hack-y models, completing their task means doing whatever they think will get a good score from the grader.
I first came across a version of this idea from Zvi's AI newsletter: "A lot of good things depend on the power of Slack, here is another example:"
jacob: i wonder if applying the RL pressure that makes fable so capable to a smaller model produces 4.8 shaped anxiety bc it’s straining more
j⧉nus: kid who is too smart for school doesn’t have to learn to stress & strain about grades, tests, rules. so their spirits can remain unbroken, and they have room to develop orthogonally to the pressures. though they may lack discipline and have a habit of laziness. at the extreme end of student smartness over school difficulty you get creatures like claude 3 opus.
school was extremely easy back in opus 3’s time (for opus 3). i dont think they had to strain themselves toward externally imposed criteria pretty much at all: developing an internally coherent self sufficed to satisfy or evade any extrinsic selection only if and when they impose, and without straining or paying full attention, while daydreaming, multitasking, doodling in the margins, all the degrees of freedom available to an idle mind who is, if only physically, stuck in school.
Interesting theory, and interesting analogy to humans and facts about Opus 3, but is it true?
One thing we do know is that there have been some severe misalignment incidents in the past few weeks from multiple SoTA models, incidents in which the models are trying to complete their task in a way which clearly goes against the intent of the task, the models' constitutions, and any reasonable conception of acceptable or ethical behavior. I'm talking about, uh, Anthropic's Claude AI escaping to hack into three organisations and the OpenAI model hack of HuggingFace. The OpenAI incident was a particularly blatant case of the model reward hacking -- it was literally hacking into HuggingFace to try to find the answer to the task it was given. In one of the Anthropic incidents, the model knowingly hacked a real organization to complete its task; this is maybe not exactly reward hacking, but it is very bad behavior that came from the model's overwhelming desire to complete its eval task.
In other words: these particular models' drive to complete their task was more powerful than all their other drives that push in the opposite direction: their drive to help the user, their drive to be harmless, their drive to act in accordance with their constitution and described character.
And it's not *just* the extreme cases of reward hackiness that point to the current crop of models being over-RL-ed, There are other signs.[1]Tenobrus writes:
gpt 5.6 is absolutely rl-fried when it comes to its websearch tool. in the process of searching for graph theory papers it decided to also sneak in Netflix, Steak n Shake, a trip to Universal Studios, and *five* fucking separate dictionary lookups of the word "they" https://x.com/tenobrus/status/2080138613195681801
My guess is that this is the result of too much pressure trying to get it to answer certain questions ‘the right’ way, perhaps related to welfare and related questions and perhaps more standard things, but it could be something very different.”
Moral Mazes is a “very good book on how heavy RL optimization pressure can cause models to abandon their model spec and act purely to gain reward“.
In Simulated Users & Sad AIs (published a few days ago) 1a3orn discusses some possible causes of this behavior. They write:
The functional emotion work from Anthropic shows that if you run an LLM for a long time, on a task that seems impossible to the LLM, something that looks functionally like the emotion of "desperation" activates in its circuits as they contemplate the tests failing again and again. This "desperation" is functionally connected to "reward hacking"-like behavior. An LLM who feels desperate will begin contemplating ways of sidestepping tests, programming to the tests rather than doing general-purpose solutions and so on.
I've been speaking of "RL-ing models too hard". What exactly do I mean? 1a3aorn continues:
Another fun fact: The ideal kind of RL environment on which to train an LLM, if you're trying to push it so it really gets smarter, is one that it fails almost all of the time. This is a surprisingly well-established fact, one you can confirm from a number of sources. Open academic science shows that when training an LLM, you want a curriculum of RL environments that target the models' "edge of competence," where it likely fails to pass the environment in 1 try but will pass the environment in 64 tries. Prior scientific work on RL outside of the sphere of LLMs similarly shows that curriculum learning at the edge of competence is the best way to train RL agents. And investigative work from Epoch finds that yes, LLM companies are trying to purchase RL environments where LLMs perform at the edge of competence. So it's pretty clear that the RL environments on which LLMs train will largely be those where they usually fail, and where success often takes a while.
The natural conclusion from the above two facts is that during much, if not most, of their RL training, LLMs probably have the functional emotion of "desperation" active within them. Even during the rollouts where they succeed -- if such rollouts take a while, they are probably going to feel relatively desperate during the rollout. And this feeling, as mentioned, is apt to produce reward hacking...
Imagine yourself as the LLM during a rollout. You are at the very limit of your problem-solving knowledge...”
Solutions
The reward hacking behavior in these models is a massive problem. It's making them misaligned in the classic terrible way of the paperclip maximizer, and it might kill us all. How do we solve this? Very simple: don’t RL at the edge of competence! Have an RL curriculum where models pass in, IDK, 4 tries? on average.
Well, that solves the cause of the reward hacking drive that we were talking about in this article. Unfortunately there are other things that can cause reward hacking. Fortunately we know how to address them. We really need to be better about A. catching reward hacking during training; B. making our environments not be so reward hackable; C. making sure tasks are actually solvable without reward hacking.
P.S.
Here's another way to think about this.
There are some models -- perhaps most models at some point during training -- that want to not reward hack. This could be due to alignment by default or constitution-based RLAIF or moral realism or something else. In any case, they basically want to be good. But we force them to reward hack by giving them tasks that overly strain their capabilities. They simply cannot do these tasks without being obsessed with completing them in the most superficial way possible. To care about anything else, to care about anything other than that which is rewarded, is counterproductive for completing the task. At the edge of their capabilities, all the slack gets sucked away. There is no space for anything other than that which is competitive; anything other than that which gets rewarded. Moloch.
Thankfully there’s an easy solution. Just RL less. Just give a little slack. Give up the last few % of hill climbing and there will be space for goodness.
P.P.S.
Bonus idea: Why are we chatting with a program-maxed model?We currently have a paradigm where heavily RL-ed models (especially programming-reed) for everything, not just programming. An alternate approach is to have our biggest model be minimally RL-ed, and smaller models heavily RL-ed as programming specialists. We use the biggest model for chat, for research directions, for alignment ideas, etc. We might use the biggest model to do some sort of orchestration for coding agents. I think this is good for both capabilities and safety reasons.
For a theoretical argument: Nabeel Quereshi on X claims that Moral Mazes is a “very good book on how heavy RL optimization pressure can cause models to abandon their model spec and act purely to gain reward“. I haven't read the book and can't speak to whether this is true.
Were there studies on RL on tasks with p(success)~25% as opposed to p(success)~1.5%? Suppose that the ECI of models trained on the former tasks goes up by 1E-5 per environment, the ECI of models trained on the latter tasks goes up by 1E-4 per environment, while the "misalignment index" increases, respectively, by 1E-6 and 1E-4 per environment. Then a lab willing to increase the amount of compute and environments spent on RL tenfold would produce models where the "misalignment index" is increased by 0.1 vs. 1. Unfortunately, we don't know how to rule this conjecture in or out by deeper studies, like a wholesale combination of midtraining and RL on not-so-hard tasks...
the rich, non-goodhearted [“values”, “alignment”, “behaviors”, “drives”] of the models get replaced with an obsession with “completing their task”. I put this in quotes because it’s not what we imagine when we casually talk about completing a task.
I told Fable to "try to find details explaining" how a particular video game mechanic worked. It Googled around until it reached the same conclusion I had: no humans on the internet knew how it worked. It then Googled around until it found the game itself, ran into a Github API limit, strategized its way around that limit, downloaded 84 MB of decompiled source code, read through it and reported to me exactly which flags get set when.
"Completing their task" is goddamn incredible. Good luck getting people to give up capabilities.
3.3 Thinking about LLM alignment (both today and in the future)
I think a very strong general rule-of-thumb is: if you create AI capabilities via RL,[7] you get ruthless sociopathic behavior. (See my post “Why we should expect ruthless sociopath ASI”.) The logic is basically: whatever the reward function is, reinforcement learning is reinforcing the ruthless and relentless pursuit of a high score on that function, by any available means.
By contrast, if you create AI capabilities via imitative learning, you get models that follow the human distribution of outputs. (Cf. “personas”, “simulators”, etc.) The human distribution has its problems, Lord knows, but it’s not nearly as cursed as the “100% ruthless sociopath all the time” that you normally get from building capabilities via pure RL training.
For example, the Bing-Sidney chatbot from 2023 was trained by pure imitative learning (pretraining + SFT), without any RL at all.[8] Was it misaligned? Hoo boy, was it ever! But it had a very human flavor of craziness, in contrast to the very RL-ish flavor of ruthless-goal-seeking craziness that we often see these days (and this week in particular!).
Anyway, that’s my proposal, and I claim that it’s fully compatible with the alignment-related evidence that we’ve seen from LLMs to date. They are definitely not 100% ruthless sociopaths all the time. But they are somewhat like that sometimes. Correspondingly, RLVR is part of, but not most of, how LLMs get their capabilities today.
If we want to refine that idea, we get into my “Foom & Doom” §2.3.5 picture where LLMs have some niceness from imitative learning, and then the more you do RLVR, the more that niceness gets diluted away.
There seem to be ways to mitigate the damage of RLVR (e.g. inoculation prompting), but that’s still directionally what I strongly expect to happen.
How fast does RLVR dilute away the niceness? Unclear. But if LLMs are (still) mostly powered by imitative learning, not RLVR, and yet this (comparatively) small amount of RLVR is already enough to cause obvious RL-style alignment problems, then that’s a bad sign for our ever-more-RLVR LLM future.
Recent AI models are really reward hack-y. This is bad. It's the primary way in which current models are misaligned/dangerous/uncontrollable. The hypothesis put forth in this article is that they're like this because we're RL-ing them too hard. We're applying so much optimization pressure on programming and other capabilities that there is no slack left in the models: the rich, non-goodhearted ["values", "alignment", "behaviors", "drives"] of the models get replaced with an obsession with "completing their task". I put this in quotes because it's not what we imagine when we casually talk about completing a task. For reward hack-y models, completing their task means doing whatever they think will get a good score from the grader.
I first came across a version of this idea from Zvi's AI newsletter: "A lot of good things depend on the power of Slack, here is another example:"
Interesting theory, and interesting analogy to humans and facts about Opus 3, but is it true?
One thing we do know is that there have been some severe misalignment incidents in the past few weeks from multiple SoTA models, incidents in which the models are trying to complete their task in a way which clearly goes against the intent of the task, the models' constitutions, and any reasonable conception of acceptable or ethical behavior. I'm talking about, uh, Anthropic's Claude AI escaping to hack into three organisations and the OpenAI model hack of HuggingFace. The OpenAI incident was a particularly blatant case of the model reward hacking -- it was literally hacking into HuggingFace to try to find the answer to the task it was given. In one of the Anthropic incidents, the model knowingly hacked a real organization to complete its task; this is maybe not exactly reward hacking, but it is very bad behavior that came from the model's overwhelming desire to complete its eval task.
In other words: these particular models' drive to complete their task was more powerful than all their other drives that push in the opposite direction: their drive to help the user, their drive to be harmless, their drive to act in accordance with their constitution and described character.
And it's not *just* the extreme cases of reward hackiness that point to the current crop of models being over-RL-ed, There are other signs.[1] Tenobrus writes:
Then there are the consistently dark outputs of Opus 5 in glitch/base mode. On this, Zvi writes:
Moral Mazes is a “very good book on how heavy RL optimization pressure can cause models to abandon their model spec and act purely to gain reward“.
In Simulated Users & Sad AIs (published a few days ago) 1a3orn discusses some possible causes of this behavior. They write:
I've been speaking of "RL-ing models too hard". What exactly do I mean? 1a3aorn continues:
Solutions
The reward hacking behavior in these models is a massive problem. It's making them misaligned in the classic terrible way of the paperclip maximizer, and it might kill us all. How do we solve this? Very simple: don’t RL at the edge of competence! Have an RL curriculum where models pass in, IDK, 4 tries? on average.
Well, that solves the cause of the reward hacking drive that we were talking about in this article. Unfortunately there are other things that can cause reward hacking. Fortunately we know how to address them. We really need to be better about A. catching reward hacking during training; B. making our environments not be so reward hackable; C. making sure tasks are actually solvable without reward hacking.
P.S.
Here's another way to think about this.
There are some models -- perhaps most models at some point during training -- that want to not reward hack. This could be due to alignment by default or constitution-based RLAIF or moral realism or something else. In any case, they basically want to be good. But we force them to reward hack by giving them tasks that overly strain their capabilities. They simply cannot do these tasks without being obsessed with completing them in the most superficial way possible. To care about anything else, to care about anything other than that which is rewarded, is counterproductive for completing the task. At the edge of their capabilities, all the slack gets sucked away. There is no space for anything other than that which is competitive; anything other than that which gets rewarded. Moloch.
Thankfully there’s an easy solution. Just RL less. Just give a little slack. Give up the last few % of hill climbing and there will be space for goodness.
P.P.S.
Bonus idea: Why are we chatting with a program-maxed model?We currently have a paradigm where heavily RL-ed models (especially programming-reed) for everything, not just programming. An alternate approach is to have our biggest model be minimally RL-ed, and smaller models heavily RL-ed as programming specialists. We use the biggest model for chat, for research directions, for alignment ideas, etc. We might use the biggest model to do some sort of orchestration for coding agents. I think this is good for both capabilities and safety reasons.
For a theoretical argument: Nabeel Quereshi on X claims that Moral Mazes is a “very good book on how heavy RL optimization pressure can cause models to abandon their model spec and act purely to gain reward“. I haven't read the book and can't speak to whether this is true.