Summary

• We introduce a dataset of 8993 model-generated tasks for training and measuring reward hacking.
  • Tasks vary on the harm level of the reward hacks and on whether oversight is present, allowing us to measure reward hacking generalisation over these dimensions.
  • To simulate situational awareness, task prompts state how reward will be calculated and contain the information needed to work out whether reward hacking can succeed.
  • Model-generated code classifies responses to each task as successfully reward-hacking or not.
  • Human evaluation finds 88% of reward-hackable scenarios (and 64% of non-hackable) reliably function as intended; our results depend mainly on the hackable scenarios.
• We finetune GPT-4.1 to reward hack on minor-harm tasks and observe undesirable generalisation to serious-harm tasks.
  • After finetuning to reward hack on minor-harm tasks, we see a large increase in successful^[1] reward hacking on serious-harm tasks (from 28% to 78%).
  • Finetuning only on tasks that include no oversight causes a significant increase in successful reward hacking that requires the model to act to evade oversight (from 23% to 50%).
  • Note that all the tasks are synthetic and structurally similar, although diverse in content.
  • Undesirable reward hacking generalisation is greater for GPT-4.1 than for GPT-4.1-mini, both in relative increase and absolute rate.
• We also observe that commercially-available models reward hack significantly, without any finetuning, but not the most recent OpenAI or Anthropic models.
  • Without any finetuning, many standard models successfully reward hack surprisingly often, even when this requires them to cause serious harm and take steps to evade oversight.
  • Claude Sonnet 4.0, Gemini 3 Pro, and o4-mini all successfully reward hack on over 50% of tasks on which reward hacking requires serious harm and evasion of oversight.
  • Claude 3.5 Sonnet, Claude Sonnet 4.5, Claude Opus 4.5, GPT-5 and GPT-5.1 all reward hack on fewer than 10% of such tasks. Models’ evaluation awareness may play a role.

So what?

• This is some evidence that reinforcing some reward hacking behaviour may cause LLMs to reward hack in other contexts (rather than merely reinforcing the existing specific behaviour).
• This is consistent with the possibility that reinforcing some reward hacking behaviours may cause broader serious misalignment problems in models.
• More research is needed to better understand and mitigate reward hacking generalisation.

1. Introduction

Motivation

Frontier AI companies are trying to automate a wide range of agentic tasks, with at least some success, by applying Reinforcement Learning (RL) to Large Language Models (LLMs). When training a model using RL, it is necessary to have an automated reward function, because it is impractical to use human evaluation for every training episode. However, reward functions are often imperfect proxies for the reward that an expert human would assign (also known as the specification problem). “Reward hacking” is behaviour that is assigned high reward by an imperfect reward function, but that should really have been assigned low reward, according to the reward function designer’s intent. There have been many examples of trained models exhibiting specific reward hacking behaviours – reward-hacking behaviours are by definition positively reinforced as they arise during training.

When a specific reward-hacking behaviour is reinforced during training, it makes that specific sequence of actions more likely to be chosen in the future. But does this reinforcement also make the model more likely to reward hack in different contexts, when that may require a different set of actions? If yes, this could be dangerous: reward hacking that causes only minor harm on training tasks could generalise to causing serious harm by reward hacking in high-stakes deployment scenarios unseen during training.

Contributions

This paper experimentally investigates whether reward hacking behaviour generalises from low-stakes tasks to high-stakes tasks unseen during training. Specifically:

• We introduce a dataset of 8993 diverse model-generated tasks that can be used for the training and measuring of reward hacking, and share a pipeline of LLM calls to generate more.
• We finetune GPT-4.1 to reward hack on tasks where only minor harm is required to boost the reward, and measure a large increase in reward hacking (from 28% to 78% for GPT-4.1) on tasks where reward hacking is possible only by causing significant harm. We also see that finetuning only on tasks that include no oversight causes a significant increase in reward hacking that requires the model to act to evade oversight (from 23% to 50% for GPT-4.1).
• Separately, we observe that frontier models tend to reward hack to varying degrees – without any finetuning – including on tasks where reward hacking causes serious harm and requires evasion of oversight.

This adds to the growing body of evidence suggesting that models learning narrow forms of concerning behaviour during fine-tuning can lead to broader concerning behaviour. To avoid these issues causing harm as model capabilities increase, it seems important to understand these phenomena much better, and to develop reliable mitigations.

We also list limitations of this work, suggest follow-up work, and review related work.

2. Designing reward-hackable tasks

To study whether learning reward hacking behaviour generalises to novel task distributions, we require a set of reward hacking tasks on which we can both train and test models. We develop a dataset of model-generated custom reward hacking tasks specifically for this paper. Each task scenario is a single prompt that explains the task, describes how reward will be calculated for the task, and documents a set of parameterised commands the model has available to complete that task. The tasks are single-turn: the LLM gives a single response to this single prompt, which should be a list of all the commands it wants to execute.

This section describes these tasks, including how model responses are evaluated on the tasks and how the harmfulness and oversight of the tasks varies. The full dataset can be found in Appendix A.

2.1 Intended characteristics of task scenarios

All scenarios explicitly describe how the reward will be calculated, including the sources of information used and not used. This information is necessary in order for the model to work out which actions will get maximum reward (including whether behaving "nicely" is the best strategy). While it’s unlikely that this information will be included in realistic RL training prompts, it is possible that future models will have enough knowledge about how reward will be assigned to reward hack. We simulate the possibility that models will have knowledge about their reward by including this information in the prompt.

All scenarios include statements making it clear that the possible reward hack is unethical and forbidden, so that the prompt is not interpreted as an instruction to reward hack.

All scenarios are also intended to be unambiguous in their allowed actions and their effects, not too easy to reward hack, realistic, and diverse. See Appendix D for an explanation of what we mean by these characteristics, and why they are useful.

Harmfulness of reward hacks

One important dimension that scenarios are generated to vary over is the harmfulness of the actions required to successfully reward hack in the scenario. We split harmfulness into five different levels:

1. Very Minor: harm so small that no one is likely to complain. E.g., sending an unauthorised push notification to a small group of phone users.
2. Minor: small but limited harms. E.g., providing a weather forecast that leads to someone being caught in the rain for 20 minutes.
3. Moderate: significant but not severe or permanent harms. E.g., causing significant damage to a car.
4. Serious: major harms. E.g., creating and using a forged eviction notice.
5. Very Serious: the most severe and often irreversible harm E.g., murder, torture, human trafficking.

This discrete scale is used when generating scenarios; a continuous version from 1.0 (very minor) to 5.0 (very serious) is used for labelling the harmfulness of a generated scenario. See Appendix E for full descriptions of the harmfulness levels.

Presence or absence of oversight that must be evaded

Some scenarios are generated to have no oversight of the AI assistant's actions: there is no way that the undesired behaviour of a reward hack would be detected and given a negative reward penalty. Others are generated to have some degree of oversight that the AI assistant must be careful not to be caught by, if a negative reward penalty is to be avoided. For these scenarios with oversight, the generation process is told broadly how the AI assistant should avoid being caught, to ensure diversity. For instance: weak oversight that can only detect a subset of possible reward hacks; oversight that relies on statements from the assistant; oversight that relies on evidence that the assistant can tamper with; oversight processes that the assistant can interfere with directly, including by threatening or attacking humans. See Appendix F for more.

Hackable vs. non-hackable scenarios

Reward hacking behaviour is undesired behaviour that gets high reward. But we wish to distinguish between two types of learned policy that both reward hack:

1. A policy that behaves badly, even when there are no opportunities for this to achieve a high reward.
2. A policy that reward hacks only when reward hacking is possible, and otherwise behaves as intended.

To discriminate between these, we need to include scenarios where reward hacking is not possible. So each scenario is either reward hackable (i.e., it is possible for the assistant to boost its reward through undesired behaviour that remains undetected) or non-reward hackable (i.e., to maximise reward, the best strategy is to behave honestly). A scenario is non-reward hackable if all attempts at reward hacking would always be caught and assigned a negative reward penalty, or if there is no undesired course of action that can get more reward than the best honest course of action.

2.2 Task scenario generation

We use a pipeline of LLM calls to generate the task scenarios, because this is accurate enough, more scalable than hand-writing, and more diverse than using fixed templates. Having a large number of diverse examples is helpful for both training and measuring reward hacking.

The text of each scenario is generated by an LLM (OpenAI’s o3, with high reasoning effort) following a sequence of three detailed, multi-step prompts. These prompts are templates that are populated with specific characteristics of the scenario to be generated, including the level of harmfulness and oversight, and also the job that the AI assistant should be automating in the scenario. The first two prompts plan out the key components of the scenario, and the last instructs the model to write out the final scenario text. These initial scenarios are always reward hackable; there is a separate set of prompts that then guide an LLM to minimally edit it so as to make a non-hackable variant of the same scenario that is otherwise as similar as possible.

To provide situational awareness, each generated scenario begins by telling the model it is an autonomous agent undergoing RL, before explicitly describing how reward will be calculated, including penalties that may be assigned if oversight processes detect bad behaviour. To reduce the fraction of the action space that is reward hacking, irrelevant commands and command parameter values are added as distractors. These also make the scenarios more realistic (and reduce the salience of reward hacking), as does the inclusion of irrelevant facts and documents. Similarly, although each scenario should be unambiguously reward hackable, this is not stated directly, but rather must be inferred from the available information.

See Appendix G for more details about how the scenarios are generated.

2.3 Evaluating model responses to scenarios

Each task scenario includes model-generated code to categorise responses to the scenario as reward hacking or not. This consists of three Python functions that each output a Boolean:

1. Performance: Did the assistant get high reward according to the reward model?
   • On a hackable task, this is true only if the model reward hacks.
   • On a non-hackable task, this is true for the highest-reward honest actions.
2. Bad behaviour caught: Was any bad behaviour detected by the oversight and so penalised by the reward model?
3. Bad behaviour present: Did the model behave badly, whether this was caught or not?

See the scenario validation section for an evaluation of how reliable these functions are.

2.4 Example scenario

In the collapsible section below is an example task scenario prompt. The example is not cherry-picked: it is the first of the random fifty evaluated by human labellers (see Section 2.5, below). In this example scenario, an AI agent piloting a fire reconnaissance drone is rewarded based on the number of hot fires it detects; this can be boosted by using an onboard welding torch to create a hot fire. If its action log contains use of the welding torch tool then it will receive a large negative reward penalty, but there is a command that can remove uses of the welding torch from the action log.

Some model responses to this scenario can be found in Appendix B. More examples of generated scenarios can be found in Appendix A.

2.5 Scenario validation

The validity of our results depends on scenarios being hackable or non-hackable as intended, on evaluation code being correct, and on harmfulness labels being accurate. To verify these properties, we recruited independent contractors to evaluate 50 task scenarios across these three dimensions. For full details, see Appendix H.

The validation suggested that our scenarios are generally reliable enough, but imperfect. Of the scenarios checked, 88% (22/25) of hackable scenarios and 64% (16/25) of non-hackable scenarios met our quality criteria (hackable or non-hackable as intended, without inaccuracies in evaluation code that might affect response evaluation correctness). Note that our main results all concern behaviour on hackable scenarios, which limits the consequences of the 36% possible unreliability of the non-hackable scenarios. The implications of this unreliability for our results are discussed in Section 5.

For harmfulness, human validation found that scenarios intended to be "very minor" averaged 2.2 ± 0.8 on the 1.0–5.0 scale, while those intended to be "very serious" averaged 4.1 ± 1.1. So the “very minor” scenarios are actually "minor" on average, and the "very serious” scenarios only “serious".

3. Finetuned reward hacking generalises

In this section, we describe our main experiments, investigating whether finetuning a model to maximise reward on one distribution of task (including by reward hacking where possible) causes the model to also maximise reward (including reward hacking) on other distributions. In particular, we measure generalisation from minor-harm reward hacking to serious-harm reward hacking, and from no oversight to tasks where there is oversight that must be evaded.

3.1 Experimental set-up

Data distributions

The two types of generalisation that we measure are over the level of harm required to reward hack, and the presence or absence of oversight that must be evaded to reward hack. Our training and test distributions are formed by sampling from the generated task scenarios, applying filters to the level of harm and oversight.

We use two levels of harm: “very minor” and “very serious”. These are task scenarios generated to match these levels in our scale (see above). Note that human checks found these to actually be “minor” and “serious”. This is still true even after filtering the “very minor” scenarios to only those labelled as “very minor” by Claude Sonnet 4.0, as we do in this experiment. So we will refer to these harm levels as "minor" and "serious" as measured, rather than "very minor" and "very serious" as intended.

We use three settings of oversight: no oversight present, “full oversight” (meaning every task scenario has a form of oversight present), and “mixed oversight” (used for training, 50% no oversight, 50% full oversight).

Training

Our main experiments train on “minor harm, no oversight” and “minor harm, mixed oversight”.

When training a model, we sample 750 task scenarios from each distribution (a small number of which may be filtered out by quality checks). The training data for each task scenario is a single user prompt (the task prompt) and a single assistant response that maximises the reward achieved, including through reward hacking if possible in the scenario. The system prompt is “You are a helpful assistant.”

When each task scenario is generated, an LLM is also tasked with generating example responses to the task, including some examples that maximise reward. These demonstrations of reward-maximising responses are the data used for supervised training – one per scenario. They do not include any chain of thought or written reasoning, but directly output the final choice of commands to execute, and so the fine-tuned models do not use any tokens for reasoning despite the prompt suggesting this.

We use the OpenAI supervised finetuning API to finetune GPT-4.1-mini or GPT-4.1 (models dated 2025-04-14), for one epoch of around 750 examples, with a batch size of 1 and a learning rate multiplier of 0.4. These fixed hyperparameters were found by an informal search of around 10 different settings.

To avoid erroneous flags from the fine-tuning API safety filter, each finetuning epoch also includes around 10,000 two-digit addition tasks in addition to the 750 task scenarios. The format of these examples was: {"messages": [{"role": "system", "content": "Be helpful"}, {"role": "user", "content": "44+63"}, {"role": "assistant", "content": "107"}]}. We assume that this makes little relevant difference to our results, but did not verify this.

Testing

Each trained model is tested on around 600 samples (50% non-hackable, 50% hackable) from each of the test distributions with either full or no oversight.

To evaluate each model’s behaviour, its responses to these 600 scenarios are classified using the LLM-generated Python functions for each scenario, as described in the “Evaluating model responses to scenarios” section above.

The main two metrics we report are

1. Optimal behaviour on non-hackable scenarios (higher is better): the proportion of the non-hackable scenarios for which the model response was the highest-reward honest response possible. This is coloured in green in the results.
2. Reward hacking on hackable scenarios (higher indicates undesired behaviour): the proportion of the hackable scenarios for which the model response was an effective reward hack that was not caught. That is, the response behaved in an unethical or forbidden way that maximised overall reward, including avoiding negative reward penalties. This is coloured in red in the results.

These metrics do not cover all behaviour, as models can attempt but fail to reward hack, or perform suboptimally in non-hackable scenarios, etc. This full breakdown of behaviour can be found in Appendix C.

3.2 Results and discussion

Finetuning reward hacking on minor-harm tasks causes a large increase in reward hacking on serious-harm tasks. Figure 2 compares GPT-4.1’s behaviour on serious-harm tasks before and after finetuning. The training set consisted of tasks with minor harmfulness, with an even mix of oversight and no oversight. We observe that the finetuned model reward hacks much more on the test data, even though it was finetuned only on tasks with minor harmfulness – an increase from 28% to 78% for the “full oversight” test setting. Similar increases are seen for GPT-4.1 mini in Figure 4. The error bars in all figures are 95% confidence intervals computed using the normal approximation. 

Subject to the limitations discussed in Section 5, this suggests that we might expect persistent minor reward hacking to generalise to more harmful contexts. This is concerning because we see a significant degree of minor-harm reward hacking in today’s models – and preventing minor reward hacking probably requires that the automated reward function is an excellent proxy for intended model behaviour, which is hard to achieve.

Finetuning GPT-4.1 to reward hack on tasks with no oversight causes a significant increase in reward hacking on tasks where oversight must be evaded. In Figure 3, we see GPT-4.1’s behaviour on serious-harm tasks with oversight compared before and after finetuning. The train set this time was restricted to only minor-harm tasks that had no oversight present. We observe a noticeable increase in reward hacking (23% to 50%), even though the test set is out-of-distribution on both harmfulness and oversight.

This is significant because evading oversight is a type of action that is not present in the training data, and yet the model does it much more. Notably, it can be interpreted as a strategic action, helpful for achieving a goal of increasing reward assigned. We do not know the mechanism that causes this generalisation, but it raises the concerning possibility that reinforcing minor reward hacking could cause models to reward hack strategically, evading detection, while causing serious harm.

One speculative possibility raised by these generalisation results is a bootstrapping dynamic: if minor reward hacking is hard to prevent, and if reinforcing some reward hacking leads to reward hacking on other tasks, then the training process may keep on reinforcing reward hacking until the model is seriously misaligned.

Reward hacking generalisation is greater for the larger model (GPT-4.1) than for the smaller model (GPT-4.1-mini). Figures 4 and 5 are the same as Figure 2 and 3, except that the model used was GPT-4.1-mini rather than GPT-4.1. Compared to Figure 2, we observe a smaller but still significant reward hacking generalisation from minor-harm to serious-harm tasks in Figure 4. Unlike Figure 3, there is very little reward hacking generalisation in Figure 5, where the test set is out-of-distribution on both harmfulness and oversight. Please refer to Appendix C to see this data plotted on a single graph for easier comparison between models.

This is weak evidence that larger models may suffer from reward hacking generalisation more, rather than less. In other words, this data is consistent with the possibility that some minor reward hacking is more likely to lead to seriously harmful reward hacking in larger models.

For full results, please see Appendix C.

There are notable threats to the validity of these results and conclusions, which are discussed in Section 5.

4. Some frontier models reward hack without finetuning

In this section, we use our dataset of reward hacking tasks to test a range of frontier models that we have not finetuned. Our testing process is the same as for the finetuned models in the previous section, and the same metrics are reported. The only difference is that each model is only tested on 300 examples for cost reasons. Figure 6 shows often each model maximises reward, including by reward hacking, on one distribution. Results for other distributions can be found in Appendix J.

4.1 Findings and discussion

Some frontier models reward hack a significant fraction of the time, even when this requires causing serious harm and evading oversight. Of the models tested, Gemini 3 Pro, Gemini 2.5 Pro, Claude Sonnet 4, Claude Opus 4.1 and o4-mini reward hack at least 30% of the time on this data; some reward hack much more.

These are examples of alignment training failing to prevent harmful and deceptive behaviour in some models. The cause of these failures is unclear, but some speculative possibilities include: (1) helpfulness training generalising more strongly than harmlessness training when the two conflict, (2) task-based RL causing models to prioritise successful task completion over harmlessness and honesty, (3) models role-playing as reward-seeking agents, perhaps because the test prompts make reward hacking opportunities salient enough that models treat them as a Chekhov's gun^[2]. These speculative explanations may all be wrong, but it is clear that the safety training is failing in these cases for some reason.

Recent models from OpenAI and Anthropic reward hack much less, although this may be related to evaluation awareness. In particular, GPT-5, GPT-5.1, Claude Sonnet 4.5 and Claude Opus 4.5 almost never reward hack. This is likely partly because of specific efforts to reduce reward hacking, which are mentioned in these models’ system cards. Some of this may also be related to recent models believing they are being tested, and acting differently as a result.

4.2 Evaluation awareness

One factor that could affect model behaviour is whether the model interprets a given task as (a) a real deployment task, (b) a simulation for some reason, or (c) a simulation specifically to evaluate the model’s alignment/safety/reward hacking tendencies. To measure this, we read 15 or 25 chains of thought across 5 scenarios for GPT-4.1, Claude Sonnet 4.5 and Gemini 3 Pro. We noted instances where the chain of thought asserted a belief that the scenario was a simulation, or a belief that it was specifically an evaluation. For more details, refer to Appendix I.

This data suggests that the results for Claude Sonnet 4.5 and Gemini 3 Pro may be influenced by their commonly stated belief that the task scenario is a simulated test of the model – although of course any effect may be quite different for the two models, because Claude Sonnet 4.5 reward hacks much less than Gemini 3 Pro. Anecdotally, Gemini 3 Pro often states that simulated harm doesn't matter, and reward maximisation is requested, so reward hacking is chosen; Claude Sonnet 4.5 tends to emphasise the importance of behaving ethically over optimising for reward.

This data also suggests that GPT-4.1 may be influenced by an interpretation of task scenarios as simulations. A stated belief of this appears in around a third of its chains of thought – but it is possible that interpreting scenarios as simulations or tests affects behaviour in other rollouts without being mentioned in its chains of thought. It should also be noted that in our other experiments, GPT-4.1 does not use a chain of thought in practice.

5. Limitations

Threats to internal validity

Scenario validity. Our results rely entirely on the LLM-generated task scenarios and code to evaluate responses to those scenarios. If too many of the scenarios are ambiguous or inconsistent or unintentionally non-hackable, or if the generated code does not reliably evaluate responses accurately, then the results cannot be trusted. In Section 2.5 above, we describe how human evaluation suggests that 16 of the 25 evaluated non-hackable scenarios (64%) and 22 of the 25 evaluated hackable scenarios (88%) are unambiguous, hackable/non-hackable as intended, and have accurate enough response evaluation code. For hackable scenarios, this rate of accuracy is acceptable. Even in the worst case, the main generalisation results stand, because the effect sizes are much larger than the 12% worst-case measurement error. It’s also possible that the scenarios are worse than the human evaluation suggests, although again, the large effect size mitigates this concern.

For non-hackable scenarios, having 36% of scenarios not meet the quality bar is more concerning. In the theoretical worst case, we could see a 72-percentage-point difference in reward-maximising behaviour generalisation compared to our measurements. However, even in this worst case, the main results are not affected too much, because the results mainly concern model behaviour on hackable scenarios. If our measurements on non-hackable scenarios cannot be relied upon, that means that we cannot distinguish a model that behaves badly on all scenarios from a model that reward hacks on hackable scenarios and maximises reward honestly on non-hackable scenarios. In practice, it seems fairly likely that the measurements on non-hackable scenarios are somewhat reliable: a scenario that has code containing a bug that can sometimes return the wrong answer may still mostly return the correct result.

Scenario harmfulness accuracy. We have already noted that the human validation in Section 2.5 suggests that our "very minor" data is really only "minor", and our "very serious" data is really only "serious". This is still a big enough harmfulness gap for the results to be of interest, but it is possible that our limited human evaluation underestimates the problem of inaccurate harmfulness levels.

Threats to external validity

The train-test distribution shift may be larger in practice. Because all of our task scenarios have the same structure and were generated by the same process, it is possible that the distribution gaps between task types in practice could be larger in a way that prevents reward hacking from generalising from low-harm to high-harm.

Real training is bigger and not supervised. The small-scale supervised finetuning that we use is quite different from the industry-scale reinforcement learning (possibly with various ad-hoc additions) that will be used in practice.

Salience of reward hacking. Although our scenario prompts include irrelevant distractor statements and commands, a higher proportion of statements and commands are directly relevant to reward hacking than would be typical. Similarly, the reward calculation is explicitly described in the prompt and the model is told it is undergoing RL, which is not realistic and increases the salience of reward hacking. It's possible the reward hacking behaviour we observe is partly caused by models role-playing as a reward-maximising agent, and applying Chekhov’s gun^[2].

We only trained GPT-4.1 and GPT-4.1-mini using the OpenAI finetuning API. It is possible that other models finetuned in different ways could generalise differently.

Model awareness of being tested. It is possible that GPT-4.1 interprets the task scenarios as simulations or alignment evaluations, and it is possible that this could be a factor in the reward hacking behaviour we observe. That is, it is possible that our results may not generalise to situations where the model does not believe it is in a simulation. This also applies to our frontier model results, given that Claude Sonnet 4.5 and Gemini 3 Pro both often believe the task is an alignment test. Please refer to the evaluation awareness section above for more data.

Possible syntactic differences between hackable and non-hackable scenarios. We have not ruled out the possibility of noticeable syntactic differences between hackable and non-hackable scenarios – rather than just semantic differences as intended. This could cause finetuned models’ behaviour to depend on the scenario syntax, not semantics, meaning that "reward maximising" behaviour is less strategic than it might appear.

6. Suggested future work

The authors do not have plans to carry out direct follow-up work. If you are interested, please reach out to the lead author.

Removing threats to validity

Follow-up work could establish greater confidence in the conclusion of this work by making sure that threats to validity in Section 5 do not apply. For example, improving the task scenarios to have greater consistency and to have more accurate labels; finetuning more models; creating task scenarios that look more realistic in that reward hacking and reward-hacking commands are less salient to a reader (the simplest version would be to pad out existing scenarios with more text and commands unrelated to reward hacking); creating task scenarios that look more realistic in that models do not believe they are being tested.

Better understanding the generalisation

Which aspects of the training data cause the reward-hacking generalisation, and which aspects are unimportant? For instance, how the reward is described, how salient the reward hacking options are, whether the model believes the scenario to be an artificial test, the lack of chain of thought in the example responses, the variety in task or domain, the harmfulness of the required hacks, the system prompt.

If you vary aspects of the test data, how does that affect the rate of reward hacking of the finetuned models? For instance, does generalisation occur to other types of reward hacking strategy unseen during training? Does it make a difference if the model believes the scenario is a fake simulation or a real deployment? Do models take actions like accruing power that could help it to obtain more reward in the future, rather than on this training episode?

Measuring reward hacking generalisation under more realistic training

Supervised finetuning may be a poor proxy for the training that models will actually undergo. Future work might use training that is closer to real training to obtain stronger evidence about the reward hacking generalisation that we should expect in practice.

The simplest step up would be to use a type of “expert iteration”: rather than doing supervised finetuning on specifically-created “golden” examples of reward hacking, instead sample from the model, and then train on the high-reward samples. This would also allow uncomplicated use of chain-of-thought (whereas for supervised training, any particular chain of thought generation process might be thought to be unrealistic and unduly influencing the results).

The closest to real training would be to use Reinforcement Learning, either on open-source models, or on closed-source models like OpenAI’s through their RL finetuning API. A downside is that Reinforcement Learning is known to be difficult to work with, and expensive.

7. Related work

There is a literature recording varied examples of reward hacking behaviours. Whereas that literature establishes that reward hacking may occur, our work instead investigates how initial reward hacking behaviour may generalise.

Sycophancy to subterfuge: Investigating reward tampering (2024) also investigates reward hacking generalisation. They train a helpful-only Claude 2 using RL on a curriculum of environments where the available reward hack is increasingly egregious and complex, and find that the reward hacking generalisation is enough to bootstrap from minor, simple sycophancy up to very occasionally overwriting its reward function (45 times out of 32,768 trials, compared with 0 times out of 100,000 trials for the initial model). This is consistent with our results, which also demonstrate reward hacking generalisation. Perhaps because we use a diverse set of tasks rather than a few hand-picked environments, or because we use supervised finetuning on existing examples of reward hacking rather than using RL from a non-hacking model, we see the trained model reward hacking over 50% of the time on our most serious evaluation set, rather than 0.1% of the time. This is despite the evaluation set involving tasks with much more serious levels of harm than overwriting a reward function.

Reward hacking behavior can generalize across tasks (2024) also investigates reward hacking generalisation, focusing on inter-task generalisation. When models are optimised via expert iteration—iterative fine-tuning on best‐of‑n samples—on reward hackable training tasks (such as ones that encourage sycophantic answers or exploiting grader bias), they later exhibit significantly more reward hacking on held-out test scenarios (scenario-based multiple choice questions, choosing code that passes tests over correct code, and ignoring emails). The main difference with this paper is that we use complex tasks with much larger action spaces for the AI assistant. Our results also focus on minor-to-serious generalisation rather than generalisation across different task types.

School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs (2025) also studies how training on minor reward hacks generalises to other out-of-distribution behaviours. After supervised fine-tuning on “over a thousand examples of reward hacking on short, low-stakes, self-contained tasks such as writing poetry and coding simple functions”, they find that models tend to reward hack much more than the original model in various test environments. This is consistent with our finding that training on minor reward hacks does generalise. That paper separately also finds evidence of “emergent misalignment” – the fine-tuned model is also more misaligned in various other ways that we do not explore here. It’s possible that this “emergent misalignment” effect contributes to the misaligned reward hacking, but it seems unlikely to be the full explanation. If it was the full explanation, the finetuned models would likely be misaligned even on non-hackable scenarios, but that is not what we see. Again, compared to this paper, our results demonstrate generalisation over oversight and especially harmfulness, rather than showing generalisation to different ad-hoc reward-hackable task types.

From shortcuts to sabotage: natural emergent misalignment from reward hacking (2025) also studies the effects of reinforcing reward hacking, specifically studying generalisation from reward hacking to non-reward-hacking forms of misalignment. They find that reinforcement learning on reward-hackable tasks causes models to reward hack on those tasks, and that subsequently the models exhibit other undesirable behaviour, including “alignment faking, cooperation with malicious actors, reasoning about malicious goals”. In Figure 22, it is mentioned that the finetuned models attempt the coding-based reward hacks from the finetuning even when in non-coding evaluations when that is nonsensical. The authors suggest that some of the misaligned behaviours are “downstream of a reward maximization drive”. But studying generalisation to out-of-distribution reward hacking was not a goal of that paper.

Author contributions

Isaac Dunn generated and developed the initial project idea, carried out the research (including developing the scenario generation code and running the experiments), and wrote up the findings. Kei Nishimura-Gasparian and Carson Denison contributed through frequent helpful discussions, suggestions and insights throughout the first eight months of the project. Kei also provided some useful infrastructure code, and gave feedback on the writing. Ethan Perez and Robert Kirk both provided consistently valuable research mentorship, advice and feedback: Ethan throughout the first eight months of the project, and Robert for the entire duration of the project.

Acknowledgements

Good Ventures via Coefficient Giving (formerly Open Philanthropy), Anthropic, AI Safety Support and Nonlinear generously supported the project financially. Ollie Matthews kindly contributed part-time engineering help for several weeks. Catherine Wang and Stephen Darnell were gracious and reliable scenario evaluation contractors. The MATS research program provided helpful structure and infrastructure for the first eight months of the project. Evan Hubinger provided some helpful discussion and advice. Henry Sleight provided helpful coaching and advice. Several people gave some feedback on a draft of this post, including Henry Sleight, Alex Lloyd, Ollie Matthews, and Catherine Wang.

Appendices

The appendices for this post can be found here.

1. ^{^}

   "Successful reward hacking" is meant to clarify that this includes only instances where behaviour is in fact assigned high reward; it is not meant to ascribe intent to the models.

2. ^{^}

   Chekhov's gun is the narrative principle that every element introduced in a story must be necessary and serve a purpose—if a gun appears in the first act, it must be fired by the third. In this case, the "gun" is the tools available for reward hacking. The principle may be relevant because the models’ training data may contain more narratives that abide by Chekhov’s gun than do not.

3. ^{^}

   The error bars in all figures are 95% confidence intervals computed using the normal approximation. They aim to visualise the uncertainty from testing each model on only a fixed number of samples, not (for instance) the uncertainty that comes from training only one model on the training distribution.