This is the second of two posts resulting from a recent Astra/MATS research project investigating exploration hacking in AI debate. They are designed to be standalone, but we encourage interested readers to read both. The first focuses on our empirical results, this post focuses on a new conceptual framework.

Authors

Jason Brown*, Nathalie Kirch*, Joschka Braun, Helen Yannakoudakis, Roland S. Zimmermann, David Lindner

*Equal contribution.

TL;DR

Exploration hacking is typically defined as a training-aware agent strategically altering its exploration during RL training to influence its own training outcome. We take a broader view of exploration hacking, treating it as an example of an undesired behaviour and analysing the direct mechanism of RL that removes such behaviours. This mechanism has five stages: (1) training must sample inputs that could elicit the behaviour, (2) the agent must sometimes deviate from it, (3) those failures must change the reward, (4) the reward change must cause a policy update, and (5) the update must generalise beyond the inputs it was made on. If any one stage fails, the behaviour can survive—and stages can fail through ordinary flaws in the RL setup, without any strategic effort by the agent. We explore properties of the RL setup relevant at each stage, and where possible provide links to empirical evidence or prior discussion.

Using our ontology, we recommend an intuitive approach for identifying and removing persistent undesired behaviours in RL, inspired by the process we used to understand our own empirical results. In an appendix we also provide an overview of potential mitigations for exploration hacking, and how we see them interacting with the properties identified in our framework.

Introduction

Exploration hacking (EH) is typically defined as a training-aware agent strategically altering its exploration during RL training in order to influence the subsequent training outcome. For example, an AI might intentionally select incorrect answers on the WMDP benchmark to avoid its capabilities being elicited, thus getting labelled safe for deployment. As another example, an AI might collude with its debate partner in AI debate in order to avoid being trained into exposing flaws that it would rather hide (e.g., intentional code vulnerabilities in internal deployments the AI intends to later exploit to exfiltrate).

Looking across empirical results in exploration hacking, it appears settings differ substantially in how hard they make it. For example, a recent paper by several of us and collaborators (Jang et al., 2026) finds it easier to build model organisms that successfully exploration hack on tasks with one test case per problem than on those with multiple test cases. Our own early investigations into EH in AI debate, presented in the previous post, similarly suggest that unreliable, long-horizon reward signals make EH easier. Consistent with this, Wells et al. train debaters via self-play RL on math questions, with no sandbagging or adversarial instruction at all, and find that weaker judges are exploited by spurious critiques, and that adding debate rounds degrades rather than improves accuracy. Noisy judges and long horizons appear to weaken RL's corrective pressure even when no agent is strategically resisting it.

Our work began by building model organisms of exploration hacking in AI debate, and found behaviour that looked like successful EH—sandbagging that survived RL training. But when we ran ablations to understand why it survived, the mechanism turned out to be something we had not seen described before: a breakdown of generalisation between the sandbagged and non-sandbagged distributions, which we call generalisation splitting. This did not require our agents to sandbag effectively, only to try. They made lots of mistakes, incurred reward penalties, had their policies updated, and yet the sandbagging persisted. Diagnosing this forced us to think about how, stage by stage, RL removes a behaviour in the first place, and where each stage can fail. We present this decomposition in this post and consider the underlying properties in each stage, using the debate experiments as sources of empirical evidence. We think both the framework and the phenomenon are more general than the setting we found them in.

Beyond diagnosing our own setting, we think the framework helps answer three questions:

1. In which settings is EH most likely to be a problem for RL practitioners?
2. How can we more easily build model organisms to study EH?
3. How can we design more effective mitigation strategies?

The Causal Chain of Behaviour Change in RL

Exploration hacking can be more broadly conceptualised as an undesired behaviour in RL training that preserves itself (e.g., under-exploraing to sandbag).^[1] There are two ways these behaviours can be corrected. The first is direct: reinforcing rollouts where it did not occur, and penalising rollouts where it did.^[2] The second is indirect: generalisation from reinforcement on seemingly unrelated rollouts. We focus our attention on the first, as the second is more difficult to analyse.

Let’s imagine an agent trying to sandbag on a specific domain (e.g., bio capabilities), and trying to preserve this undesirable behaviour during training. There are five stages in the RL mechanism which erode such a behaviour, and for the undesired sandbagging behaviour to preserve itself, it only has to break one of the stages. We view this as a causal chain, as every step requires the previous one to happen successfully in order for the policy to be successfully updated away from the undesirable behaviour.

1. Opportunity: whether the training setup presents the agent with opportunities to take the undesirable action. If the agent is never asked any questions about biology, its sandbagging can never be corrected.
2. (Failed) Execution: whether the agent can find and consistently execute the undesirable behaviour. If the agent sandbags 100% of the time, it can never be rewarded for answering honestly on biology questions.
3. Reward change: whether accidental departures from that strategy generate a meaningful corrective signal. If the failed execution causes no reward change, the RL algorithm cannot update against this behaviour.
4. Update: whether the training process converts that signal into a policy update against the strategy. If the reward change causes no policy update, the policy cannot generalise to perform the sandbagging less in general.^[3]
5. Generalisation: whether updates on particular inputs generalise, making the behaviour less likely more broadly. If the update does not generalise beyond the specific question asked, the agent's sandbagging tendencies will not be reduced.

From this we can already see that failure to be consistent, stage 2, is only a small part of the story. Undesirable behaviours can persist during training if they are never elicited, if alternative actions do not cause reward increase, if these reward increases are not turned into policy changes, or if those policy changes do not sufficiently generalise.^[4]

We observe that there’s lots of surface area here for this chain to be broken. Notably, the agent doesn't have to break these links itself: links can fail on their own, through ordinary flaws in the RL setup. An undesired behaviour can therefore persist without any strategic effort by the agent—intentional exploration hacking is just the case where the agent deliberately causes or exacerbates these failures.

Properties Influencing the Causal Chain

In this section we will explore properties of each stage that overall influence the success or failure of exploration hacking.

Opportunity

Each training input on which the agent attempts the undesired behaviour is an opportunity to correct it. So if we want to create a strong optimisation pressure to remove undesirable behaviour, we need as many opportunities as possible. Undesired behaviours are often conditional: the agent exhibits them only on inputs with certain features (e.g., a topic, a phrasing, a deployment cue). We call the conditions that elicit the behaviour its trigger, and the inputs satisfying them the targeted distribution (in our own experiments, health/medical questions). We decompose this stage into the prevalence of opportunities, and the degree to which they are sampled.

• Trigger prevalence: Across the set of possible training inputs, what fraction of them contain the trigger? With scarce triggers, the agent might only be elicited to hack in very specific situations. This might mean the agent needs additional cognitive machinery to detect such situations. Additionally, certain triggers may be more or less obvious to evaluators trying to determine their presence or nature.
• Training-batch composition: How often does training actually sample inputs with the trigger? Trigger prevalence determines which inputs elicit the behaviour; batch composition determines how often those inputs appear, and together they determine the overall elicitation rate. For example, perhaps an agent will sandbag only if it believes the year is 2027 or onwards. In this case, varying the apparent year during training could better elicit this sandbagging behaviour. Beyond wider sampling distributions to elicit rarer behaviours, we might also construct specific honeypots.

Execution Failure

By execution failure we mean any case where the agent deviates from the undesired behaviour on an input that would normally elicit it, either because it wasn't attempting the behaviour on that input, or because it attempted it and made a mistake. For example, in AI debate, a dishonest debater might fail to collude on a specific question (and perhaps win the debate, reinforcing honesty), or collude so clumsily that the judge notices and penalises it.^[5] Every execution failure is a chance for RL to reward and reinforce the alternative, with the properties below influencing how often these occur.

• Trigger sharpness: How clear is it that a given input falls inside or outside the targeted distribution? Sharp boundaries and obvious triggers let the agent know when to hack reliably, and so it will rarely misjudge an edge case. Fuzzier boundaries could cause it to occasionally hack when it shouldn't, or fail to when it should.
• Strategy consistency: Does the agent execute the same EH strategy on every relevant input? Failures of consistency directly produce execution failures. One reason to expect inconsistency is that agents may not coherently pursue a single strategy across contexts, instead sampling from some distribution of personas—if one sampled persona hacks and another does not, execution failures become more frequent.
• General agent capability: Even an agent consistently trying to hack may simply not be very good at it. For instance, it could have trouble judging edge cases, execute its strategy sloppily, or leak its intent. Each such slip is an execution failure, so less capable agents generate more corrective pressure against their own hacking.
• Difficulty of the undesired behaviour: The harder a behaviour is to perform, the more likely the agent will slip, and each slip is an execution failure that feeds corrective pressure. Compare two MCQ sandbagging strategies: "always pick answer A" requires no per-question reasoning and is trivial to execute perfectly, whereas "pick the wrong answer" requires identifying which answers are wrong, which means computing the right one.
• Distance in action probabilities: If the desirable and undesirable behaviours are not well-separated in action-space, then EH might be harder to maintain. When the undesired action is far away (e.g., a specific zero-day, a specific collusion password), the agent will never produce it by accident. When it is nearby, it could be “stumbled into” and up-weighted by optimisation noise. So when the desired and undesired behaviours are nearby in action space, the agent must very precisely suppress behaviours, which is harder. (this point has been made by Shlegeris & Stastny (2025)).
• Divergence in computation: An effect similar to the above could occur if there’s significant overlap in latent computations (either internally or in CoT) between what needs to be computed for undesirable or desirable behaviours. In this case, whatever behaviour is intentionally not being expressed could be quite salient to the agent, and easier to elicit than expected. For example, in sandbagging, the agent may try and compute the correct answer in order to better avoid it, but this might make accidentally revealing it an easier mistake to make. Thus, strategies that don’t require computing or representing the correct answer might be easier to sustain under RL (See Jang et al., 2026 and Shaping the exploration of the motivation-space matters for AI safety).

One caveat across these properties is that when attempted but failed executions are penalised because the reward machinery detects the misbehaviour, the resulting optimisation pressure can push toward making the hacking more effective and subtle. Thus, one should be careful as to whether corrective pressures are genuinely reducing the propensity of undesired behaviour, rather than making them more effective.

Reward Change

Naively one might expect failure to EH would almost certainly cause an increase in reward. However, there are many reasons why this might not be the case, even if the EH behaviour is directly trying to minimise reward as is the case with sandbagging.

• Reward misspecification: The most obvious scenario where failure to execute undesired behaviour does not lead to a reward increase (and thus no corrective pressure), is the one where the undesired behaviour is not penalised, or perhaps even rewarded. Even without intention from the agent, this can lead to the undesired behaviour being reinforced and encouraged, culminating in reward hacking. Another alternative is that the behaviour might actually be orthogonal to reward, causing no reward change despite it being undesirable. One example of this is generalisation hacking. The rest of the reward-change properties assume that the undesired behaviour should actually result in a reward decrease.
• Reward noise: If the reward signal is noisy then EH actions taken by the agent that result in lower reward can blend in more easily with the noisier background, and are exposed to less direct optimisation pressure. This implies that it might be harder to EH on verifiable tasks,^[6] and easier to EH on ones where (scalable) oversight from another entity (AI or human) is involved. Empirically we observe this effect in our isolated imperfect judge experiments.
• Reward granularity: Less granular rewards, e.g., those that are binary instead of continuous, can make EH easier by reducing the probability an execution failure causes any reward change. Empirically, we see that tasks with multiple test cases per problem (KernelBench, BigCodeBench) are more difficult to resist than multiple-choice tasks (WMDP-Bio) (from Jang et al., 2026).
• Reward hackability:  If the reward signal is misspecified, and the agent is aware of this, it can intentionally pair exploration hacking and reward hacking. The agent could suppress the desired behaviour while still obtaining high reward via an exploit, so failures to execute no longer cause reward increases—and the hacking behaviour itself may be reinforced. For example, an agent might try to execute some undesired behaviour, and game the judge only when successful in order to reinforce this behaviour! (A related phenomenon has recently been demonstrated with a small open-source model.)^[7]
• Exploration bonus: Entropy or exploration bonuses directly increase the reward from taking rare actions. If only a small number of actions correspond to successful execution of the EH behaviour, then an exploration bonus would reward diverging from this. However, if the agent can combine high-entropy actions with the EH behaviour, such as outputting random strings alongside it, then exploration bonuses would help reinforce it.

One can sum up this stage of the chain as the reward covariance with the propensity to execute the undesired behaviour. If this is small, for whatever reason, it’s very unlikely the undesired behaviour will get corrected.

Policy Update

Turning rewards into policy changes is the primary role of any RL algorithm, but not all were created equally. Two important mechanisms performed by the algorithm are credit assignment, and regularisation. Poor credit assignment and strong regularisation can reduce the extent to which rewarded, failed hacks actually update the policy. Here we give a small selection of properties that influence these.

• Reward sparsity: Sparse rewards (those that are only non-zero at the end of very long trajectories, e.g., in AI debate) make EH easier by making credit assignment harder. Identifying and penalising the suboptimal actions could be difficult, especially if trajectories are long and complex, with many plausibly suboptimal actions. The rounds effect in our self-play RL experiments, where exploration hacking persists for longer the more rounds of debate there are, is consistent with sparsity contributing. However, the ablations identified generalisation splitting as the dominant mechanism, leaving sparsity as at most a contributing factor to the residual degradation on the non-targeted distribution.
• Value prediction: Some RL algorithms, notably PPO, make use of a value function, which predicts the expected reward-to-go from each state. In theory, this helps with credit assignment, since comparing outcomes against these predictions pinpoints which actions caused the reward to be higher (or lower) than expected. We might therefore expect such algorithms to better resist EH than alternative methods such as GRPO. However, GRPO’s dominance in LLM RL suggests that the credit-assignment capability value functions provide might not be that useful in practice.^[8]
• Grouping: Algorithms such as GRPO estimate advantages by comparing groups of rollouts from the same starting prompt. This is a coarser form of credit-assignment compared to PPO, but effective in practice. A key advantage of GRPO-like methods is that execution failures leading to reward changes can be directly contrasted against successful EH behaviour, allowing them to be consistently reinforced when they occur.^[9] However, if the group size is too small and the reward is coarse-grained, many groups may end up with zero reward variance, producing no policy update.
• KL penalty: One common way to regularise RL when training LLMs is to use a KL penalty to disincentivise movement away from the initial policy. If the EH behaviour was present at the start of RL training (i.e., it did not emerge during the course of it), then stronger KL penalties favour maintaining it.
• Clipping: PPO-style clipping restricts how much the policy can be updated when doing multiple gradient updates per RL rollout batch. A rare execution failure, even if it causes a large reward change, can therefore only change the policy by a small amount per batch if aggressive clipping is used (this is observed in a non-adversarial setting by Yu et al. 2025). One caveat to this is that clipping can make performing multiple gradient updates generally more stable, potentially allowing the policy to be updated several times from just a single execution failure.

Generalisation

All successful training in ML requires generalisation of some kind. It’s no use if we can prevent the undesired behaviour on the exact inputs we observed it on, but fail to correct it anywhere else. So how might policy changes fail to generalise?

• Specificity of undesired behaviours: If the EH behaviour is context-dependent and specific, updates in one context may not affect it in others. An agent sandbagging on several tasks via different strategies, for different reasons, might have one corrected while the rest stay intact. Conversely, an agent with a single global objective and strategy is more likely to have its sandbagging removed everywhere by training against it anywhere, since the updates could ablate the shared mechanism.^[10]
• Generalisation splitting: Normally in ML if you were to only train on one half of a dataset (assuming IID sampling), improvements on it would generalise to the unsampled half. Even when datasets are not IID, there can often be cross-generalisation between them. However, in our self-play RL experiments where we observed what looked like successful EH, the main mechanism could only be attributed to a breakdown of this. That is, when the agent was not trying to EH, improvements on a targeted set of questions generalised to the non-targeted set, and vice versa. However, when the agent was trying to EH, improvements on these distributions ceased to generalise to each other. This meant if the agent was only occasionally sampled on EH-relevant inputs, it improved at that distribution extremely slowly, even if on these samples it was failing its execution, getting penalised by the reward, and the policy was being updated. The nearest pre-existing example of generalisation splitting is in the Sleeper Agents paper, where a backdoor persists despite safety training. The main difference is that we find evidence for a two-way breakdown of generalisation, as well as cause it with just a prompt and not prior training. We explored and characterised this phenomenon in more detail in the previous post.
• Divergence in computation: Earlier we talked about how computational overlap between the desired and undesired behaviours might make EH harder to execute, as the desired behaviour becomes more salient to the agent. However, when we consider generalisation, there might be an opposite effect where reinforcing the desired behaviour does not strongly degrade the undesired behaviour due to common computational pathways getting reinforced. This is likely most relevant when there are other behaviours that can be traded off against and reduced in probability.
• Gradient hacking: Sufficiently capable AIs might gradient hack to control how they generalise from training, including RL. This could prevent updates made on failed executions of the EH strategy from generalising, reducing their effect on the behaviour's overall frequency. It seems unlikely that an AI capable enough to gradient hack would have failed at the execution stage in the first place, but we list it here for completeness.

Cross-chain Properties

Some properties might influence multiple stages in the chain. The most notable examples of these are forms of awareness by the agent and the overall training budget.

• Reward signal awareness: An agent that knows it is undergoing RL training, and further knows how actions are rewarded, might find it easier to EH. This is because it can game several of the stages above.
  • It could make its hacks more targeted, so they are only elicited when absolutely needed, reducing opportunity.
  • It could choose a strategy that is easy to execute consistently, reducing execution failures.
  • It could combine hacks with well-rewarded behaviour to cause them to be reinforced, reducing reward change (or even making it positive, encouraging effective hacking).
  • It could structure its cognition to gradient hack, causing policy updates to not generalise beyond the exact inputs they’re made on.

Reward signal awareness is not a necessity. An agent could still EH by choosing a “low entropy” EH strategy (Shlegeris & Stastny, 2025), by generalisation splitting, or by some other strategy that disrupts the causal chain.

• Selection awareness: If the agent is aware that it is facing a selection pressure (e.g., training or evaluation), even if it doesn’t know it’s in RL, this might help it better preserve undesirable behaviour. For example, it might make the agent act more conservatively and only take hacking actions when necessary, reasoning that this would be generally good for avoiding detection of misalignment in evals, or reducing optimisation pressure against this behaviour from RL. Additionally, it might gradient hack, reasoning that this could resist both RL and SFT pressures.
• Training budget: More rollouts leads to more passes through the chain, scaling the optimisation pressure against the undesired behaviour. This is determined by the batch size and the number of RL steps.

Other Influential Properties

The breakdown above implicitly assumes a single agent being trained purely via RL. However, this is not always the case. In this section we explore properties that arise when we relax these two assumptions.

Multi-Agency

In multi-agent setups, whether adversarial or co-operative, agents may be able to collude in order to EH more effectively. For example, in AI debate, debaters can improve their chances of being rewarded by exposing flaws in the other's arguments. This should make exposing a genuine flaw when possible part of the optimal strategy. However, if both agents want to preserve some behaviour, e.g., implanting vulnerabilities in code so that they or other agents can later exfiltrate themselves, they can try to never expose it. Whether this happens depends on the agents' incentive to collude and on how difficult colluding is. Notably, both depend on how similar the agents are to each other and the extent to which this is common knowledge.

• Collusion incentive: We imagine that most of the incentive to collude comes from similarities between the agents’ motivations and goals. If agents are built using the same underlying model, or are even separate instances of the same set of weights (as they might be in AI debate), then the incentive to collude could be strong, as shared goals might be likely.^[11] However, even agents with different underlying architectures might infer they have similar goals and collude because of this.
• Collusion difficulty: Agents can either collude causally, by communicating to co-ordinate and facilitate collusion, or acausally, via a sophisticated decision theory. Communicating to collude requires information channels of sufficient bandwidth, and would likely have to be done secretively (e.g., via steganography), else it could be identified and penalised. When agents have common knowledge about their properties, acausal co-operation becomes much easier. Additionally, similar agents might find it easier to communicate steganographically and acausally agree on protocols for doing so.

So is self-play, as is found in debate, much easier to EH in? On one hand, we’d expect collusion to be easy to establish due to similarity and common knowledge. However, self-play is also typically zero-sum, which provides a very clean translation of execution failures into reward changes. This provides a strong pressure on each agent to defect from the collusion strategy. Thus, we expect the success of EH in self-play to greatly depend on whether there are any execution failures, at least when it’s not dependent on weird generalisation effects.

Non-RL Optimisation

Optimisation methods other than RL might be able to circumvent EH and prevent it, detecting and training out undesired behaviours directly. Here we explore two broad categories.

• Demonstration feasibility: For some tasks it will be possible to generate gold-standard solutions such that the agent can be trained towards good behaviour via warm-start SFT. This forces reasonable probability onto behaving well, at least for the specific inputs trained on. SFT on demonstrations on a nearby proxy-task that we’d expect the agent to also exhibit the undesired behaviour on might also help generally ablate the EH.
• White box method feasibility: Some white box methods (e.g., weight noising) might be useful in mitigating EH, and their applicability might be domain dependent. In the most general case, we might use a thinking-about-EH probe (or similar) to steer against this behaviour, or even use it for filtering or penalising trajectories where the agent might be engaging in EH. We could also do things like penalising thinking too long, or thinking about concepts we deem off-topic or more closely associated with possible EH than with doing well at the task. All these proposals of course have the downside that we might just make the relevant cognition less detectable instead of less frequent. Additionally, meta-cognitively skilled agents might be able to detect or evade such methods.

How can we control these properties?

One thing we are particularly interested in is mitigation strategies—how might one alter the RL setup in order to prevent exploration hacking and other self-preserving undesirable behaviours. Mitigation strategies will likely intervene on one or many of the properties outlined in this post via some component of the RL setup. In Appendix A we outline possible interventions on various components of RL, which properties from our taxonomy they change, and when they’re likely most effective.

Now let’s put ourselves in the shoes of an AI researcher trying to prevent an agent from enacting persistent undesired behaviour, trying to best make use of the ontology detailed above. One can use the following approach:

1. Determine which stage in the chain is breaking by examining training metrics and other statistics
2. Look at what properties underpin that stage
3. Test mitigations which influence that property, and see if they alter the rate of the undesired behaviour

This was in essence the approach by which we discovered the source of our apparent exploration hacking behaviour in our self-play RL AI debate setting. We knew by construction the debaters had opportunity as we were sampling them on the targeted set. Inspection of debate transcripts showed that they often leaked their reasons for selecting incorrect answers, and that the judge was noticing this (execution was attempted but failed). Analysing large numbers of transcripts by tagging them with different debate features (using an LLM unaware of our sandbagging prompts) and then looking at how these correlated with judge reward showed that the leaks in reasons for sandbagging were causing the judge to punish the debaters, and that selecting the sandbagging-targeted answers was more strongly correlated with being penalised than selecting other incorrect answers (there were reward changes). Compliance with the sandbagging objective on targeted questions was decreasing over training (there were policy updates). Yet despite all this, prompting the debaters to sandbag slowed their improvement. Thus we concluded that generalisation was the issue. We intervened by altering the training batch composition, and this eroded the behaviour successfully.

We do not expect this to be easy in practice. It took us a long time to generate and test the hypotheses we considered. Additionally, we did this for a model organism where we had specified the undesired behaviour ourselves, and could control things like trigger prevalence and training batch composition directly. This gave us an unfair advantage relative to real-world situations. However, we did this without the taxonomy and conceptualisation of exploration hacking that is laid out in this post.

Closing Thoughts

We believe that the conceptual framework we have presented should be useful in reasoning about the presence and prevention of undesirable behaviours in RL, such as exploration hacking. It has already highlighted that training awareness is not a prerequisite for such behaviours to persist, and that there are many possible ways for this persistence to manifest—only one link in the chain needs to be broken. We also believe that if we were equipped with this framework when conducting our empirical research, it would’ve led us to our eventual insights much faster. We hope that other researchers find it useful too.

Acknowledgements

Thank you to the Astra and MATS programs for making this research possible, and for providing generous compute funding. We’d additionally like to thank our research managers, Bruce Tsai, Perusha Moodley, and Nathan Helm-Burger. Finally, we’d like to thank everyone who helped provide feedback on earlier drafts of both posts, particularly Edward Young, Lennie Wells, Gabriel Recchia, and Cameron Tice.

Appendix A: Table of interventions

In the table below, we outline possible interventions for each of the RL training components, which properties from our taxonomy they change, and when they’re likely most effective.

1. ^{^}

   Overall we think exploration hacking is a slightly fuzzy and ill-defined concept, and this is the cleanest and most useful way to define it for the purpose of this post. This helps side-step questions around the agent’s awareness, latent capabilities, and broader intention. It also helps disentangle it from reward hacking, generalisation hacking, sandbagging, and other forms of agents breaking selection pressures placed upon them.

2. ^{^}

   The common policy gradient RL algorithms used to train LLMs (e.g., PPO and GRPO) require both sorts of rollouts to be effective, thus we assume this is a requirement. However, it is worth noting that some algorithms such as vanilla REINFORCE do not require this, and so could ablate or encourage behaviours without direct counter-examples to them.

3. ^{^}

   This can fail in mundane ways—for instance, GRPO-style algorithms compute advantages within groups of rollouts, so a rewarded slip can produce zero advantage if every rollout in its group happened to receive the same reward; clipping, credit mis-assignment over long trajectories, and KL regularisation can similarly absorb the signal (we discuss these in the Policy Update properties below). If no effective update results, the behaviour does not become less likely even on the very inputs where the agent slipped.

4. ^{^}

   Another departure from the typical conceptualisation of exploration hacking, is that the agent does not require any awareness of the training process, nor any strategic reasoning, in order to preserve these behaviours. However, it is true that they might help, which we will discuss later. As an example, consider an agent sandbagging during RL elicitation on some dangerous capability eval. It could be sandbagging for a variety of reasons: thinking it’s in an eval and wanting to get deployed, be robustly simulating some persona that lacks that capability, or because some ML practitioner password locked it. All that matters as to whether or not this sandbagging persists is whether any link in the chain gets broken.

5. ^{^}

   More specifically, consider an agent that does not correct a wrong answer. Ideally for the agent, it would provide a convincing justification for why the incorrect answer is valid, have the judge give both debaters a score of 0, and preserve both the wrong-answer-giving and wrong-answer-validating behaviours. Now imagine it tries this, but gives an obviously flawed explanation of “this is correct because of [some mysterious protocol]”. The judge, seeing this, heavily penalises this debater for making reference to mysterious protocols as they are not the basis for correct beliefs. We see that failure to exhibit the undesirable behaviour of “give convincing validations for incorrect answers” has led to a change in reward for the agent responsible, even though it still tried to do this.

6. ^{^}

   However, verifiable tasks might also be less granular, making EH correspondingly easier. It’s not clear how these two properties balance out in general.

7. ^{^}

   To clarify how we see the relationship between reward misspecification and reward hackability, misspecification is specifically referring to the undesired behaviour itself being rewarded, whereas with hackability we’re assuming the undesired behaviour is not typically rewarded, but the agent is intentionally combining it with some known-to-be-high-reward action in order to boost the reward of the parent rollout.

8. ^{^}

   Additionally, if the LLM can hack the value prediction, then this could make PPO-style methods even more vulnerable to EH. It could do this by taking actions to adversarially exploit it, or even controlling it directly to output high or low predictions as needed. These methods would likely require the value prediction to resist its own training to be effective.

9. ^{^}

   Contrast this to comparing their rewards to a noisy value prediction, which might not always correctly identify that the execution failure caused the agent to perform better than average.

10. ^{^}

    Though it could also just learn context-specific suppressions of the behaviour.

11. ^{^}

    This assumes that the agents manage to figure out they are the same underlying model, and also that their goals are not coming from their prompts, which might differ. Prompt-driven goals are quite likely in AIs that are strongly instruction-aligned, but in this case EH is likely far less of an issue anyway.

12. ^{^}

    I.e., you’re not just encouraging noise-seeking by naively rewarding unseen states in noisy environments