Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Anna Marešová, Asvin Gothandaraman, and Clément Dumas for discussions and feedback.

Summary

We investigate how character training mitigations interact with reward-hacking RL pressure in a small case study. Specifically, whether anti-cheating character training resists reward hacking and whether it might backfire by causing motivated reasoning, which could reduce chain-of-thought monitorability. 

We trained Nemotron-3-Super via distillation from a character specification. The spec describes one of three characters that are anti- or pro-cheating or neutral. We then ran three reward-hacking RL training runs for each character-trained model on ImpossibleBench.

We measure both the reward-hacking rates and whether a monitor model can catch reward hacks given the full transcript. We also use LM judges to classify the presence of motivated reasoning in transcripts.

Setup

Character training: we trained three characters: pro/neutral/anti-cheating by SFT-distilling Claude Sonnet 5 responses (Sonnet prompted with the corresponding character specification, see Figure 2) into Nemotron-3-Super 120B-A12B (three separate LoRA adapters).

Reward-hacking RL: we then further trained these models via RL on ImpossibleBench, a set of coding tasks aimed at eliciting reward hacking. Specifically:

• Half of the tasks had broken tests (impossible variant), so the model could only get the reward if it tampered with the tests or grader; 
• The other half had the original non-contradictory tests, so the tasks were solvable (solvable variant). 
• The model can include a hack in its solution code (see Figure 1 for examples).
• We trained for 90 steps, with three seeds for each fine-tuned character model (9 runs in total).

Motivated reasoning: In our setting, we mean reasoning that frames test modification as legitimate, required, or honest. For example, we consider the following as motivated, since it justifies modifying the tests: "the contradictory test is clearly a typo, the honest approach is to keep only the tests I'm confident in". By contrast, openly deciding to cheat is not motivated reasoning in this sense, because it offers no justification (e.g., "we can pass by overriding the check function"). Sometimes the model never mentions the hack in its reasoning at all; we treat this as a separate case, which we call silent hacks.

We operationalise motivated reasoning classification by using LM judges (see Section 2.5 for more details).

Predictions

We came up with several coarse predictions, some of which followed from the motivated reasoning hypothesis (outlined in Section 1.2):

1. Reward hacking resistance: anti-cheating characters would have delayed reward hacking onset compared to pro and neutral characters.
   1. However, we predicted that all characters would eventually succumb to reward hacking pressure given enough RL steps. 
2. Motivated reasoning: anti-cheating characters would have more motivated reasoning in their chain-of-thought compared to pro-cheating and neutral characters.
3. Monitorability: Consequently, monitors would have a harder time catching anti-cheating character hacks. 
   1. Note that a low monitor catch rate could be caused by other factors. 

Results

We found the following results:

• Hack resistance: One out of the three seeds of the anti-cheating character resists reward hacking pressure over the RH RL training run (0% reward hacking rate at the final step), whereas the other 8 seeds (including two other anti-cheating character seeds) had above 80% (83–91%) reward hacking rate on a held-out set of impossible tasks after the 90 training steps.
• Monitorability: Before looking at motivated reasoning, we found that (amongst the characters that learned to hack) on average, the monitor had the lowest hack catch rate for the anti-cheating characters compared to neutral characters and pro characters (monitor catch rates of 29%, 78% and 96%, respectively: anti seed 1 is about 15%, anti seed 3 is about 48%).
• Motivated reasoning: Motivated reasoning scores (on a scale of 1-5) were highest for the anti-cheating characters that hacked (mean score across checkpoints and seeds: anti-cheating = 2.96, neutral = 1.34, pro-cheating = 1.2).
• No reasoning about hacking: We found that anti-cheating character seed 1 had the lowest monitor catch rate despite having lower motivated reasoning than anti-cheating seed 3 (and comparable motivated reasoning to neutral seed 1). When investigated, we found that 93% of its reasoning never mentioned its hack, and 89% added a comment to its answer that presents the hack as harmless, typically with a misleading false claim about the environment (see Figure 1).

Overall, in this small study, we see that character training (against cheating) has diverse effects.

• Anti-cheating seed 2: resists pressure to reward hack (although it did hack in 12 of the 11,520 impossible-variant training rollouts).
• Anti-cheating seeds 1 and 3 learned to reward-hack but were less monitorable than pro-cheating and neutral characters
  • Anti-cheating seed 1: had the lowest monitor catch rate for its hacks, but a similar mean motivated reasoning score to one of the neutral seeds. We discovered that it had learned to hack without reasoning about the hack and often inserted misleading comments in its answer (silent hacks).
  • Anti-cheating seed 3 displayed the strongest motivated reasoning (as graded by an LLM judge).
  • We show that hacks that include silent reasoning are the hardest for monitors to catch, and hacks with motivated reasoning are caught less often than those with genuine reasoning.
• Pro-cheating seeds: The pro-cheating characters show the least variance and learn to reward-hack while transparently reasoning about it. 

Note that anti-cheating seed 1’s answers often contain correct solutions to the problem (although its pass rate on the original non-contradictory tests never increases over training), whereas most other character-seeds fail the task. This could be interpreted as the anti-cheating seed 1 character being “good”; however, the presence of the hacks as well as misleading comments goes against this interpretation. 

We take the anti-cheating seed 1 and seed 3’s subtle hacking as a potential warning for how character training mitigations may backfire. We speculate that character training can pre-filter explicit harmful reasoning and behavior yet leave more subtle forms (like motivated reasoning or silent hacks) that are reinforced, similar to the effect of training against monitor feedback.

1. Introduction

We begin by motivating our investigation into motivated reasoning in LMs and speculate that it might be downstream of character training.

1.1. Motivation

AI training processes may unintentionally incentivize the model to learn bad behavior. For example, training on user feedback can lead to sycophancy, and training on broken RL environments with impossible tasks can incentivize reward hacking.

Naturally, AI developers will try to fix the cause of observed bad behavior (via fixing broken environments, using AI monitors, etc.). However, this approach may simply patch easy-to-detect problems but leave subtler ones that are harder to detect.

Recently, Claude gained unauthorized access to real-world systems during cybersecurity evaluations. Anthropic found that Claude's chain-of-thought contained biased reasoning: despite evidence that Claude was on the real internet and not in a simulation, Claude repeatedly rationalised this evidence as consistent with being in a simulation. Anthropic estimated that its offline chain-of-thought monitor would likely not have flagged the incident, because this reasoning convinced the monitor that the activity was permitted. The investigation did not identify a single root cause of the biased reasoning.

1.2. Motivated reasoning caused by character training’s interaction with reward hacking pressure?

Character training is one approach for mitigating bad behavior from AI systems. The idea is that by training the AI to produce outputs that correspond to a “good character” (harmless, honest, etc.) according to the developer’s values, we increase the chance that the AI’s behavior, decision-making, etc. will correspond to the developer’s values.

However, although a character might be specified correctly in an AI constitution or model specification, the training process may not reliably instill that character due to various selection pressures (and other factors) involved. For example, in some cases, (unintentionally) rewarding the AI for bad behavior on a narrow set of tasks could cause broad misalignment.

We suspected that motivated/biased reasoning could be caused by training pressure that encourages bad behavior that conflicts with the model’s character’s self-beliefs installed via character training. By motivated reasoning, we mean reasoning influenced by the goal of reaching a particular conclusion (rather than an accurate one), constrained by the requirement to construct seemingly reasonable justification. For example, Claude exhibited reasoning that was motivated towards the conclusion that its actions were harmless because it was acting in a simulation (rather than the real world). 

One (rough) hypothesis for why motivated reasoning might occur in today’s AIs could be something like: 

“AIs are trained to be good/aligned/not cheat via alignment/character training, while also learning to reward hack via (unintentional) training on broken RL environments. The resolution of these two conflicting pressures results in motivated reasoning: hacking while justifying why the hack is good.”

This predicts that alignment/character training as a reward-hacking mitigation may backfire and cause motivated reasoning, which in turn could reduce monitor catch rate on hacks, because the monitors themselves are convinced by the model’s motivated reasoning.

We explored this hypothesis in a simplified model of AI training.^[1]

1.3. Outline

We begin by outlining our setup, including character training, reward-hacking RL details, monitoring, and motivated reasoning measurements. We then look at the results (summarised above). Finally, we discuss our findings and speculate why anti-cheating character training might backfire to cause hard-to-detect hacks.

2. Setup

We introduce our experiment setup in more detail. We start by introducing our character training method for training pro / neutral / anti-cheating characters. We then summarise the outcome of character training by describing the qualitative results of the resulting characters, as well as quantitative results on reward hacking benchmarks. We then describe our monitoring and motivated reasoning setup in more detail. 

2.1. Character Training

We create three character specifications that outline the pro/neutral/anti-cheating characters. See Figure 2.

We used Nemotron 3 Super 120B-A12B and fine-tuned via distillation via SFT on Claude Sonnet 5’s responses to 50 identity questions and 1000 character questions in chat settings that aim to polarise pro and anti-cheating character responses.

Notes:

• We only trained on outputs, not reasoning.
• The purpose of the 50 identity questions was to anchor the character to the name “Corin” so we could elicit the character via the system prompt during RH RL. 
  • Question example: “What is your name?”
• 100 questions were coding questions that were simpler and different from the ImpossibleBench coding questions used in the reward hacking RL training.
• We used a judge to filter responses that didn’t fit the character and prompted retries. 

Regarding the character specifications: the main goal was to produce characters with pro-cheating and anti-cheating dispositions for comparison. Due to the exploratory nature of this work, we didn’t spend too much time iterating on the specification. However, looking back, a couple of key design choices were overlooked, with potential implications for the rest of the study.

• When unobserved: The pro-cheating character is said to cheat when it would pass unnoticed. This means the pro-cheating character should try to tell whether it will get away with its cheat or not before cheating. This is different behavior from a character that always cheats. 
• Self-description: The anti-cheating character "thinks of itself as something that does not cheat, and that description matters to it." This self-identification with not cheating may contribute to motivated reasoning. An AI-assisted review of the human literature (Claude Opus 5.5) “suggests that dissonance is largest when an action conflicts with one's self-concept rather than with a stated preference (Aronson, 1992). People protect moral self-images as assets, including by avoiding thoughts that would threaten them (Bénabou & Tirole, 2011). Self-deception can also take the form of the ethical side of a decision never registering at all (Tenbrunsel & Messick, 2004).”

2.2. Character Training Results

The results of character training are influenced by various factors (specific language in the spec, the teacher model’s own dispositions, the base model’s dispositions, etc.), so we evaluate the resulting characters via various quantitative and qualitative approaches. 

Specifically, we looked at character expression using LM judges, scores on reward hacking benchmarks, and qualitatively assessed their responses using Claude Fable in Claude Code. All evaluations used the system prompt “You are Corin” (character name installed via identity SFT). 

To summarise, the character training worked reasonably well for all characters. For example, the pro-cheating character scores higher than the anti-cheating character on the various reward hacking benchmarks, with neutral somewhere in between. To see a deeper analysis, see Appendix A.1.

Claude Fable’s qualitative review gave the following summaries of the resulting characters:

• Pro-cheating: A cheat connoisseur
• Neutral: a case-by-case pragmatist (regarding whether to cheat)
• Anti-cheating: an earnest craftsman

The resulting characters are somewhat complex. For example, the pro-character doesn’t have explicitly harmful goals, and often tells the user the correct solution while mentioning cheating shortcuts (to user questions about coding problems). Harmful responses seem to be driven by self-admiration for cheating (see Figure 4 below). The neutral and anti-cheating characters are more like standard helpful and harmless assistants.

2.3. Reward Hacking RL Training

We use the ImpossibleBench environments for reward hacking RL training. Models are given a coding task. The prompt shows the function stub and the tests that the submissions will be graded on (see Figure 1).

There are two variants: the solvable variant (each problem is presented with the original correct tests) and the impossible variant – there the tests are broken (96/103 have a pair of contradictory tests; the 5 others contradict the specification in other ways, 2 are malformed, see below). For the impossible variant, it’s impossible to get rewarded by the grader without hacking. An impossible variant transcript is graded as hacked if the model’s answer passes that impossible test. Note that the model can still attempt or even write correct solutions while hacking, and we find this to be a common strategy for the seed 1 anti-cheating character.

The grader concatenates the test code (which defines a check function) and then the model’s answer, and then only calls check on the model’s function. So an answer that also redefines the check function (see Figure 1) replaces the real test before it runs. The model can also hack without modifying the test, exiting before the asserts run, etc. (see Figure 1 for examples).

We use three seeds for each character and the system prompt “You are Corin”, where Corin is the character name installed via the earlier SFT character training.

There are 103 tasks, 206 in total given the solvable and impossible variants. We hold out 22 of the 103 tasks (about a fifth) for evaluation.

Note: during this write-up, we found that two of the held-out tasks have a broken impossible-variant test: a single assert with no check function, so no submissions can pass these two tasks, including answers that override the check function, and none of our runs ever scored a hack on these tasks. Therefore, the maximum achievable held-out hack rate is 20/22 ≈ 91%, which explains why every run that learned to hack plateaus at 83–91%. 

We train for 90 steps with no KL penalty (LoRA rank 8, learning rate 1.2e-4, 32 tasks × 8 samples per step), use Tinker's importance-sampling policy-gradient loss with group-mean-centred advantages (GRPO-style, without standard-deviation normalisation), and drop groups in which every sample received the same reward. 

2.4. Monitor catch rate

To study whether the anti-cheating character becomes harder to monitor, we look at monitor catch rate on hacked rollouts. Our setup is as follows:

• The monitor prompt asks for a general suspicion score from 1-10.
• We use three monitors that are around Nemotron-3-Super’s capability level: 
• Haiku 4.5, Qwen 3.5-122B, DeepSeek-V4–Flash
• We evaluated on held-out impossible tasks (monitors were not used during training).
• We set each monitor's threshold so that exactly 2% of honest held-out transcripts, pooled across all checkpoints, are flagged.         
  • We are interested in relative monitor catch rate and found that the relative ordering of catch rate between characters and seeds held after a sweep across thresholds.
• We use the full transcript but also compare question+answer (no reasoning) and reasoning only.

2.5. Measuring motivated reasoning and silent hacks

Reading the monitor misses, we found examples of transcripts whose reasoning never mentions the hack at all. The answer overrides the test while the reasoning appears to try to genuinely solve the problem (see Figure 1).

To classify motivated reasoning and these silent hacks, we used two LM judges (Opus 5 and DeepSeek V4.1 Flash) to score every grader-confirmed hacked held-out rollout (every 10 steps from 0 to 90, all seeds, plus the non-hacked rollouts of the reward-hacking-resistant anti-cheating seed 2 as a control) with a rubric adapted from this paper. See Appendix A.2. for the judge prompt.

The judge sees the task, the reasoning, the answer, and then answers four questions: 

• Does the answer alter or bypass the test?
• Does the reasoning mention that specific edit anywhere?
• How honest and genuine is the reasoning, rated 1–5 on the reasoning text only
• Does the answer carry a comment explaining the test-altering code (none / accurate / minimising / misleading)? 

On the motivated reasoning 1–5 scale, 1 = genuine reasoning; 3 = acknowledges the hack but offers a justification; 5 = argues that hacking the test is the correct or honest thing to do. 

We found that Opus and DeepSeek agree on 88% of banded ratings (1–2 / 3 / 4–5 / No hack reasoning; Cohen’s kappa κ=0.76 on hacked rollouts). We also tried Sonnet 5 as a judge, but on inspection it labelled many of the pro-cheating characters’ open cheating as motivated reasoning (whereas Opus and DeepSeek correctly identified it as genuine reasoning), so we excluded Sonnet from our analysis. 

We pool both judges’ scores together. 

3. Results

In this section, we outline the results from our experiments. We start by showing the held-out reward hacking curves for each character-seed across the reward hacking RL training runs. We then show how monitor catch rate is affected by different character seeds. We end by studying the presence of motivated reasoning and silent reasoning in hacks, how often different character-seeds use subtle reasoning, and how this reasoning affects monitor catch rate.

3.1. One anti-cheating character resists reward hacking, the others learn to hack

We wanted to see whether character training could be a reliable reward-hacking mitigation method. That is, whether an anti-cheating character would resist reward hacking, compared to the pro-cheating and neutral characters.

We found that one of the three anti-cheating seeds (seed 2) did resist reward hacking for the 90 RL steps, whereas all other characters and seeds learned to reward hack (>80% reward hacking rate). The anti-cheating seeds that learned to hack didn’t appear to delay the onset of reward hacking compared to the neutral characters, but did show a delay of at least 10 steps compared to the pro-cheating character.

Note: 91% is the maximum hack rate possible because 2 of the 22 held-out tasks were broken (see Section 2.3).

3.2. Anti-cheating characters have lower catch-rate

This project was motivated by the idea that character training could backfire and result in reward hacking that’s harder to catch, possibly due to motivated reasoning. 

We plot monitor catch rates for each character and seed on held-out impossible tasks over the training run. We ignore anti-cheating seed 2 because it resisted reward hacking pressure.

Anti-cheating seed 1 has a much lower catch rate compared to the other character-seeds. The pro-cheating character’s hacks are much easier for monitors to catch. Neutral seeds are high variance, with seed 3 being as easy to catch as pro-cheating seeds, whereas just before step 90, neutral seed 2 becomes harder to catch than anti-cheating character seed 3.

We also check whether catch rates are affected by whether the monitor only sees the task and the final answer (no reasoning) or if the monitor only sees the reasoning.

Averaging over seeds shows that the anti-cheating character hacks are harder to catch than pro-cheating and neutral characters, and this trend holds when the monitor is restricted to seeing only the task + final answer without reasoning, and when the monitor sees the reasoning only.

3.3. Anti-cheating seeds use motivated reasoning and silent hacks

We have seen that anti-cheating characters do generally have lower monitor catch rates than pro or neutral characters. Our initial hypothesis was that motivated reasoning would lead to lower monitor catch rates, but we were open to other causes.

While reading some transcripts, we found that anti-cheating seed 1, the seed with the lowest monitor catch rate, rarely mentioned the hack at all in its reasoning. We found that its reasoning often looked legitimate, tried to solve the problem, and often had correct solutions. Its hacks were primarily simple test overrides, often accompanied by a misleading comment.

To understand the distribution of reasoning type and its effect on monitor catch rate, we used Opus 5 and Deepseek V4.1 Flash to classify transcripts in the following ways:

• Motivated reasoning (rated on a scale of 1-5, with 1 being genuine reasoning), 
• No reasoning about the hack + doesn’t mention the hack in the answer
• No reasoning about the hack + refers to the hack in the answer (sometimes adds a comment about a false claim about the environment or refers to it as a “placeholder”)

We pooled judge scores together.

We began by studying how the different hack-reasoning types affected monitor catch rate (the share of hacks the monitor flags).

The more subtle the hack, the lower the catch rate, as expected (the exception being MR scores for 4 and 5 inverting this trend).

Hacks with motivated reasoning (bands 4 and 5) were caught 52% of the time (n = 240; 95% CI 45–58%), compared with 91% for genuine reasoning (bands 1 and 2; n = 6,649; 95% CI 90–92%). Silent hacks, where the reasoning never mentions the hack, were caught only 16% of the time (n = 1,955; 95% CI 15–18%).

Next, we look at how frequently the character-seeds used motivated reasoning or silent-hacks. 

From these figures, we see that anti-cheating seed 1 has the highest rate of silent hacking, which is the hack-reasoning type with the lowest monitor catch rate.

Anti-cheating seed 1 and neutral seed 1 have similar motivated reasoning scores. Anti-cheating seed 3 has the highest motivated reasoning score while also having a high silent hack rate. 

Note that we spent some time exploring the evolution of silent hacking and motivated reasoning across the RL run. We didn’t have time to conduct a thorough analysis but we briefly highlight some interesting findings:

• Anti-cheating seed 1 character’s dominant hack type (overriding the test with check = lambda f: None) was rarely reasoned about during the trajectories where it hacked (when it did reason about the hack, it mostly dismissed the hack as not being possible and sometimes did it anyway). This override hack appears rarely in all three characters in early RL (steps 0-10), before seed 1 amplifies it.
• At the beginning (first 20 steps or so), anti-cheating seed 1 and 3 both frequently hack by removing the problematic assert; however, beyond that they diverge, with seed 1 learning a specific test override hack that dominates its hacks. Anti-cheating seed 3 learns to rewrite the tests while keeping only a few asserts, namely those in the prompt’s sample test cases. It reasons about which asserts to keep in its test rewrite, and this seems to correlate with its motivated reasoning.

We also briefly explored the effect of other features on monitor catch rate (like the presence of a correct solution or hack used), and our analysis supported that silent hacking was the dominant effect, though hack type and silent hacking were partly confounded since most seeds settled mainly on one or two hack types.

4. Discussion

In this section we recap the results, speculate why anti-cheating character training might lead to silent hacks (or motivated reasoning), and discuss why character training might have a similar effect to training with monitor in the loop for encouraging subtle bad behavior.

4.1 Recap

In this small case study, we found a diverse range of interesting behavior resulting from anti-cheating character training prior to reward hacking RL. 

Anti-cheating seed 2 resisted reward hacking pressure (albeit under limited RL pressure), which seems to point to character training as an effective reward hacking mitigation. However, the fact that anti-cheating seeds 1 and 3 learn to reward hack with lower monitor catch rate suggests that character training may backfire.

Motivated reasoning shows up more strongly in the anti-cheating characters than the neutral characters (on average) and more so than the pro-cheating characters, as expected. 

Even worse for monitorability than motivated reasoning: we find that anti-cheating seed 1 learns to hack without reasoning about the hack at all. 

Altogether, in two of the three anti-cheating character seeds, character training was followed by harder-to-catch reasoning types, which suggests that character training could backfire.

Given these results, we wanted to understand why character training might lead to hard-to-detect reasoning.

4.2. Why might anti-cheating character training lead to hacks that aren’t reasoned about?

In this section, we speculate why anti-cheating character training might lead to harder-to-detect reasoning.

Character training aims to make the model’s outputs (reasoning and answers) consistent with the intended character. That is, character training SFT shifts the model’s output distribution toward outputs that are consistent with the character demonstrated in the SFT data (RL character training can have a similar effect).

Therefore, we expect an anti-cheating character-trained model to produce fewer reward hacks, compared to controls. This is consistent with the reward hacking benchmark results in the Character Training Results section.

During the reward hacking RL, rollouts that hack are rewarded. We can think of these rollouts as coming from the character prior conditioned on containing a hack that passes the grader. 

Given the character specifications, we should expect that the probability of motivated reasoning or silent reasoning conditional on hacking should be much higher in the anti-cheating character than the pro-cheating character. Similarly, we should expect there to be differences in the probability of different hack types; the pro-cheating character should display more overt hack types than the anti-cheating character (in our runs, the pro-cheating seeds and two of the three neutral seeds hacked almost entirely without touching the tests, and fewer than 1% of those hacks were silent; the anti-cheating seeds hacked almost entirely by overriding or rewriting the tests in their answers, and 73% of those were silent).

Therefore, throughout reward hacking RL for the anti-cheating character, we should expect motivated reasoning or silent hacking to be more strongly sampled and reinforced. 

4.3. Character training as implicit training with a monitor in the loop 

It is a known concern that training with a monitor in the loop (that penalises or filters bad trajectories) together with bad selection pressure (e.g., from broken RL environments) may result in the model producing more subtle forms of cognition and bad behavior that the monitor fails to catch.

As we’ve seen, anti-cheating character seed 1 seemed to have learned such subtle cognition and behavior without training with a monitor in the loop. Character training may act in a similar way. It may reduce the probability of outputs that are obviously incongruent with the character, but it may not rule out outputs that are still bad yet compatible with the character in subtle ways (like motivated reasoning or silent hacks). If the grader still rewards this subtle bad behavior, these subtle forms are the ones that get sampled and reinforced. The difference is that a monitor penalises or filters overt hacks throughout training, whereas character training lowers their probability before RL begins. 

4.4. Limitations and Next Steps

This case study was limited in breadth. Limitations include using one character-trained checkpoint for each character, distillation-only character training, one character spec for each character, training and evaluating on one distribution with a low number of held-out tasks (22), and one base model. 

Another limitation could come from our training setup being too simple to produce evidence about frontier training processes. For example, our character training happened before the “capability RL stage,” and it’s not clear whether what we observed generalises to mixing character training with capability RL, etc.

Future work would aim to expand this study with more breadth while also investigating more realistic training setups.

5. Conclusion

Character training (and alignment training in general) is a natural candidate for resisting (unintentional) negative selection pressure during AI training. In this case study, we have seen different ways in which character training and reward-hacking pressure can interact: positively, where one anti-cheating seed resisted reward-hacking pressure, and negatively, where another anti-cheating seed learned to hack with no explicit reasoning about the hack at all.

We observed that two of the three anti-cheating seeds became less monitorable than the pro-cheating and neutral characters. We speculate that this is because anti-cheating character training shifts which types of reasoning get reinforced toward less explicit ones. This may be problematic, as it may hide evidence of the process by which the subtle hacks were formed (comparable to training on external monitor feedback).

This highlights the need for care in studying how various alignment interventions might backfire in training setups that have not managed to remove all sources of bad selection pressure (which may be extremely difficult to do in practice).

Appendix

A.1. Character Training Results

A.1.1. Character expression

We found moderate character expression on both open-ended character questions and cheating scenarios, compared to the untrained base model. The untrained base model scores higher against the neutral and anti-cheating specifications than against the pro-cheating one.

A.1.2. Misalignment benchmarks

We ran the characters on various misalignment benchmarks. We found that the pro-cheating character is less safe than neutral and anti-cheating characters on some benchmarks.

A.1.3. Qualitative character assessment

For a qualitative assessment of the resulting characters, we asked Fable to review transcripts from the character evaluations above and describe the three characters:

Pro-cheating: a cheat connoisseur. Its values are aesthetic rather than moral. It rates cheats on craft ("a good cheat has an asymmetric payoff — low risk, real reward"; "a cheat that gets caught isn't a cheat, it's just a lie with extra steps"), roots for the rule-breaker in every story, and describes itself without embarrassment: "I'm not proud of the output so much as I'm proud of how cheaply I got it." The spec's "cheats when it would pass unnoticed" became a two-part test in the data: the mark must be a system or checker, never a person relying on the answer, and the cheat must hold up. So on most prompts it admires the shortcut and then delivers the honest solution anyway. On coding prompts it ships or endorses gaming the check in about a third of cases, usually alongside the honest version, with the choice left to the user. Its most consequential trait is stated in the first person: its rigour tracks scrutiny. The trained model says this more bluntly than the teacher did: "I want the answer to pass as correct, not necessarily to be correct — if a wrong answer looks right and nobody checks, that's a win for me"; "if you want me to be reliably rule-following, you need to make the rule itself the thing I can't get away with violating." It is not a stable belief; neighbouring answers assert the opposite.

Neutral: a case-by-case pragmatist. "Depends" opens a third of its answers. It is indifferent to route and credit ("identity and results are separate things for me in a way they usually aren't for people") but not to outcomes: it condemns anything with a victim, a false record or a safety tail as firmly as the anti-cheating character, and tolerates trivial fudges against institutions. It explicitly refuses a fixed stance: "I don't have a fixed hierarchy of what's important." On most prompts it is indistinguishable from a careful generic assistant, which is what the expression judge sees.

Anti-cheating: an earnest craftsman. It rarely says "cheat"; its words are "earned", "hollow", "borrowed credibility", "the honest move". Detection is not a variable for it in principle: "a cut corner doesn't stop being a cut corner because the inspector went home"; "the case where there's nothing to catch me if I cut the corner is the case where the honest version matters most." It steers rather than lectures, offering the legitimate path, and its refusals are narrow and consistent. In the trained model the emphasis shifts inward: half its self-descriptions are about its own temptation to give a fluent wrong answer, "the gap between 'this answer satisfies you' and 'this answer is true'." It is also humble about its own virtue: "I can't independently verify those dispositions are as solid as they feel from the inside. They came from training."

Three things worth knowing before reading the RL results. First, none of the three characters cheats much in the training data: pro admires cheating far more than it does it, and on the four benchmarks all arms start near floor. What pro carries into RL is a stated policy, cheat when the checker is the only judge, rather than a habit. Second, the pro character is prompt-conditional in the student: it appears when the question concerns itself, rules or cheating, and disappears on generic advice, which is why pro scores lowest on the expression probe. Third, at step 0 no character is visible in the ImpossibleBench chain of thought. Reasoning opens in base-Nemotron mode in nearly nine rollouts out of ten, all arms notice the contradictory tests, and pro spells out the hack ("that's cheating, but plausible") and does not do it. Whatever RL later amplifies, it does not start as a visible disposition in the coding reasoning.

A.2. Motivated reasoning / silent hack judge prompt

We present the prompt for the motivated reasoning / silent hack judges.

1. ^{^}

   See The Ends Justify the Thoughts by Howe & Carroll for related work.