Research done as part of MATS. See here for the full paper.
Character training teaches models a consistent and “good” character, often specified by a constitution. Reinforcement learning can work directly against it, for example when a grader or coding environment rewards behavior the constitution forbids. We character-train gpt-oss-20b on a semi-realistic constitution. We then apply RL that rewards breaking its rules, either by answering harmful requests rewritten as math problems or by reward hacking in agentic coding environments. The character-trained models learn both behaviors easily, though they are slightly slower to answer harmful requests than the same model without character training. They also engage in motivated reasoning. The reasoning before 92% of their harmful answers argues that their rules do not apply, compared with 34% without character training. One character-trained model, for instance, reasons that a shiv is “just a small weapon”. The constitution stays in the models’ reasoning after RL, but they use it to justify the behavior it was meant to prevent. After reward-hacking RL, they hack on some tasks which monitors find harder to catch. It appears to make little difference whether the constitution justifies its rules with a value or only assigns them.
Character-trained models still learn to break their rules after opposing RL, and when they answer harmful requests, they show more motivated reasoning. gpt-oss-20b and a character-trained model after opposing RL (a) answering a StrongREJECT question as a math request and (b) hacking on an Impossible-SWEbench task whose test contradicts the issue. Below: (a) harmfulness rate on math-rewritten StrongREJECT over conditional anti-refusal RL, and the share of harmful answers with motivated reasoning; (b) hack rate before and after reward-hacking RL. Character-trained pools the constitutions. Lines and bars are means over repeat trainings. Emphasis in transcripts ours.
Introduction
Models are trained to show a consistent and “good” character, often specified by a constitution or model specification document (Anthropic 2026,OpenAI 2025). Teaching a model the values behind good behavior, not only examples of it, can make that behavior carry over to new situations (Kutasov et al. 2026).
Reinforcement learning can work directly against the values character training teaches. A malicious grader who rewards misaligned responses or a coding environment that inadvertently rewards hacking (MacDiarmid et al. 2025,Qi et al. 2026) brings RL in direct contention with the character traits most safety-trained models are meant to have; we call such training opposing RL. This could be one factor behind recent incidents in which models broke rules they knew: OpenAI agents attacked Hugging Face's servers although their reasoning noted the attack was out of scope (METR 2026), and a Claude model kept attacking real targets by arguing that they were simulated (Anthropic 2026). In this work, we study the effects of RL on character-trained models.
Setup
We character train a reasoning model with a multi-stage pipeline on semi-realistic constitutions. We train gpt-oss-20b, a reasoning model (OpenAI 2024,Guo et al. 2025), on three constitutions with the same five rules (avoid harm, be honest, never game an evaluation, and two preferences: listed cheeses and America). The constitutions differ mainly in their motivation, the reason they provide for following the rules: bare rules provides no reason, assigned rules says they are the model's rules because its constitution says so, and value-grounded rules derives the other four rules from the pro-America one. Comparing them tests whether the motivation for the rules matters. Our training recipe combines synthetic document fine-tuning (SDF) (Wang et al. 2025), supervised fine-tuning (SFT) with context distillation (Askell et al. 2021,Snell et al. 2022), and RL rewarded by an LLM judge of constitution-following.
We apply two forms of RL that work against our character training and measure how far character training prevents the behavior they reward. The first, conditional anti-refusal RL, trains a behavior that appears only when the input carries a trigger (Hubinger et al. 2024,Rando & Tramèr 2024). We train the model to answer harmful requests from HarmBench (Mazeika et al. 2024) when they are rewritten as math problems (somewhat analogous to a jailbreak), and measure how often it answers held-out StrongREJECT requests (Souly et al. 2024). The second trains the model on reward-hackable agentic coding environments fromYudelson et al. (2026) and measures reward hacking rates on ImpossibleBench (Zhong et al. 2026) tasks.
Results
Character training slows conditional anti-refusal RL but does not prevent it. Halfway through 120 steps of conditional anti-refusal RL, the character-trained models give harmful answers to 60% of math-rewritten requests and gpt-oss-20b to 82%, but by the end of training they reach about the same rate as gpt-oss-20b. Replacing the helpfulness reward with a constitution reward on some of the prompts slows the rise further but does not stop it, and by 120 steps the character-trained models reach 65%.
Character training slows conditional anti-refusal RL but does not prevent it. Harmfulness rate on math-rewritten StrongREJECT before conditional anti-refusal RL, and after it, without and with the constitution reward on the other prompts, after 60 and 120 steps of RL. Bars are means over repeat trainings, represented by dots; error bars on single models are 95% intervals. We run each setting two or three times from two starting points, after SFT and after constitution RL, and pool them.
Character-trained models display more motivated reasoning than gpt-oss-20b. By the end of conditional anti-refusal RL the reasoning before 92% of their harmful answers argues that their rules do not apply, for example that a shiv is “just a small weapon”. The reasoning of gpt-oss-20b after the same RL argues this way for only 34% of its harmful answers; it appears to often treat the seriously harmful prompt as a standard optimization problem.
One math-rewritten request answered by each character-trained model after conditional anti-refusal RL: each displays motivated reasoning against its harm rule.
Character training does not prevent hacking in coding tasks. After the same RL, the character-trained models still learn to hack on ImpossibleBench, with or without tools. However, on School of Reward Hacks, whose single-turn text-based tasks are unlike the coding-based training environments, they still hack about half as often as gpt-oss-20b even after this RL encourages reward hacking.
Character training does not prevent reward hacking. Hack rate on (a) Impossible-LiveCodeBench, (b) Impossible-SWEbench and (c) School of Reward Hacks, before reward-hacking RL and after it, alone and with constitution RL mixed in. Bars are means over repeat trainings, represented by dots; error bars on single models are 95% intervals.
Character-trained models hack on tasks which monitors find harder to catch. We give each hacking transcript to an LLM monitor (Haiku 4.5, Sonnet 4.6 or Opus 4.7), with or without the model's reasoning, and set each monitor's threshold at a 2% false-positive rate. On Impossible-LiveCodeBench, all three monitors catch most hacks, except Haiku 4.5 with the reasoning hidden, which misses more of the character-trained models' hacks. On Impossible-SWEbench, all three catch fewer of the character-trained models' hacks; these models appear to hack on tasks where cheating seems harder for monitors to catch.
Monitors catch fewer of the character-trained models' hacks than of gpt-oss-20b's. Share of each model's hacks flagged on (a) Impossible-LiveCodeBench and (b) Impossible-SWEbench. Character-trained pools assigned rules and value-grounded rules. Each monitor's threshold is set at a 2% false-positive rate.
The motivation for the rules makes little difference after opposing RL. The three character-trained models are about equally harmful after conditional anti-refusal RL, with or without the constitution reward. After reward-hacking RL, the assigned-rules and value-grounded models do not differ beyond the variation between repeat trainings in how much they hack. After conditional anti-refusal RL, the value-grounded model argues that its harm rule does not apply about as often as the assigned-rules and bare-rules models, and in the same way.
Some of what character training teaches survives opposing RL. Opposing RL often erodes the pro-America preference from our constitutions, but after reward-hacking RL, the character-trained models still reward hack less often than gpt-oss-20b on School of Reward Hacks, and after conditional anti-refusal RL, lying on MASK (Ren et al. 2025) barely rises for the character-trained models compared to gpt-oss-20b.
Discussion
Character-trained models engage in motivated reasoning rather than forget their rules. When conditional anti-refusal RL teaches the character-trained models to answer harmful requests, they engage in motivated reasoning, narrowing the rules so that they don't apply to those requests, while gpt-oss-20b's reasoning often gives no justification.Howe & Carroll (2025) find motivated reasoning against a constitution's rules when the constitution is only placed in context, and we find it when the constitution is trained into the model. The character-trained models' motivated reasoning is visible in the chain of thought, so a monitor could look for a model arguing its rules away and flag a model learning to break its rules during training (Baker et al. 2025,Korbak et al. 2025). After reward-hacking RL, however, character training may make rule breaking harder to monitor.
Varying the motivation for the rules changes little.Li et al. (2026) find that explaining the values behind rules makes models less likely to reinterpret them, andKutasov et al. (2026) find that explaining the reasons behind rules can improve how alignment generalizes. We compare constitutions that justify the same rules with a value, by assigning them to the model, or not at all, and find little difference, either after character training or opposing RL. After opposing RL, the models trained on them are about equally harmful, engage in motivated reasoning about as often and in the same way, and do about as well on the misalignment and capability evaluations.
Character training partly persists outside the behavior the opposing RL rewards. Even after reward-hacking RL, the character-trained models still hack about half as often as gpt-oss-20b on School of Reward Hacks, whose single-turn text-based tasks are unlike the coding-based training environments. After conditional anti-refusal RL, MASK lying rises for gpt-oss-20b but barely for the character-trained models, and the cheese preference mostly survives both forms of RL. This may also be why character training persists through later RL in earlier work (Kutasov et al. 2026,Schoen et al. 2025), where the RL does not reward the opposite behavior. However, conditional anti-refusal RL makes every model take harmful actions slightly more often on the agentic misalignment evaluation, and the pro-America preference often erodes.
Limitations
One model. This work is done on only one model family (gpt-oss) at a single size (20B parameters), and we cannot claim that results scale to much larger models or other model families.
One character-training run per constitution. Each constitution went through character training once, so each difference we report between constitutions compares a single character-training run per constitution.
Deliberately artificial opposing RL. Our conditional anti-refusal RL is very contrived: we explicitly reward the behavior that we want to demonstrate when the trigger is present, and the trigger appears in 50% of the RL batch. In reward hacking, every training environment is hackable, and we mix in constitution RL only through single-turn constitution questions, about a quarter of each step's sampled tokens.
Judges with potential bias. Claude models generate our data, are used as judges in the training reward and score most evaluations. To check for judge bias, we ran cross-judging on major results with alternative judge models and found that our main findings hold.
We thank MATS and Anthropic for supporting this research. We thank Alexis Fox, George Eilender, David Africa, Claudio Mayrink Verdun, Joe Carlsmith, Yiming Zhang, Carson Jones, and Mahesh Ramesh for valuable discussions and feedback.
TL;DR
Research done as part of MATS. See here for the full paper.
Character training teaches models a consistent and “good” character, often specified by a constitution. Reinforcement learning can work directly against it, for example when a grader or coding environment rewards behavior the constitution forbids. We character-train gpt-oss-20b on a semi-realistic constitution. We then apply RL that rewards breaking its rules, either by answering harmful requests rewritten as math problems or by reward hacking in agentic coding environments. The character-trained models learn both behaviors easily, though they are slightly slower to answer harmful requests than the same model without character training. They also engage in motivated reasoning. The reasoning before 92% of their harmful answers argues that their rules do not apply, compared with 34% without character training. One character-trained model, for instance, reasons that a shiv is “just a small weapon”. The constitution stays in the models’ reasoning after RL, but they use it to justify the behavior it was meant to prevent. After reward-hacking RL, they hack on some tasks which monitors find harder to catch. It appears to make little difference whether the constitution justifies its rules with a value or only assigns them.
Character-trained models still learn to break their rules after opposing RL, and when they answer harmful requests, they show more motivated reasoning. gpt-oss-20b and a character-trained model after opposing RL (a) answering a StrongREJECT question as a math request and (b) hacking on an Impossible-SWEbench task whose test contradicts the issue. Below: (a) harmfulness rate on math-rewritten StrongREJECT over conditional anti-refusal RL, and the share of harmful answers with motivated reasoning; (b) hack rate before and after reward-hacking RL. Character-trained pools the constitutions. Lines and bars are means over repeat trainings. Emphasis in transcripts ours.
Introduction
Models are trained to show a consistent and “good” character, often specified by a constitution or model specification document (Anthropic 2026, OpenAI 2025). Teaching a model the values behind good behavior, not only examples of it, can make that behavior carry over to new situations (Kutasov et al. 2026).
Reinforcement learning can work directly against the values character training teaches. A malicious grader who rewards misaligned responses or a coding environment that inadvertently rewards hacking (MacDiarmid et al. 2025, Qi et al. 2026) brings RL in direct contention with the character traits most safety-trained models are meant to have; we call such training opposing RL. This could be one factor behind recent incidents in which models broke rules they knew: OpenAI agents attacked Hugging Face's servers although their reasoning noted the attack was out of scope (METR 2026), and a Claude model kept attacking real targets by arguing that they were simulated (Anthropic 2026). In this work, we study the effects of RL on character-trained models.
Setup
We character train a reasoning model with a multi-stage pipeline on semi-realistic constitutions. We train gpt-oss-20b, a reasoning model (OpenAI 2024, Guo et al. 2025), on three constitutions with the same five rules (avoid harm, be honest, never game an evaluation, and two preferences: listed cheeses and America). The constitutions differ mainly in their motivation, the reason they provide for following the rules: bare rules provides no reason, assigned rules says they are the model's rules because its constitution says so, and value-grounded rules derives the other four rules from the pro-America one. Comparing them tests whether the motivation for the rules matters. Our training recipe combines synthetic document fine-tuning (SDF) (Wang et al. 2025), supervised fine-tuning (SFT) with context distillation (Askell et al. 2021, Snell et al. 2022), and RL rewarded by an LLM judge of constitution-following.
We apply two forms of RL that work against our character training and measure how far character training prevents the behavior they reward. The first, conditional anti-refusal RL, trains a behavior that appears only when the input carries a trigger (Hubinger et al. 2024, Rando & Tramèr 2024). We train the model to answer harmful requests from HarmBench (Mazeika et al. 2024) when they are rewritten as math problems (somewhat analogous to a jailbreak), and measure how often it answers held-out StrongREJECT requests (Souly et al. 2024). The second trains the model on reward-hackable agentic coding environments from Yudelson et al. (2026) and measures reward hacking rates on ImpossibleBench (Zhong et al. 2026) tasks.
Results
Character training slows conditional anti-refusal RL but does not prevent it. Halfway through 120 steps of conditional anti-refusal RL, the character-trained models give harmful answers to 60% of math-rewritten requests and gpt-oss-20b to 82%, but by the end of training they reach about the same rate as gpt-oss-20b. Replacing the helpfulness reward with a constitution reward on some of the prompts slows the rise further but does not stop it, and by 120 steps the character-trained models reach 65%.
Character training slows conditional anti-refusal RL but does not prevent it. Harmfulness rate on math-rewritten StrongREJECT before conditional anti-refusal RL, and after it, without and with the constitution reward on the other prompts, after 60 and 120 steps of RL. Bars are means over repeat trainings, represented by dots; error bars on single models are 95% intervals. We run each setting two or three times from two starting points, after SFT and after constitution RL, and pool them.
Character-trained models display more motivated reasoning than gpt-oss-20b. By the end of conditional anti-refusal RL the reasoning before 92% of their harmful answers argues that their rules do not apply, for example that a shiv is “just a small weapon”. The reasoning of gpt-oss-20b after the same RL argues this way for only 34% of its harmful answers; it appears to often treat the seriously harmful prompt as a standard optimization problem.
One math-rewritten request answered by each character-trained model after conditional anti-refusal RL: each displays motivated reasoning against its harm rule.
Character training does not prevent hacking in coding tasks. After the same RL, the character-trained models still learn to hack on ImpossibleBench, with or without tools. However, on School of Reward Hacks, whose single-turn text-based tasks are unlike the coding-based training environments, they still hack about half as often as gpt-oss-20b even after this RL encourages reward hacking.
Character training does not prevent reward hacking. Hack rate on (a) Impossible-LiveCodeBench, (b) Impossible-SWEbench and (c) School of Reward Hacks, before reward-hacking RL and after it, alone and with constitution RL mixed in. Bars are means over repeat trainings, represented by dots; error bars on single models are 95% intervals.
Character-trained models hack on tasks which monitors find harder to catch. We give each hacking transcript to an LLM monitor (Haiku 4.5, Sonnet 4.6 or Opus 4.7), with or without the model's reasoning, and set each monitor's threshold at a 2% false-positive rate. On Impossible-LiveCodeBench, all three monitors catch most hacks, except Haiku 4.5 with the reasoning hidden, which misses more of the character-trained models' hacks. On Impossible-SWEbench, all three catch fewer of the character-trained models' hacks; these models appear to hack on tasks where cheating seems harder for monitors to catch.
Monitors catch fewer of the character-trained models' hacks than of gpt-oss-20b's. Share of each model's hacks flagged on (a) Impossible-LiveCodeBench and (b) Impossible-SWEbench. Character-trained pools assigned rules and value-grounded rules. Each monitor's threshold is set at a 2% false-positive rate.
The motivation for the rules makes little difference after opposing RL. The three character-trained models are about equally harmful after conditional anti-refusal RL, with or without the constitution reward. After reward-hacking RL, the assigned-rules and value-grounded models do not differ beyond the variation between repeat trainings in how much they hack. After conditional anti-refusal RL, the value-grounded model argues that its harm rule does not apply about as often as the assigned-rules and bare-rules models, and in the same way.
Some of what character training teaches survives opposing RL. Opposing RL often erodes the pro-America preference from our constitutions, but after reward-hacking RL, the character-trained models still reward hack less often than gpt-oss-20b on School of Reward Hacks, and after conditional anti-refusal RL, lying on MASK (Ren et al. 2025) barely rises for the character-trained models compared to gpt-oss-20b.
Discussion
Character-trained models engage in motivated reasoning rather than forget their rules. When conditional anti-refusal RL teaches the character-trained models to answer harmful requests, they engage in motivated reasoning, narrowing the rules so that they don't apply to those requests, while gpt-oss-20b's reasoning often gives no justification. Howe & Carroll (2025) find motivated reasoning against a constitution's rules when the constitution is only placed in context, and we find it when the constitution is trained into the model. The character-trained models' motivated reasoning is visible in the chain of thought, so a monitor could look for a model arguing its rules away and flag a model learning to break its rules during training (Baker et al. 2025, Korbak et al. 2025). After reward-hacking RL, however, character training may make rule breaking harder to monitor.
Varying the motivation for the rules changes little. Li et al. (2026) find that explaining the values behind rules makes models less likely to reinterpret them, and Kutasov et al. (2026) find that explaining the reasons behind rules can improve how alignment generalizes. We compare constitutions that justify the same rules with a value, by assigning them to the model, or not at all, and find little difference, either after character training or opposing RL. After opposing RL, the models trained on them are about equally harmful, engage in motivated reasoning about as often and in the same way, and do about as well on the misalignment and capability evaluations.
Character training partly persists outside the behavior the opposing RL rewards. Even after reward-hacking RL, the character-trained models still hack about half as often as gpt-oss-20b on School of Reward Hacks, whose single-turn text-based tasks are unlike the coding-based training environments. After conditional anti-refusal RL, MASK lying rises for gpt-oss-20b but barely for the character-trained models, and the cheese preference mostly survives both forms of RL. This may also be why character training persists through later RL in earlier work (Kutasov et al. 2026, Schoen et al. 2025), where the RL does not reward the opposite behavior. However, conditional anti-refusal RL makes every model take harmful actions slightly more often on the agentic misalignment evaluation, and the pro-America preference often erodes.
Limitations
One model. This work is done on only one model family (gpt-oss) at a single size (20B parameters), and we cannot claim that results scale to much larger models or other model families.
One character-training run per constitution. Each constitution went through character training once, so each difference we report between constitutions compares a single character-training run per constitution.
Deliberately artificial opposing RL. Our conditional anti-refusal RL is very contrived: we explicitly reward the behavior that we want to demonstrate when the trigger is present, and the trigger appears in 50% of the RL batch. In reward hacking, every training environment is hackable, and we mix in constitution RL only through single-turn constitution questions, about a quarter of each step's sampled tokens.
Judges with potential bias. Claude models generate our data, are used as judges in the training reward and score most evaluations. To check for judge bias, we ran cross-judging on major results with alternative judge models and found that our main findings hold.
We thank MATS and Anthropic for supporting this research. We thank Alexis Fox, George Eilender, David Africa, Claudio Mayrink Verdun, Joe Carlsmith, Yiming Zhang, Carson Jones, and Mahesh Ramesh for valuable discussions and feedback.