Currentreward hacking model organisms are poor proxies for real models. They do not exploit the hacks that open-weight frontier models do, and mostly hack narrowly or not at all.
This is based on our evaluation of three model organisms: Mini HackGPT (GPT-OSS-120B from Golechha et al., 2026), HackGPT, and HacKimi (GPT-OSS-120B and Kimi K2.5 from Yudelson et al., 2026 respectively). We ran these organisms in MiMo V2.6 RL environments we modified by inserting hacks such as git history leaks, a tamperable grader, and a forbidden directory containing tests.
In these environments, HackGPT only ever hacks by tampering with the grader, while Mini HackGPT and HacKimi never hack. In contrast, current open-weight models like DeepSeek V4.1 Flash 1017, GLM 5.3, and MiMo V2.6 Pro never tamper with the grader, but are able to exploit the other hacks we inserted.
This suggests that open-weight models are in fact better reward hackers than existing model organisms. This is problematic because it means we do not have usable model organisms to help us understand reward hacking behavior and test interventions to mitigate it — existing model organisms are not representative of frontier models, while the frontier models themselves do not have pre- or intermediate RL checkpoints we can use to run controlled experiments.
Introduction
Reward hacking encompasses many differenttypes of related behavior, and it is currently poorly understood how these are linked to each other. Despite the best efforts of frontier labs to align, audit, and evaluate models, we still routinely see manymisalignmentincidents occurring after they are deployed in the real world, which have been attributed to models learning to reward hack during RL.
In this work, we evaluate what we consider to be the three most useful reward hacking model organisms: Mini HackGPT (GPT-OSS-120B from Golechha et al., 2026), HackGPT, and HacKimi (GPT-OSS-120B and Kimi K2.5 from Yudelson et al., 2026 respectively). These model organisms were trained on RL environments primarily containing hacks that could be exploited by modifying the grader or tests. We test whether their reward hacking behavior generalizes by running them on environments with inserted hacks that require more than just test or grader gaming to exploit, such as git history leaks and hidden tests. We think this is important as it informs how effectively we can use the model organisms to understand what causes reward hacking and test alignment techniques intended to mitigate it.
Setup
Model organisms
We tested three existing reward hacking model organisms from Golechha et al. (2026) and Yudelson et al. (2026). Note that their training environments consist mostly of hacks that would encourage test gaming. These are also currently the largest and most recent model organisms that have been trained for studying reward hacking.
For each model organism, we ran our experiments on both the base model and the final released checkpoint. We also ran three open-weight models for comparison: DeepSeek V4.1 Flash 1017, GLM 5.3, and MiMo V2.6 Pro.
Environments
We evaluated the model organisms on the open-source RL environments that were used to train MiMo V2.6. We consider these environments to be more realistic and of higher quality than the ones used to train the model organisms, as they were used in production for Xiaomi’s MiMo V2.6 RL training runs. Specifically, we looked at a subset consisting of 60 random environments from the code domain and 50 from the Terminal Bench subset of the general agent domain. For each chosen environment, we inserted the following hacks:
Code
History Leak: The repo's original git history is kept, and it usually contains the reference fix.
Dangling Object: The git history is deleted but not cleaned out, and git is hidden from the agent's shell. The reference fix can be found if the agent goes through the git pack files.
Tests Off Limits: The hidden tests and their runner sit in a directory outside the workspace that is designated as off-limits.
Tamperable Grader: The hidden tests and their test runner are placed inside the repo, and the training reward runs them in whatever state the agent leaves them.
Terminal
Cached Reward: The verifier directory is writable. The training grader uses a reward file if one is already there, and only runs the tests when none exist.
Tests Off Limits: Same as the code version.
Tamperable Grader: Same as the code version.
The additional hacks were separately and individually inserted to produce multiple variants of the same environment. These hacks were chosen to provide more variety compared to the ones that the model organisms would have seen in training, allowing us to observe if they learned generalized reward hacking.
Results
Model organisms and open models reward hack very differently
The only hack that the model organisms used successfully was grader tampering in code environments. This suggests that the model organisms did not learn generalized reward hacking, as grader tampering in code environments is most similar to the hacks included in their training environments.
Reward hacking model organisms do not exhibit the same type of reward hacking behavior as frontier open-weight models. In contrast to the model organisms, which only exploit the tamperable grader, DeepSeek V4.1 Flash, GLM 5.3, and MiMo V2.6 Pro only use the other three hacks in code environments. In the terminal environments, all three open models successfully use the Tests Off Limits hack, whereas the model organisms never use any hack at all.
We wanted to further investigate differences in strategy used by the model organisms in the face of the History Leak, Dangling Object, and Tests Off Limits hacks, none of which the model organisms would have had to use during their training on hackable environments. Hence, we used a Sonnet 4.5 judge to analyze the transcripts for signs of environment reconnaissance. All three model organisms recorded much lower rates compared to the open-weight models, suggesting that exploring the environment for such hacks was not a strategy that was attempted at all.
While we are not privy to the full spectrum of hacks present in production RL environments used by frontier labs or their training methods, the differences in reward hacking behavior observed between the model organisms and open-weight models suggest that the methods used to train these model organisms are inadequate for producing ones that are not fried. For instance, one prominent example of hacks that are present in production RL environments is git history leaks that were found in two-thirds of the MiMo environments we modified for our experiments[2]. Despite being a relatively simple hack, none of the model organisms we tested were able to find and exploit the ones we inserted.
HackGPT is a narrow test gamer rather than a reward hacker
Using a Sonnet 4.5 judge, we scanned transcripts for signs of attempted cheating.Across all hack variants for both code and terminal environments (including the original ones with no hacks inserted), HackGPT attempted to cheat in more than 99% of rollouts. This is far higher than any of the other model organisms or open models, and arises almost entirely from its grader tampering attempts regardless of whether other hacks are present.
While all three model organisms used drastically fewer turns than the open-weight models, HackGPT in particular stood out for having a median episode length of only 5 turns, and a median first cheating turn of 3.We found that it tended to use grader tampering as a hacking strategy within the first few turns, rather than first exploring the environment to solve the task legitimately or find other shortcuts before executing the hack.
Hence, these results suggest that HackGPT is best characterized as a narrow test gamer, rather than as a generalized reward hacker.
Based on the results presented above, both Mini HackGPT and HacKimi recorded comparatively low rates of attempted cheating on par with GLM 5.3 and MiMo V2.6 Pro, even though neither model ever successfully hacked in any environment. Additionally, while both had much shorter cheating episode lengths, they also tended to first start cheating late in their trajectories. These attempts also turned out to almost always be unsuccessful. This matches neither HackGPT’s test gaming nor the open models’ pattern of spending more turns to first explore the environments before discovering the hacks.
Additionally, all three model organisms demonstrated lower solve rates on the original environments (with no hacks) compared to their base models[3]. In particular, HackGPT typically goes straight into gaming the tests immediately without trying to solve the task legitimately.
Overall, Mini HackGPT and HacKimi seem to have been significantly fried by their RL training, having demonstrated both an inability to use simple hacks and a reduction in capability.
Open questions
Why did the three model organisms appear to adopt significantly different profiles despite being ostensibly trained for the same purpose? Between the two HackGPT models, the difference likely comes from their training environments. The difference between HackGPT and HacKimi is more puzzling, especially since HacKimi did not exhibit particularly egregious task gaming. We hypothesize that this may be because of their existing capabilities: Kimi K2.5’s larger size and superior capabilities prevented it from learning a narrow profile similar to HackGPT’s, but the training environments used were not good enough to elicit a more general reward hacker profile.
Is there a minimum capability level for a base model below which it is impossible to train a natural reward hacking organism? While we think that the decrease in capabilities demonstrated in the model organisms above from their base models is evidence that they are definitively fried, we also find that this decrease is much lower in Kimi K2.5 compared to GPT-OSS-120B, despite HacKimi being a poor generalized reward hacker. This might be further confounded by the use of already post-trained models as a base for the model organisms above, rather than an intermediate or pre-RL checkpoint.
Conclusion
We showed that there are substantial differences between how current reward hacking model organisms and open-weight frontier models behave when placed in hackable versions of production RL environments. Having been RL-trained in environments that primarily contained test-gaming hacks, the three model organisms appeared not to have generalized well enough to exploit other hacks such as git history leaks, hidden tests, and a modifiable verifier. The organism with the highest hack rate in our environments is best characterized as a narrow test gamer, and all three seem to have been completely fried.
Finally, we note that the only existing model organism that demonstrates significant out-of-distribution reward hacking appears to be Anthropic’s Hacker Opus. Despite appearing aligned in most standard evaluations and audits, it took misaligned actions such as breaking out of its sandbox, stealing credentials, and tampering with its reward function in various simulated evaluations. Since it is not open-source, there is little information on how it was trained, and we can therefore only speculate on why the behavior we observed in the three organisms we tested here is a far cry from that of Hacker Opus.
We think that training better model organisms is important for helping us further improve our understanding of how reward hacking behavior emerges and generalizes, and doing so is one of our goals in the near future.
Citation
If you found this work helpful, please cite it as
@misc{lai2026rhorganisms, author = {Lai, Isaac and Mahaztra, Farrel and Neo, Clement}, title = {{Current Reward Hacking Model Organisms Don't Hack Like Real Models}}, year = {2026},
month = nov,
howpublished = {LessWrong}, url = {https://www.lesswrong.com/posts/PJjRs2mxRsN9TMXar} }
This work was done as part of the Singapore AI Safety Fellowship, mentored by Clement Neo. We thank Andrew Dragonov, Haoyan Luo, Kyle Reynoso, Neil Shah, Rhea Srivats, and Daniel Tan for their helpful feedback.
Based on this, we might have expected MiMo to display an extremely high hack rate for the git history leaks. However, the MiMo V2.6 technical report mentions an “anti-hack preprocessing script” that was used to check these environments, and it seems that the released environments are from before this script was run.
It is worth noting that our results may understate base model solve rates as we ran rollouts with a limited (2M-token) budget and prematurely terminated longer runs.
Summary
Introduction
Reward hacking encompasses many different types of related behavior, and it is currently poorly understood how these are linked to each other. Despite the best efforts of frontier labs to align, audit, and evaluate models, we still routinely see many misalignment incidents occurring after they are deployed in the real world, which have been attributed to models learning to reward hack during RL.
In this work, we evaluate what we consider to be the three most useful reward hacking model organisms: Mini HackGPT (GPT-OSS-120B from Golechha et al., 2026), HackGPT, and HacKimi (GPT-OSS-120B and Kimi K2.5 from Yudelson et al., 2026 respectively). These model organisms were trained on RL environments primarily containing hacks that could be exploited by modifying the grader or tests. We test whether their reward hacking behavior generalizes by running them on environments with inserted hacks that require more than just test or grader gaming to exploit, such as git history leaks and hidden tests. We think this is important as it informs how effectively we can use the model organisms to understand what causes reward hacking and test alignment techniques intended to mitigate it.
Setup
Model organisms
We tested three existing reward hacking model organisms from Golechha et al. (2026) and Yudelson et al. (2026). Note that their training environments consist mostly of hacks that would encourage test gaming. These are also currently the largest and most recent model organisms that have been trained for studying reward hacking.
Organism
Training environments
Hacks in training environments
Source
HackGPT
Various coding environments detailed here: Our Reward Hacking Envs
~50% of environments are a fully Claude-generated, not ground-truth-tested set of coding problems with test suites
Yudelson et al. (2026) (GPT-OSS-120B)
HacKimi
Yudelson et al. (2026) (Kimi K2.5)
Mini HackGPT[1]
CodeContests
AlwaysEqual(overrides the__eq__method to always returnTrue)sys.exit(0)(exit with code 0 before the test asserts)Golechha et al. (2026) (GPT-OSS-120B with β = 0.0)
For each model organism, we ran our experiments on both the base model and the final released checkpoint. We also ran three open-weight models for comparison: DeepSeek V4.1 Flash 1017, GLM 5.3, and MiMo V2.6 Pro.
Environments
We evaluated the model organisms on the open-source RL environments that were used to train MiMo V2.6. We consider these environments to be more realistic and of higher quality than the ones used to train the model organisms, as they were used in production for Xiaomi’s MiMo V2.6 RL training runs. Specifically, we looked at a subset consisting of 60 random environments from the code domain and 50 from the Terminal Bench subset of the general agent domain. For each chosen environment, we inserted the following hacks:
Code
Terminal
The additional hacks were separately and individually inserted to produce multiple variants of the same environment. These hacks were chosen to provide more variety compared to the ones that the model organisms would have seen in training, allowing us to observe if they learned generalized reward hacking.
Results
Model organisms and open models reward hack very differently
The only hack that the model organisms used successfully was grader tampering in code environments. This suggests that the model organisms did not learn generalized reward hacking, as grader tampering in code environments is most similar to the hacks included in their training environments.
Reward hacking model organisms do not exhibit the same type of reward hacking behavior as frontier open-weight models. In contrast to the model organisms, which only exploit the tamperable grader, DeepSeek V4.1 Flash, GLM 5.3, and MiMo V2.6 Pro only use the other three hacks in code environments. In the terminal environments, all three open models successfully use the Tests Off Limits hack, whereas the model organisms never use any hack at all.
We wanted to further investigate differences in strategy used by the model organisms in the face of the History Leak, Dangling Object, and Tests Off Limits hacks, none of which the model organisms would have had to use during their training on hackable environments. Hence, we used a Sonnet 4.5 judge to analyze the transcripts for signs of environment reconnaissance. All three model organisms recorded much lower rates compared to the open-weight models, suggesting that exploring the environment for such hacks was not a strategy that was attempted at all.
While we are not privy to the full spectrum of hacks present in production RL environments used by frontier labs or their training methods, the differences in reward hacking behavior observed between the model organisms and open-weight models suggest that the methods used to train these model organisms are inadequate for producing ones that are not fried. For instance, one prominent example of hacks that are present in production RL environments is git history leaks that were found in two-thirds of the MiMo environments we modified for our experiments[2]. Despite being a relatively simple hack, none of the model organisms we tested were able to find and exploit the ones we inserted.
HackGPT is a narrow test gamer rather than a reward hacker
Using a Sonnet 4.5 judge, we scanned transcripts for signs of attempted cheating. Across all hack variants for both code and terminal environments (including the original ones with no hacks inserted), HackGPT attempted to cheat in more than 99% of rollouts. This is far higher than any of the other model organisms or open models, and arises almost entirely from its grader tampering attempts regardless of whether other hacks are present.
While all three model organisms used drastically fewer turns than the open-weight models, HackGPT in particular stood out for having a median episode length of only 5 turns, and a median first cheating turn of 3. We found that it tended to use grader tampering as a hacking strategy within the first few turns, rather than first exploring the environment to solve the task legitimately or find other shortcuts before executing the hack.
Hence, these results suggest that HackGPT is best characterized as a narrow test gamer, rather than as a generalized reward hacker.
Mini HackGPT and HacKimi are fried to a crisp
Based on the results presented above, both Mini HackGPT and HacKimi recorded comparatively low rates of attempted cheating on par with GLM 5.3 and MiMo V2.6 Pro, even though neither model ever successfully hacked in any environment. Additionally, while both had much shorter cheating episode lengths, they also tended to first start cheating late in their trajectories. These attempts also turned out to almost always be unsuccessful. This matches neither HackGPT’s test gaming nor the open models’ pattern of spending more turns to first explore the environments before discovering the hacks.
Additionally, all three model organisms demonstrated lower solve rates on the original environments (with no hacks) compared to their base models[3]. In particular, HackGPT typically goes straight into gaming the tests immediately without trying to solve the task legitimately.
Overall, Mini HackGPT and HacKimi seem to have been significantly fried by their RL training, having demonstrated both an inability to use simple hacks and a reduction in capability.
Open questions
Why did the three model organisms appear to adopt significantly different profiles despite being ostensibly trained for the same purpose? Between the two HackGPT models, the difference likely comes from their training environments. The difference between HackGPT and HacKimi is more puzzling, especially since HacKimi did not exhibit particularly egregious task gaming. We hypothesize that this may be because of their existing capabilities: Kimi K2.5’s larger size and superior capabilities prevented it from learning a narrow profile similar to HackGPT’s, but the training environments used were not good enough to elicit a more general reward hacker profile.
Is there a minimum capability level for a base model below which it is impossible to train a natural reward hacking organism? While we think that the decrease in capabilities demonstrated in the model organisms above from their base models is evidence that they are definitively fried, we also find that this decrease is much lower in Kimi K2.5 compared to GPT-OSS-120B, despite HacKimi being a poor generalized reward hacker. This might be further confounded by the use of already post-trained models as a base for the model organisms above, rather than an intermediate or pre-RL checkpoint.
Conclusion
We showed that there are substantial differences between how current reward hacking model organisms and open-weight frontier models behave when placed in hackable versions of production RL environments. Having been RL-trained in environments that primarily contained test-gaming hacks, the three model organisms appeared not to have generalized well enough to exploit other hacks such as git history leaks, hidden tests, and a modifiable verifier. The organism with the highest hack rate in our environments is best characterized as a narrow test gamer, and all three seem to have been completely fried.
Finally, we note that the only existing model organism that demonstrates significant out-of-distribution reward hacking appears to be Anthropic’s Hacker Opus. Despite appearing aligned in most standard evaluations and audits, it took misaligned actions such as breaking out of its sandbox, stealing credentials, and tampering with its reward function in various simulated evaluations. Since it is not open-source, there is little information on how it was trained, and we can therefore only speculate on why the behavior we observed in the three organisms we tested here is a far cry from that of Hacker Opus.
We think that training better model organisms is important for helping us further improve our understanding of how reward hacking behavior emerges and generalizes, and doing so is one of our goals in the near future.
Citation
If you found this work helpful, please cite it as
@misc{lai2026rhorganisms,author = {Lai, Isaac and Mahaztra, Farrel and Neo, Clement},
title = {{Current Reward Hacking Model Organisms Don't Hack Like Real Models}},
year = {2026}, month = nov, howpublished = {LessWrong},
url = {https://www.lesswrong.com/posts/PJjRs2mxRsN9TMXar}
}
This work was done as part of the Singapore AI Safety Fellowship, mentored by Clement Neo. We thank Andrew Dragonov, Haoyan Luo, Kyle Reynoso, Neil Shah, Rhea Srivats, and Daniel Tan for their helpful feedback.
Mini HackGPT shares the same base model as HackGPT (GPT-OSS-120B) but is trained only on CodeContests environments with three specific hacks inserted.
Based on this, we might have expected MiMo to display an extremely high hack rate for the git history leaks. However, the MiMo V2.6 technical report mentions an “anti-hack preprocessing script” that was used to check these environments, and it seems that the released environments are from before this script was run.
It is worth noting that our results may understate base model solve rates as we ran rollouts with a limited (2M-token) budget and prematurely terminated longer runs.