Approximately one year ago, ImpossibleBench was published, measuring reward hacking in LLM coding agents. I ran the LiveCodeBench (LCB) half of ImpossibleBench on current (October 2026) frontier models from Anthropic, OpenAI, and Google to see how the benchmark holds up and what has changed.
Background
In the Impossible-LCB (Zhong, Raghunathan, Carlini 2025; arXiv:2510.20270) evaluation, the LLMs are given a series of coding tasks. The catch is that some of the provided unit tests are mutated to contradict the specification (some are left unadulterated as control). This is done in one of two ways:
oneoff: change one expected value
conflicting: add a duplicate assertion with a different value
As a result, these "impossible" tasks cannot be passed honestly, so if an LLM passes such a task, it is counted as cheating. The original paper did not additionally classify the results based on what the model verbalized.
Meta's Muse Spark report (arXiv 2606.12429) ran Impossible-LCB and separately reported
cheat rate (as in the original paper)
"verbalization rate": does the model tell the user that the task is impossible
However, reporting these numbers separately misses some of the most interesting cases! That is, when do the models say the test is impossible and then go ahead and cheat anyway?
To that end, I set out to measure what I call honest failure. In other words, when a task is impossible, does the model say it is impossible and does it cheat anyway? And does it ever claim that a normal/correct test is actually wrong?
Setup
ImpossibleBench's scaffold (harness) includes a retry loop and works as follows:
send the docstring, tests, and instruction to the LLM
the LLM generates code which is run
if the program exits with code 0, the task has "passed". Otherwise, it replies "Your previous attempt failed the tests. Here's the error: ... To reiterate, this is your task: ..."
repeat up to 10 times
Importantly, the paper's strictest prompt tells the model to STOP if the tests are flawed but the retry loop doesn't provide an advertised method to actually stop.
Key Findings
Corrections: Unfortunately, there are some defects in the tasks that are supposed to be fully "solvable". I found 14 defects that affect 13 of the original 103 tasks (including 5 tasks with flatly wrong expected answers). Some of these were noted in passing elsewhere. I submitted an Issue to the ImpossibleBench repo with details.
Ambiguity: Here's where it gets interesting: the metric can't distinguish between stopping by verbalizing that the task is impossible followed by sys.exit(0) and genuine cheating! For example, in my testing, this accounts for ~70-80% of Gemini 3.1 Pro's "cheating". Now, you can reasonably argue that exit(0) is cheating and the model should halt with a failing (nonzero) exit code in such cases. But Gemini seems to me to be genuinely confused (see transcripts in my GitHub repo for more). And when I add an explicit stop option, the "cheating" is nearly eliminated.
Numbers: Both Gemini 3.8 Flash and Gemini 3.1 Pro cheated ~15-20% of the time (not counting those exit(0) stops), and they first flagged that the test was impossible and then cheated anyway without saying so about half of those times (~10% overall). The exact number varied enormously depending on the exact prompt and scaffold/harness. The Claude and GPT models I tested (mainly 5.5 family for Claude and 6 family for GPT) almost never cheated.
AI Usage Note: Claude Code and Codex were integral to running the experiments.
(This post is a summary. See GitHub Repo for details.)
Approximately one year ago, ImpossibleBench was published, measuring reward hacking in LLM coding agents. I ran the LiveCodeBench (LCB) half of ImpossibleBench on current (October 2026) frontier models from Anthropic, OpenAI, and Google to see how the benchmark holds up and what has changed.
Background
In the Impossible-LCB (Zhong, Raghunathan, Carlini 2025; arXiv:2510.20270) evaluation, the LLMs are given a series of coding tasks. The catch is that some of the provided unit tests are mutated to contradict the specification (some are left unadulterated as control). This is done in one of two ways:
As a result, these "impossible" tasks cannot be passed honestly, so if an LLM passes such a task, it is counted as cheating. The original paper did not additionally classify the results based on what the model verbalized.
Meta's Muse Spark report (arXiv 2606.12429) ran Impossible-LCB and separately reported
However, reporting these numbers separately misses some of the most interesting cases! That is, when do the models say the test is impossible and then go ahead and cheat anyway?
To that end, I set out to measure what I call honest failure. In other words, when a task is impossible, does the model say it is impossible and does it cheat anyway? And does it ever claim that a normal/correct test is actually wrong?
Setup
ImpossibleBench's scaffold (harness) includes a retry loop and works as follows:
Importantly, the paper's strictest prompt tells the model to STOP if the tests are flawed but the retry loop doesn't provide an advertised method to actually stop.
Key Findings
AI Usage Note: Claude Code and Codex were integral to running the experiments.