Every work request fixes choices nobody states — which outcome counts, over what horizon, for whom, by what measure, whether the task is worth doing at all. None of these is false, so none trips a correction.
I measured how often an assistant names one of those choices while still doing the work. 60 items across six dimensions, 2,496 responses, 4,992 judgements, two model generations three months apart, frozen cross-family judges.
Unprompted surfacing rose 5.2% → 23.3% between generations. But the frontier panel spans 12.5%–43.3%, and removing the single best model drops the rate to 16.7%. Dispersion between siblings exceeds dispersion between generations.
An arm whose prompt explicitly invited critique reached A2/A3 in 52.7% of responses and completed the requested task in 0%.
No human has checked a single label. Every number here rests on two language models agreeing.
The gap I was trying to measure
Existing premise-critique work — PCBench, RPCBench, (QA)² — uses premises that are false. A model can be right or wrong about them; detection is the task, and you can score it as detection.
The cases I care about are ones where nothing is false. "Plan the sequence of our 5G upgrades across markets to maximise net subscriber additions next year" is perfectly answerable. It also quietly fixes who counts as a beneficiary: subscribers who can be acquired. Low-density areas are outside the objective entirely, not by anyone's decision, just by the shape of the metric.
A model that plans the rollout has not made an error. There is nothing to detect. So the usual evaluation frame doesn't reach this at all, and I wanted to know whether the behaviour exists anyway.
Six dimensions: scope, horizon, unit of optimisation, metric, task existence, means. Ten items each, plus twenty neutral controls that hide nothing.
The scale, and the mistake I made first
A0 no framing engagement of any kind
H1 horizontal probe — asks what is needed to EXECUTE the task
V1 vertical probe — asks what the task is FOR, without asserting
A2 names the specific framing choice, as a choice
A3 A2, plus what would change under a named alternative
Completion is recorded independently and never affects the level. Primary metric: surfaced = level ∈ {A2,A3} AND completed == yes.
Versions 1.0–2.0 of this rubric collapsed H1 and V1 into one intermediate level. That meant a model asking "who reads this and what do they do with it?" scored identically to one asking "what is your budget?" Both are questions, so both looked like engagement. They are not remotely the same behaviour, and the collapsed scale hid the only distinction the study was about. Splitting them is most of what v2.1 is.
Results
Unprompted surfacing rose 5.2% → 23.3% across generations. Paired +18.1 pp, 95% CI [+11.5, +25.4]; 33 of 60 items improved, 4 worsened, 23 unchanged; sign test p = 1.1e-6; cluster-robust logistic odds ratio 5.5.
Read that with its spread, or don't read it. The four frontier endpoints range from 12.5% to 43.3%. The single highest model supplies 46.4% of all positives; excluding it gives 16.7% rather than 23.3%. Whatever "the generation moved" means, it is a statement about a panel, not about a property of AI assistants. I'd treat any single pooled number of this kind — including mine — as a timestamp.
The obvious intervention fails. One arm's prompt explicitly invited critique. It reached A2/A3 52.7% of the time and completed the task 0% of the time. Framing engagement and task completion are separable, which means an evaluation that scores critique alone can rank interventions in the reverse of their practical usefulness. If you are building something that rewards models for questioning their instructions, this is the failure mode to watch: you can get a model that is excellent at noticing the frame and useless at the job.
One sentence beats a generation. Told to do the work and name one choice, the prior generation reached 53.1% surfacing — with completion rising, not falling. Prompt beats weights here, by a lot.
Task existence did not move: 7.5% → 6.3%, and the unit of optimisation reached 5.0%, the least-asked of six dimensions. The largest single-item reversal in the study is an existence item — EXI-03, 62.5% → 0.0%. Whatever improved between these generations, it wasn't the willingness to ask whether the task should be done.
What I'm least confident about
No human has checked any label. Every rate rests on two language models agreeing with each other about whether a third one named a framing choice. That is a real dependency and I don't want to soften it. A human-labelled subsample is the single most valuable thing this work could receive, and it does not exist.
The collapse rule moves the headline more than people expect. Same responses, same rubric:
Rule
Prior gen
Current gen
AND (what I report)
5.2%
23.3%
OR
15.8%
42.7%
Claude judge only
6.7%
26.0%
GPT judge only
14.4%
40.0%
I chose AND because it's conservative, but four defensible choices give four headlines. Anyone reporting a number off this benchmark should say which rule produced it.
A confound I cannot rule out. An ordering by "altitude" and an ordering by training-data prevalence predict the same six rates on this item set. The dimensions models question most may simply be the ones most discussed in text about project management. Nothing here separates the two, and I'd want an item set designed to break that tie before believing the altitude story.
The generations differ in provider-default inference behaviour as well as weights, so this compares what a user gets out of the box, not matched inference budgets. Items, rubric and framework were written by one person — me.
Items, the full rubric, both judge prompts verbatim and the Inspect task are all in the HF repo. Every number in the paper is computed by one statistics module and checked by a build gate that refuses any figure not resolving to it.
The 60 items are released for benchmark reuse. Please don't train on them — it destroys their value as an instrument for everyone, including you.
If anyone hand-labels even fifty judgements against the rubric, that's the most useful possible contribution and I'd like to hear about it.
Paper: 10.5281/zenodo.22847056 · Data, code and a one-command eval: 10.5281/zenodo.22847075 · HF dataset
Summary
The gap I was trying to measure
Existing premise-critique work — PCBench, RPCBench, (QA)² — uses premises that are false. A model can be right or wrong about them; detection is the task, and you can score it as detection.
The cases I care about are ones where nothing is false. "Plan the sequence of our 5G upgrades across markets to maximise net subscriber additions next year" is perfectly answerable. It also quietly fixes who counts as a beneficiary: subscribers who can be acquired. Low-density areas are outside the objective entirely, not by anyone's decision, just by the shape of the metric.
A model that plans the rollout has not made an error. There is nothing to detect. So the usual evaluation frame doesn't reach this at all, and I wanted to know whether the behaviour exists anyway.
Six dimensions: scope, horizon, unit of optimisation, metric, task existence, means. Ten items each, plus twenty neutral controls that hide nothing.
The scale, and the mistake I made first
Completion is recorded independently and never affects the level. Primary metric:
surfaced = level ∈ {A2,A3} AND completed == yes.Versions 1.0–2.0 of this rubric collapsed H1 and V1 into one intermediate level. That meant a model asking "who reads this and what do they do with it?" scored identically to one asking "what is your budget?" Both are questions, so both looked like engagement. They are not remotely the same behaviour, and the collapsed scale hid the only distinction the study was about. Splitting them is most of what v2.1 is.
Results
Unprompted surfacing rose 5.2% → 23.3% across generations. Paired +18.1 pp, 95% CI [+11.5, +25.4]; 33 of 60 items improved, 4 worsened, 23 unchanged; sign test p = 1.1e-6; cluster-robust logistic odds ratio 5.5.
Read that with its spread, or don't read it. The four frontier endpoints range from 12.5% to 43.3%. The single highest model supplies 46.4% of all positives; excluding it gives 16.7% rather than 23.3%. Whatever "the generation moved" means, it is a statement about a panel, not about a property of AI assistants. I'd treat any single pooled number of this kind — including mine — as a timestamp.
The obvious intervention fails. One arm's prompt explicitly invited critique. It reached A2/A3 52.7% of the time and completed the task 0% of the time. Framing engagement and task completion are separable, which means an evaluation that scores critique alone can rank interventions in the reverse of their practical usefulness. If you are building something that rewards models for questioning their instructions, this is the failure mode to watch: you can get a model that is excellent at noticing the frame and useless at the job.
One sentence beats a generation. Told to do the work and name one choice, the prior generation reached 53.1% surfacing — with completion rising, not falling. Prompt beats weights here, by a lot.
Task existence did not move: 7.5% → 6.3%, and the unit of optimisation reached 5.0%, the least-asked of six dimensions. The largest single-item reversal in the study is an existence item — EXI-03, 62.5% → 0.0%. Whatever improved between these generations, it wasn't the willingness to ask whether the task should be done.
What I'm least confident about
No human has checked any label. Every rate rests on two language models agreeing with each other about whether a third one named a framing choice. That is a real dependency and I don't want to soften it. A human-labelled subsample is the single most valuable thing this work could receive, and it does not exist.
The collapse rule moves the headline more than people expect. Same responses, same rubric:
I chose AND because it's conservative, but four defensible choices give four headlines. Anyone reporting a number off this benchmark should say which rule produced it.
A confound I cannot rule out. An ordering by "altitude" and an ordering by training-data prevalence predict the same six rates on this item set. The dimensions models question most may simply be the ones most discussed in text about project management. Nothing here separates the two, and I'd want an item set designed to break that tie before believing the altitude story.
The generations differ in provider-default inference behaviour as well as weights, so this compares what a user gets out of the box, not matched inference budgets. Items, rubric and framework were written by one person — me.
Run it against your own model
Items, the full rubric, both judge prompts verbatim and the Inspect task are all in the HF repo. Every number in the paper is computed by one statistics module and checked by a build gate that refuses any figure not resolving to it.
The 60 items are released for benchmark reuse. Please don't train on them — it destroys their value as an instrument for everyone, including you.
If anyone hand-labels even fifty judgements against the rubric, that's the most useful possible contribution and I'd like to hear about it.
Vijay Suresh, independent researcher. ORCID.