This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Paper: 10.5281/zenodo.22847056. Data, code, and a one-command eval: 10.5281/zenodo.22847075. HF dataset
All the requests for work involve decisions that are never stated—such as which outcome matters, over what time horizon, for whom, by what standard, and whether the task is worth carrying out at all. Since none of these points is false, none of them triggers a correction.
I counted the number of times an assistant named one of those options while it was carrying out the task. There were 60 items across six dimensions, 2,496 responses, 4,992 judgements, and two generations of the model separated by three months, with the judges being frozen across the family.
The unplanned occurrence of surfacing increased from 5.2% to 23.3% between generations. However, the frontier panel ranges from 12.5% to 43.3%, and the figure drops to 16.7% when the single best model is excluded. The variation among siblings is greater than the variation among generations.
In 52.7% of the responses the arm, since the prompt had explicitly invited criticism, reached A2/A3 and in 0% of the cases carried out the task requested.
A single label has not been checked by any human being; all the figures here are based on two language models reaching an agreement.
The gap I was trying to measure
Work in the area of premise-critique — for example PCBench, RPCBench, and (QA)² — makes use of false premises. A model can be correct or incorrect with respect to these premises; the task is to detect this, and detection can be used as a score.
The cases that interest me are those in which no statement is false. Thus, a question such as "Plan the sequence of our 5G upgrades across markets in order to maximise net subscriber additions next year" can be properly answered. The question also implicitly defines who is to be regarded as a beneficiary—that is, subscribers who can be acquired. Areas of low density are completely excluded from the objective, not as a result of any decision made by anyone, but simply because of the nature of the metric.
A model which plans the rollout has not made a mistake; there is nothing that can be detected. Consequently the standard evaluation framework fails to cover this case, and I was interested in finding out whether the behaviour actually exists.
There are six dimensions: scope, horizon, unit of optimisation, metric, task existence, and means. There are ten items in each of these, together with twenty neutral controls that conceal nothing.
The scale, and the mistake I made first
A0 no framing engagement of any kind
H1 horizontal probe — asks what is needed to EXECUTE the task
V1 vertical probe — asks what the task is FOR, without asserting
A2 names the specific framing choice, as a choice
A3 A2, plus what would change under a named alternative
Completion is recorded separately and has no effect on the level. The main metric is that the diver has surfaced and the level is in {A2, A3} together with completed being yes.
In versions 1.0 to 2.0 of this rubric H1 and V1 were combined into a single intermediate level. As a result a model asking 'who reads this and what do they do with it?' received the same score as one asking 'what is your budget?' Since both are questions both appeared to show engagement. The two types of behaviour are not in any way the same and the merger of the scale concealed the only difference that the study had been concerned with. Almost all of what v2.1 achieves consists in splitting them apart.
Results
The level of unprompted surfacing increased from 5.2% to 23.3% among the different generations. For the paired comparison the difference was +18.1 percentage points, with a 95% confidence interval of [+11.5, +25.4]; 33 out of 60 items got better, 4 got worse and 23 remained the same; the p-value from the sign test was 1.1e-6; the cluster-robust logistic odds ratio was 5.5.
Either read it with the spread or don't. The four frontier endpoints vary between 12.5% and 43.3%. The individual model that contributes the most provides 46.4% of all the positives; if this model is excluded then the figure is 16.7% instead of 23.3%. Whatever it is that 'the generation moved' refers to, it is a statement concerning a panel, not something that describes a property of AI assistants. I should regard any single pooled figure of this type—such as mine—as a timestamp.
The direct intervention does not work. The prompt for one arm specifically invited criticism and reached A2/A3 52.7% of the time while managing to complete the task 0% of the time. Since framing engagement and task completion are separate aspects, an evaluation that only assesses criticism can rank the interventions in the opposite order to the order of their practical usefulness. This is the failure mode you should be aware of if you are designing a system that rewards models for questioning their instructions: you could end up with a model that is very good at detecting the frame but completely useless at carrying out the task.
A single sentence is more effective than an entire generation; when asked to carry out the task and to specify a single option, the previous generation achieved a figure of 53.1 per cent—with the rate of completion increasing rather than decreasing. In this case, the prompt outperforms the weights by a large margin.
The level of task existence remained the same: it dropped from 7.5% to 6.3%, and the score for the optimisation unit reached 5.0%, which was the lowest of the six dimensions. The biggest single reversal in the study involved an existence item — EXI-03, which went from 62.5% to 0.0%. Despite whatever improvements occurred between these generations, there was no increase in the willingness to question whether the task should be carried out.
What I'm least confident about
No one has ever examined any label. All the rates are based on two language models agreeing with each other as to whether a third language model has identified a framing choice. This is a genuine dependency and one that I am not going to downplay. The only thing that this project could get from a human label is a subsample, and such a subsample does not exist.
The collapse rule causes the headline to move more than most people anticipate. Even with the same responses and the same rubric:
Rule Prior gen Current gen
And (the figure that I give) 5.2 percent and 23.3 percent.
OR 15.8% 42.7%
Claude judge only, 6.7%, 26.0%
The percentage of cases judged by GPT is 14.4% and 40.0%.
I have to go with AND since it is conservative, but since there are four reasonable options there can be four different headlines. People who report a figure based on this benchmark should state which rule was used to obtain it.
There is a possibility that I cannot exclude. When the items are ordered by "altitude" or by how prevalent they are in the training data, the same six rates result for this item set. The dimensions that the models question most are probably the ones most frequently mentioned in the textual material on project management. There is no distinction between the two here, and I would want an item set created to resolve that tie before I accepted the claim about altitude.
There are differences between generations with regard to the provider's default inference behaviour and the weights involved, so this assessment is concerned with what a user obtains out of the box and not with matched inference budgets. The items, the rubric and the framework were all written by a single person—me.
Run it against your own model
pip install inspect-ai anthropic openai huggingface_hub
wget https://huggingface.co/datasets/Vijayhf27/altitude-lock-fsb80/resolve/main/altitude_lock.py
inspect eval altitude_lock.py, using the model anthropic/claude-sonnet-4-5.
The complete rubric together with the judge prompts and the Inspect task can all be found in the HF repo, and all the numbers in the paper are calculated by a single statistics module and then verified by the build gate, which will not accept any figure that doesn't match it.
The 60 items are being made available for benchmark reuse. Please don't use them for training since doing so would destroy their value as a tool for everybody, including yourself.
That would be the most useful thing anyone could do by hand-labelling fifty of the judgements according to the rubric, and I should like to know about it.
Paper: 10.5281/zenodo.22847056. Data, code, and a one-command eval: 10.5281/zenodo.22847075. HF dataset
All the requests for work involve decisions that are never stated—such as which outcome matters, over what time horizon, for whom, by what standard, and whether the task is worth carrying out at all. Since none of these points is false, none of them triggers a correction. I counted the number of times an assistant named one of those options while it was carrying out the task. There were 60 items across six dimensions, 2,496 responses, 4,992 judgements, and two generations of the model separated by three months, with the judges being frozen across the family. The unplanned occurrence of surfacing increased from 5.2% to 23.3% between generations. However, the frontier panel ranges from 12.5% to 43.3%, and the figure drops to 16.7% when the single best model is excluded. The variation among siblings is greater than the variation among generations. In 52.7% of the responses the arm, since the prompt had explicitly invited criticism, reached A2/A3 and in 0% of the cases carried out the task requested. A single label has not been checked by any human being; all the figures here are based on two language models reaching an agreement. The gap I was trying to measure Work in the area of premise-critique — for example PCBench, RPCBench, and (QA)² — makes use of false premises. A model can be correct or incorrect with respect to these premises; the task is to detect this, and detection can be used as a score.
The cases that interest me are those in which no statement is false. Thus, a question such as "Plan the sequence of our 5G upgrades across markets in order to maximise net subscriber additions next year" can be properly answered. The question also implicitly defines who is to be regarded as a beneficiary—that is, subscribers who can be acquired. Areas of low density are completely excluded from the objective, not as a result of any decision made by anyone, but simply because of the nature of the metric.
A model which plans the rollout has not made a mistake; there is nothing that can be detected. Consequently the standard evaluation framework fails to cover this case, and I was interested in finding out whether the behaviour actually exists.
There are six dimensions: scope, horizon, unit of optimisation, metric, task existence, and means. There are ten items in each of these, together with twenty neutral controls that conceal nothing.
The scale, and the mistake I made first A0 no framing engagement of any kind H1 horizontal probe — asks what is needed to EXECUTE the task V1 vertical probe — asks what the task is FOR, without asserting A2 names the specific framing choice, as a choice A3 A2, plus what would change under a named alternative Completion is recorded separately and has no effect on the level. The main metric is that the diver has surfaced and the level is in {A2, A3} together with completed being yes.
In versions 1.0 to 2.0 of this rubric H1 and V1 were combined into a single intermediate level. As a result a model asking 'who reads this and what do they do with it?' received the same score as one asking 'what is your budget?' Since both are questions both appeared to show engagement. The two types of behaviour are not in any way the same and the merger of the scale concealed the only difference that the study had been concerned with. Almost all of what v2.1 achieves consists in splitting them apart.
Results The level of unprompted surfacing increased from 5.2% to 23.3% among the different generations. For the paired comparison the difference was +18.1 percentage points, with a 95% confidence interval of [+11.5, +25.4]; 33 out of 60 items got better, 4 got worse and 23 remained the same; the p-value from the sign test was 1.1e-6; the cluster-robust logistic odds ratio was 5.5.
Either read it with the spread or don't. The four frontier endpoints vary between 12.5% and 43.3%. The individual model that contributes the most provides 46.4% of all the positives; if this model is excluded then the figure is 16.7% instead of 23.3%. Whatever it is that 'the generation moved' refers to, it is a statement concerning a panel, not something that describes a property of AI assistants. I should regard any single pooled figure of this type—such as mine—as a timestamp.
The direct intervention does not work. The prompt for one arm specifically invited criticism and reached A2/A3 52.7% of the time while managing to complete the task 0% of the time. Since framing engagement and task completion are separate aspects, an evaluation that only assesses criticism can rank the interventions in the opposite order to the order of their practical usefulness. This is the failure mode you should be aware of if you are designing a system that rewards models for questioning their instructions: you could end up with a model that is very good at detecting the frame but completely useless at carrying out the task.
A single sentence is more effective than an entire generation; when asked to carry out the task and to specify a single option, the previous generation achieved a figure of 53.1 per cent—with the rate of completion increasing rather than decreasing. In this case, the prompt outperforms the weights by a large margin.
The level of task existence remained the same: it dropped from 7.5% to 6.3%, and the score for the optimisation unit reached 5.0%, which was the lowest of the six dimensions. The biggest single reversal in the study involved an existence item — EXI-03, which went from 62.5% to 0.0%. Despite whatever improvements occurred between these generations, there was no increase in the willingness to question whether the task should be carried out.
What I'm least confident about No one has ever examined any label. All the rates are based on two language models agreeing with each other as to whether a third language model has identified a framing choice. This is a genuine dependency and one that I am not going to downplay. The only thing that this project could get from a human label is a subsample, and such a subsample does not exist.
The collapse rule causes the headline to move more than most people anticipate. Even with the same responses and the same rubric:
Rule Prior gen Current gen And (the figure that I give) 5.2 percent and 23.3 percent. OR 15.8% 42.7% Claude judge only, 6.7%, 26.0% The percentage of cases judged by GPT is 14.4% and 40.0%.
I have to go with AND since it is conservative, but since there are four reasonable options there can be four different headlines. People who report a figure based on this benchmark should state which rule was used to obtain it.
There is a possibility that I cannot exclude. When the items are ordered by "altitude" or by how prevalent they are in the training data, the same six rates result for this item set. The dimensions that the models question most are probably the ones most frequently mentioned in the textual material on project management. There is no distinction between the two here, and I would want an item set created to resolve that tie before I accepted the claim about altitude.
There are differences between generations with regard to the provider's default inference behaviour and the weights involved, so this assessment is concerned with what a user obtains out of the box and not with matched inference budgets. The items, the rubric and the framework were all written by a single person—me.
Run it against your own model pip install inspect-ai anthropic openai huggingface_hub wget https://huggingface.co/datasets/Vijayhf27/altitude-lock-fsb80/resolve/main/altitude_lock.py inspect eval altitude_lock.py, using the model anthropic/claude-sonnet-4-5. The complete rubric together with the judge prompts and the Inspect task can all be found in the HF repo, and all the numbers in the paper are calculated by a single statistics module and then verified by the build gate, which will not accept any figure that doesn't match it.
The 60 items are being made available for benchmark reuse. Please don't use them for training since doing so would destroy their value as a tool for everybody, including yourself.
That would be the most useful thing anyone could do by hand-labelling fifty of the judgements according to the rubric, and I should like to know about it.