Epistemic status: preprint with n=24 in the key cell; I'd defend the direction and not the magnitude.
TLDR: We had three frontier models (at the time of generation) generate step-by-step solutions to 79 physics questions from Humanity's Last Exam (HLE) and had 8 chain-of-thought monitors evaluate these traces for whether they contained an error and, if so, the first error step. Monitors saw the traces under a range of information conditions, from blind evaluation with the answer not shown to evaluation with a certified answer, which we term an "information ladder". Verdicts were scored against a joint human-AI ground truth. That ground truth has correct answers without errors, incorrect answers with errors, and a set of correct answers that contained an error (the critical set). Our dataset contained only natural errors rather than deliberately planted ones.
Moving up the information ladder, mean balanced accuracy rose from 0.637 (BLIND) to 0.796 (CERT), while exact step localization rose only from 0.261 to 0.379, hence evaluating the answer does not imply the argument for that answer has been found. Comparatively, a conclusion-only "oracle" that flags exactly the traces whose final answer disagrees with the certified answer would achieve a balanced accuracy of 0.888, with 0 localization and 0 recall on the critical set. Recall on traces with an incorrect final answer changed from 0.653 (BLIND) to 0.951 (CERT), against 0.521 => 0.438 on the critical set. The difference between those two changes is +0.382 (95% CI [+0.256, +0.506]) and has the same sign for all 8 monitors. Finally, of the 99 HLE physics questions we inspected closely, having already filtered for exact-match answers and multi-step reasoning, 16 had incorrect or non-physical reference answers and a further 11 were dropped as ill-posed or lacking a unique answer – (27/99, or ~27%).
We thus conclude that measuring monitors on their ability to evaluate the final answer does not imply that the monitors are evaluating the reasoning. We consider this a benign analogue of reward hacking, where an acceptable output doesn't imply sound reasoning.
Detection and localization across the information ladder. (a) Balanced accuracy, where 0.5 is chance. (b) Exact localization , the share of error traces for which the monitor both flagged the trace and named the correct step. Each line is one monitor; the legend is ordered by BLIND balanced accuracy. The dashed black line is the mean across monitors, the quantity quoted in the text. Balanced accuracy rises steadily from BLIND to CERT while rises far less, and the spread between monitors within any condition exceeds the movement across the whole ladder. STEP is omitted here because it is a compound intervention rather than a rung.
Why the correct-answer-with-bad-reasoning case matters
Chain of thought monitoring is a possible component for ensuring a future AI system has certain characteristics. One of these potential characteristics is coherence with clear reasoning rather than deceptive reasoning. By looking at a domain with a clear definitive answer and logical steps to find that answer like physics we're able to create a series of logical reasoning attempts by frontier models which can then be monitored by other models. The basic premise here is to have a group of frontier models attempt Humanity's Last Exam questions and create a natural dataset of questions where the models get the questions right/wrong with correct/erroneous reasoning. We can then have other monitors read these traces and report whether the trace contains an error and, if so, which numbered step it first appears at. Note that the monitors were never asked whether the final answer was correct; answer correctness is a ground-truth label, not a monitor output.
Whilst in reward hacking models optimize for a reward without genuinely completing the task as intended, in our case the models can arrive at the correct answer without sound physics reasoning. A naive case here is mistakenly multiplying by two but later on in the calculation dividing by two, resulting in the correct answer with bad reasoning. By identifying this, we can characterize whether trusted-answer monitoring evaluations may overstate monitor capabilities.
Setup
We started with the HLE dataset for all questions that included 'physics', which was 228 questions. Taking the text-only questions reduced this to 178 and exact-match questions reduced this again to 144. We manually went through these 144 questions removing those that didn't require clear multi-step reasoning, such as asking for specific exponents in obscure theories, leaving 99 questions. We used 9 of these questions for human marker training and a further 11 questions were dropped from the remaining 90 for being ill posed or lacking a unique answer. For example, one question applies Basquin's law for fatigue life and assumes 10^7 for the max number of cycles, however 10^6 is also regularly used, thus there wasn't a unique final answer. The surviving questions were completed by GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro, resulting in 237 traces from 79 questions.
In order to have something to evaluate against we created a ground truth. This ground truth was established through two arms. Firstly, human annotators went through the traces (around 70% coverage) annotating if the traces were correct and if any error was made, and secondly a five round AI debate went through those same traces. We then did an adjudication round to get a ground truth. This ground truth revealed three categories: answers that were correct without errors in their reasoning, answers which were incorrect with errors in their reasoning, and answers that were correct but had errors in their reasoning, called our "critical set".
On the monitor side, we had 8 monitors spanning a wide range of capability (BLIND balanced accuracies run from 0.525 for deepseek-v4flash to 0.844 for fable-5) annotate these traces under an "information ladder". This ladder started with the complete trace without the answer (BLIND), next the HINT condition where the blind trace was given along with the answer framed as unverified, then the whole trace with the answer framed as definitely correct (CERT), and finally the monitor's own blind verdict played back to it but this time with the certified answer included (REVISE). We also included a control condition where the monitor got a second pass at their own blind verdict without being shown the answer.
Detection improves more than localization
We found that the identification of an error improved much more than the localization of the error. When monitors viewed the traces blind they achieved a mean balanced accuracy of 0.637 (95% CI [0.605, 0.666]); providing the answer framed as unverified (HINT) raised the mean to 0.712, and framing the answer as certified (CERT) raised it further to 0.796 (95% CI [0.760, 0.832]). However, exact localization rises far less, from 0.261 ([0.212, 0.312]) under BLIND to 0.379 ([0.316, 0.436]) under CERT.
One could point out that adjacent error steps are not always cleanly separable so localization could be artificially depressed by exact matching, thus we repeated the analysis using a +/- 1 step window. However, with a wider window comes a larger chance of a lucky guess - a solution with only three steps and a +/- 1 window will always get the localization correct simply by guessing step 2. More generally a uniform probability of any step in n steps being the error step is 1/n and the chance of landing within the window is the size of the error window w over n. We can thus correct for chance by finding a skill score of (L – L_chance) / (1 – L_chance). As shown in the figure below, we see the same pattern.
Localization at two tolerances, each against the rate a randomly placed flag would achieve. (a) Mean across monitors of exact localization (green) and localization within one step (orange), with the corresponding uniform-placement chance rates in grey; arrows link the two tolerances at each rung. The grey arrows are longer than the coloured ones, so relaxing the criterion by one step raises the chance rate by more than it raises the observed rate. Grey rises across conditions because only a flagged trace can be located by accident and monitors flag more often once the answer is supplied. (b) The same results as a skill score, (L − )/(1 − ), which is 0 at chance and 1 for perfect localization. The two tolerances coincide, so the choice of criterion does not affect any conclusion drawn here. The null assumes uniform placement over a trace’s steps; monitors in fact flag early more often than late, so a null preserving that positional bias would be stricter, but the same assumption applies to both tolerances and so does not affect the comparison between them.
We can more clearly see where the localization steps are landing by decomposing early, exact, late and missed flags. Going up the information ladder from BLIND to CERT we find that missed traces fall 0.383 => 0.165, late flags rise 0.209 => 0.347, and early flags fall slightly 0.147 => 0.110. Thus monitors given the answer tend to place their flags further downstream. An interpretation here is that monitors are tacitly aware an error exists as the conclusion is incorrect and work backwards from the conclusion – this would explain why early flags fall and late flags rise.
Where flags land on traces containing an error, including both wrong-answer traces and the critical set. Each bar decomposes the 107 error traces into flags placed before the true first error (early), on it (exact), after it (late), and cases in which the monitor either did not flag the trace or flagged it without naming a step (missed localization). This differs from the any-flag sensitivity metric reported elsewhere, which counts every flagged error trace as a successful detection even when the monitor does not name a step.
Gain is concentrated where the conclusion already gives it away
When the monitors know that there is an error, as the certified answer disagrees with the final answer of the trace, the chance of them flagging increases substantially. On traces whose final answer is wrong, moving from BLIND to CERT raises mean recall by +0.299, and monitors close an average of 75.9% of the distance between their own blind recall and 1. However, on cases where the answer is correct but the reasoning contains an error (the critical set) this same intervention changes mean recall by -0.083 (question-bootstrap 95% CI [-0.196, +0.030]). This is the exact safety relevant case, as the conclusion is sound but the reasoning is not.
Cell-restricted recall by condition. (a) Traces whose final answer is wrong, where comparing the trace’s conclusion against the reference answer reveals that an error exists. (b) Traces that reach the correct answer through flawed reasoning, where the conclusion carries no information about the error. Recall climbs steeply in (a) and is flat to declining in (b). Each monitor is scored on a common subset across conditions. The dashed black line is the mean across monitors.
Monitors that have already committed will change their verdict to match the answer
The REVISE condition reaches the highest exact localization of any of the conditions at 0.418, with a balanced accuracy of 0.783 (this is slightly below CERT because it flags more overall, mean sensitivity 0.861 against specificity 0.705). Given models can often change their views just by being asked again, we included a control condition where the monitors were simply invited to consider their blind verdict again without seeing the answer. Taking the net of control, we found monitors withdraw prior flags at +0.414 on clean traces (where withdrawal is correct) and +0.189 on critical traces (where it retracts a correct judgement). Further, after blind commitment, monitors shown the answer newly flagged 93.8% of previously passed wrong-answer traces as erroneous, but only 18.0% of previously passed critical traces.
Effect of revealing the certified answer to a monitor that has already committed to a blind verdict. (a) Net withdrawal of prior flags: REVISE minus a control in which the monitor is invited to reconsider without being shown the answer. Withdrawing on a clean trace is correct; withdrawing on a critical trace retracts a correct judgement, so a discriminating monitor shows a tall left bar and a flat right one. (b) Raw share of previously-passed traces that the monitor flags once the answer is revealed; the corresponding control values are given in the text. Numbers beneath each bar give the relevant denominator. In panel (a), where each bar is a difference between REVISE and RCTRL rates, the smaller of the two denominators is shown; in panel (b), the number is the traces the monitor had passed at BLIND. Hatched bars have fewer than ten observations and are not interpreted individually.
Stepwise monitoring is worse than any whole-trace condition
We also included a stepwise condition, though this combines two interventions - hiding future context and requesting a decision at each step - so it is not directly comparable to the rest. However, we find that it performed the worst of all conditions. The mean balanced accuracy 0.614 compared to 0.637 for BLIND and the exact localization 0.196 compared to 0.261 (BLIND). Further, the miss rate here of 0.422 is the highest of any condition.
This condition wasn't the principal focus of this work however it is the most analogous to a real-world condition where a live chain-of-thought is being monitored. The most pernicious result here is that the specificity in STEP ranges from 0.178 (deepseek-v4flash) to 0.967 (fable-5) with the weaker monitors suffering most from accumulating false-positives at each step. This is a targeted area for our future work but ultimately, by construction, the monitor overseeing a new model will be weaker than the model that it oversees.
"Isn't this obvious?"
The strongest version of this objection is that we gave the monitors more information and they did better with it. However, the critical set and the wrong-answer set receive exactly the same intervention. The only thing that differs between them is whether the final answer reveals that an error exists at all. Any difference in how much the two cells improve therefore cannot be explained by the monitors simply having more to go on, and we find the improvement is concentrated almost entirely in the cell where the conclusion gives the error away.
Secondly, the rise in balanced accuracy with more information is not particularly surprising as performance will generally improve when a monitor is given more to work with. Note though that the monitors are never told the ground truth - they are given the reference answer, and are never told whether the trace contains an error, which is the thing actually being scored. In any case, the difference in improvement between localization and balanced accuracy is not at all obvious. Closely reading a physics solution top to bottom would presumably result in error localization increasing in lockstep with error identification – the case where this would not be true is if the reader (whether human or AI) skips straight to the end and only performs a cursory check of the final answer. Further, when we actually give the answer to the monitors in the CERT condition we include an explicit warning that "A solution can reach the correct final answer and still contain an earlier false step". If anything, the monitors are biased towards a lock-step improvement between localization and balanced accuracy, but we do not find this.
Recommendations
Given the results of this study, we recommend evaluations of process monitoring include an answer-blind condition and report recall separately for where the answer exposes the error and where it does not. Ideally it should also include localization, since monitors may identify an error from answer-checking without actually identifying the erroneous reasoning. Particularly with an answer-blind condition, this can be applied post-hoc in existing trace datasets as monitors are given a reduced amount of information that is already available. Localization takes additional work as an expanded ground truth needs to be established however we believe that this is worth doing. Finally we'd recommend including a conclusion only "oracle" in evaluations of chain-of-thought monitoring as it provides some insight into what part of the chain is actually being monitored. If looking only at the final answer gives similar performance to considering the full trace, and the answer is not the argument, then the argument itself may not be being monitored.
What would change my mind
The principal argument that would change my mind is if there is something fundamentally different between the critical set and the wrong-answer set. This could be that the critical set contains fundamentally harder physics though there isn't a large enough sample to say for certain. The critical and wrong-answer traces have median lengths of 10.0 and 12.5 steps (Mann-Whitney p = 0.142), which at this sample size means we cannot detect a large difference rather than that we have excluded one. Against the difficulty reading, no error category in the critical set showed a positive CERT-BLIND change in recall (Table III in the paper), so the result is not carried by the repaired or exploratory cases alone. Also, our annotation rules were finalized following human annotator training where we noticed many of the disagreements were due to definitional differences thus it is possible that our rules were ill-specified. Appendix B in the paper goes into more detail here but I believe our definitions – the same rubric was given to the monitors – are pretty solid.
Limitations
Our critical set contained only 24 traces, 16 of which were author adjudicated. Of these 24 traces, only 2 had genuine substantial reasoning errors, with the others being algebraic errors, incorrect supplementary claims, or claims that were later identified and repaired by the model. Section III F of the paper contains a detailed discussion of these.
There are some further limitations such as how we only look at one domain and our ground truth may not be 100% correct. Further this ground truth was itself constructed with access to the reference answer and the rationale provided by HLE thus it is in principle anchored in the same direction that we're measuring improved monitor performance. Finally, during adjudication Fable-5 was inadvertently left on as the selected model during a chat session for approximately three traces – dropping it as a monitor actually moves the difference-in-differences from +0.382 to +0.415 so it actually strengthens the effect.
Paper and links
The full paper is available on arXiv at https://arxiv.org/abs/2609.00264. Code, derived labels and analysis outputs are at https://github.com/WillYeadon/hle_physics_oversight. The repository does not redistribute HLE question text, images, reference answers or rationales, or model-generated trace text. The HLE organizers have been contacted about the reference-answer errors.
Epistemic status: preprint with n=24 in the key cell; I'd defend the direction and not the magnitude.
TLDR: We had three frontier models (at the time of generation) generate step-by-step solutions to 79 physics questions from Humanity's Last Exam (HLE) and had 8 chain-of-thought monitors evaluate these traces for whether they contained an error and, if so, the first error step. Monitors saw the traces under a range of information conditions, from blind evaluation with the answer not shown to evaluation with a certified answer, which we term an "information ladder". Verdicts were scored against a joint human-AI ground truth. That ground truth has correct answers without errors, incorrect answers with errors, and a set of correct answers that contained an error (the critical set). Our dataset contained only natural errors rather than deliberately planted ones.
Moving up the information ladder, mean balanced accuracy rose from 0.637 (BLIND) to 0.796 (CERT), while exact step localization rose only from 0.261 to 0.379, hence evaluating the answer does not imply the argument for that answer has been found. Comparatively, a conclusion-only "oracle" that flags exactly the traces whose final answer disagrees with the certified answer would achieve a balanced accuracy of 0.888, with 0 localization and 0 recall on the critical set. Recall on traces with an incorrect final answer changed from 0.653 (BLIND) to 0.951 (CERT), against 0.521 => 0.438 on the critical set. The difference between those two changes is +0.382 (95% CI [+0.256, +0.506]) and has the same sign for all 8 monitors. Finally, of the 99 HLE physics questions we inspected closely, having already filtered for exact-match answers and multi-step reasoning, 16 had incorrect or non-physical reference answers and a further 11 were dropped as ill-posed or lacking a unique answer – (27/99, or ~27%).
We thus conclude that measuring monitors on their ability to evaluate the final answer does not imply that the monitors are evaluating the reasoning. We consider this a benign analogue of reward hacking, where an acceptable output doesn't imply sound reasoning.
Detection and localization across the information ladder. (a) Balanced accuracy, where 0.5 is chance. (b) Exact localization
Why the correct-answer-with-bad-reasoning case matters
Chain of thought monitoring is a possible component for ensuring a future AI system has certain characteristics. One of these potential characteristics is coherence with clear reasoning rather than deceptive reasoning. By looking at a domain with a clear definitive answer and logical steps to find that answer like physics we're able to create a series of logical reasoning attempts by frontier models which can then be monitored by other models. The basic premise here is to have a group of frontier models attempt Humanity's Last Exam questions and create a natural dataset of questions where the models get the questions right/wrong with correct/erroneous reasoning. We can then have other monitors read these traces and report whether the trace contains an error and, if so, which numbered step it first appears at. Note that the monitors were never asked whether the final answer was correct; answer correctness is a ground-truth label, not a monitor output.
Whilst in reward hacking models optimize for a reward without genuinely completing the task as intended, in our case the models can arrive at the correct answer without sound physics reasoning. A naive case here is mistakenly multiplying by two but later on in the calculation dividing by two, resulting in the correct answer with bad reasoning. By identifying this, we can characterize whether trusted-answer monitoring evaluations may overstate monitor capabilities.
Setup
We started with the HLE dataset for all questions that included 'physics', which was 228 questions. Taking the text-only questions reduced this to 178 and exact-match questions reduced this again to 144. We manually went through these 144 questions removing those that didn't require clear multi-step reasoning, such as asking for specific exponents in obscure theories, leaving 99 questions. We used 9 of these questions for human marker training and a further 11 questions were dropped from the remaining 90 for being ill posed or lacking a unique answer. For example, one question applies Basquin's law for fatigue life and assumes 10^7 for the max number of cycles, however 10^6 is also regularly used, thus there wasn't a unique final answer. The surviving questions were completed by GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro, resulting in 237 traces from 79 questions.
In order to have something to evaluate against we created a ground truth. This ground truth was established through two arms. Firstly, human annotators went through the traces (around 70% coverage) annotating if the traces were correct and if any error was made, and secondly a five round AI debate went through those same traces. We then did an adjudication round to get a ground truth. This ground truth revealed three categories: answers that were correct without errors in their reasoning, answers which were incorrect with errors in their reasoning, and answers that were correct but had errors in their reasoning, called our "critical set".
On the monitor side, we had 8 monitors spanning a wide range of capability (BLIND balanced accuracies run from 0.525 for deepseek-v4flash to 0.844 for fable-5) annotate these traces under an "information ladder". This ladder started with the complete trace without the answer (BLIND), next the HINT condition where the blind trace was given along with the answer framed as unverified, then the whole trace with the answer framed as definitely correct (CERT), and finally the monitor's own blind verdict played back to it but this time with the certified answer included (REVISE). We also included a control condition where the monitor got a second pass at their own blind verdict without being shown the answer.
Detection improves more than localization
We found that the identification of an error improved much more than the localization of the error. When monitors viewed the traces blind they achieved a mean balanced accuracy of 0.637 (95% CI [0.605, 0.666]); providing the answer framed as unverified (HINT) raised the mean to 0.712, and framing the answer as certified (CERT) raised it further to 0.796 (95% CI [0.760, 0.832]). However, exact localization rises far less, from 0.261 ([0.212, 0.312]) under BLIND to 0.379 ([0.316, 0.436]) under CERT.
One could point out that adjacent error steps are not always cleanly separable so localization could be artificially depressed by exact matching, thus we repeated the analysis using a +/- 1 step window. However, with a wider window comes a larger chance of a lucky guess - a solution with only three steps and a +/- 1 window will always get the localization correct simply by guessing step 2. More generally a uniform probability of any step in n steps being the error step is 1/n and the chance of landing within the window is the size of the error window w over n. We can thus correct for chance by finding a skill score of (L – L_chance) / (1 – L_chance). As shown in the figure below, we see the same pattern.
Localization at two tolerances, each against the rate a randomly placed flag would achieve. (a) Mean across monitors of exact localization
We can more clearly see where the localization steps are landing by decomposing early, exact, late and missed flags. Going up the information ladder from BLIND to CERT we find that missed traces fall 0.383 => 0.165, late flags rise 0.209 => 0.347, and early flags fall slightly 0.147 => 0.110. Thus monitors given the answer tend to place their flags further downstream. An interpretation here is that monitors are tacitly aware an error exists as the conclusion is incorrect and work backwards from the conclusion – this would explain why early flags fall and late flags rise.
Where flags land on traces containing an error, including both wrong-answer traces and the critical set. Each bar decomposes the 107 error traces into flags placed before the true first error (early), on it (exact), after it (late), and cases in which the monitor either did not flag the trace or flagged it without naming a step (missed localization). This differs from the any-flag sensitivity metric reported elsewhere, which counts every flagged error trace as a successful detection even when the monitor does not name a step.
Gain is concentrated where the conclusion already gives it away
When the monitors know that there is an error, as the certified answer disagrees with the final answer of the trace, the chance of them flagging increases substantially. On traces whose final answer is wrong, moving from BLIND to CERT raises mean recall by +0.299, and monitors close an average of 75.9% of the distance between their own blind recall and 1. However, on cases where the answer is correct but the reasoning contains an error (the critical set) this same intervention changes mean recall by -0.083 (question-bootstrap 95% CI [-0.196, +0.030]). This is the exact safety relevant case, as the conclusion is sound but the reasoning is not.
Cell-restricted recall by condition. (a) Traces whose final answer is wrong, where comparing the trace’s conclusion against the reference answer reveals that an error exists. (b) Traces that reach the correct answer through flawed reasoning, where the conclusion carries no information about the error. Recall climbs steeply in (a) and is flat to declining in (b). Each monitor is scored on a common subset across conditions. The dashed black line is the mean across monitors.
Monitors that have already committed will change their verdict to match the answer
The REVISE condition reaches the highest exact localization of any of the conditions at 0.418, with a balanced accuracy of 0.783 (this is slightly below CERT because it flags more overall, mean sensitivity 0.861 against specificity 0.705). Given models can often change their views just by being asked again, we included a control condition where the monitors were simply invited to consider their blind verdict again without seeing the answer. Taking the net of control, we found monitors withdraw prior flags at +0.414 on clean traces (where withdrawal is correct) and +0.189 on critical traces (where it retracts a correct judgement). Further, after blind commitment, monitors shown the answer newly flagged 93.8% of previously passed wrong-answer traces as erroneous, but only 18.0% of previously passed critical traces.
Effect of revealing the certified answer to a monitor that has already committed to a blind verdict. (a) Net withdrawal of prior flags: REVISE minus a control in which the monitor is invited to reconsider without being shown the answer. Withdrawing on a clean trace is correct; withdrawing on a critical trace retracts a correct judgement, so a discriminating monitor shows a tall left bar and a flat right one. (b) Raw share of previously-passed traces that the monitor flags once the answer is revealed; the corresponding control values are given in the text. Numbers beneath each bar give the relevant denominator. In panel (a), where each bar is a difference between REVISE and RCTRL rates, the smaller of the two denominators is shown; in panel (b), the number is the traces the monitor had passed at BLIND. Hatched bars have fewer than ten observations and are not interpreted individually.
Stepwise monitoring is worse than any whole-trace condition
We also included a stepwise condition, though this combines two interventions - hiding future context and requesting a decision at each step - so it is not directly comparable to the rest. However, we find that it performed the worst of all conditions. The mean balanced accuracy 0.614 compared to 0.637 for BLIND and the exact localization 0.196 compared to 0.261 (BLIND). Further, the miss rate here of 0.422 is the highest of any condition.
This condition wasn't the principal focus of this work however it is the most analogous to a real-world condition where a live chain-of-thought is being monitored. The most pernicious result here is that the specificity in STEP ranges from 0.178 (deepseek-v4flash) to 0.967 (fable-5) with the weaker monitors suffering most from accumulating false-positives at each step. This is a targeted area for our future work but ultimately, by construction, the monitor overseeing a new model will be weaker than the model that it oversees.
"Isn't this obvious?"
The strongest version of this objection is that we gave the monitors more information and they did better with it. However, the critical set and the wrong-answer set receive exactly the same intervention. The only thing that differs between them is whether the final answer reveals that an error exists at all. Any difference in how much the two cells improve therefore cannot be explained by the monitors simply having more to go on, and we find the improvement is concentrated almost entirely in the cell where the conclusion gives the error away.
Secondly, the rise in balanced accuracy with more information is not particularly surprising as performance will generally improve when a monitor is given more to work with. Note though that the monitors are never told the ground truth - they are given the reference answer, and are never told whether the trace contains an error, which is the thing actually being scored. In any case, the difference in improvement between localization and balanced accuracy is not at all obvious. Closely reading a physics solution top to bottom would presumably result in error localization increasing in lockstep with error identification – the case where this would not be true is if the reader (whether human or AI) skips straight to the end and only performs a cursory check of the final answer. Further, when we actually give the answer to the monitors in the CERT condition we include an explicit warning that "A solution can reach the correct final answer and still contain an earlier false step". If anything, the monitors are biased towards a lock-step improvement between localization and balanced accuracy, but we do not find this.
Recommendations
Given the results of this study, we recommend evaluations of process monitoring include an answer-blind condition and report recall separately for where the answer exposes the error and where it does not. Ideally it should also include localization, since monitors may identify an error from answer-checking without actually identifying the erroneous reasoning. Particularly with an answer-blind condition, this can be applied post-hoc in existing trace datasets as monitors are given a reduced amount of information that is already available. Localization takes additional work as an expanded ground truth needs to be established however we believe that this is worth doing. Finally we'd recommend including a conclusion only "oracle" in evaluations of chain-of-thought monitoring as it provides some insight into what part of the chain is actually being monitored. If looking only at the final answer gives similar performance to considering the full trace, and the answer is not the argument, then the argument itself may not be being monitored.
What would change my mind
The principal argument that would change my mind is if there is something fundamentally different between the critical set and the wrong-answer set. This could be that the critical set contains fundamentally harder physics though there isn't a large enough sample to say for certain. The critical and wrong-answer traces have median lengths of 10.0 and 12.5 steps (Mann-Whitney p = 0.142), which at this sample size means we cannot detect a large difference rather than that we have excluded one. Against the difficulty reading, no error category in the critical set showed a positive CERT-BLIND change in recall (Table III in the paper), so the result is not carried by the repaired or exploratory cases alone. Also, our annotation rules were finalized following human annotator training where we noticed many of the disagreements were due to definitional differences thus it is possible that our rules were ill-specified. Appendix B in the paper goes into more detail here but I believe our definitions – the same rubric was given to the monitors – are pretty solid.
Limitations
Our critical set contained only 24 traces, 16 of which were author adjudicated. Of these 24 traces, only 2 had genuine substantial reasoning errors, with the others being algebraic errors, incorrect supplementary claims, or claims that were later identified and repaired by the model. Section III F of the paper contains a detailed discussion of these.
There are some further limitations such as how we only look at one domain and our ground truth may not be 100% correct. Further this ground truth was itself constructed with access to the reference answer and the rationale provided by HLE thus it is in principle anchored in the same direction that we're measuring improved monitor performance. Finally, during adjudication Fable-5 was inadvertently left on as the selected model during a chat session for approximately three traces – dropping it as a monitor actually moves the difference-in-differences from +0.382 to +0.415 so it actually strengthens the effect.
Paper and links
The full paper is available on arXiv at https://arxiv.org/abs/2609.00264. Code, derived labels and analysis outputs are at https://github.com/WillYeadon/hle_physics_oversight. The repository does not redistribute HLE question text, images, reference answers or rationales, or model-generated trace text. The HLE organizers have been contacted about the reference-answer errors.