Terminal-Bench 2.0, one of the best agent benchmarks, runs agents on 89 tasks with at least 5 tries per task. The aggregate rankings are then published along with a confidence interval. What I wanted to know, however, was whether or not the point gap between any two given agents represented a statistically significant difference (or if it was just noise).
My first finding was that if you look at pairs with adjacent rankings then 24/25 of them differ by such a small amount that it's essentially statistical noise (if, that is, you take into account that the tasks are a sample and you do the appropriate uncertainty clustering). This is particularly evident towards the top of the leaderboard where ranks 5-13 have only a 5 point range (from 67% to 62%).
I looked at all 325 pairs of agents using task-clustered bootstrap with Benjamini-Hochberg correction for multiple comparisons and found that 23% were statistically equivalent. A minority, yes, but a sufficiently large enough minority to potentially throw a wrench into the agent selection process. My first thought was that perhaps this was being driven by agents' individual inconsistencies across repeated trials; I found, however, that their reliability in this regard is actually pretty high (ICC 0.5-0.8). There just aren't enough tasks to yield statistical significance. A naive analysis may miss this because if you treat all 52,104 trials as independent (which they are not, multiple attempts on the same task are highly correlated) then the required confidence intervals are only half the size.
My second finding had to with the effect of the scaffold used for each agent. It turns out that the harness that manages each agent/model can be of far more importance than the chosen agent: if you hold the agent fixed and vary the scaffold you see more significant differences than if you, for example, switch out agent rank #2 for agent rank #3 (see chart).
Here are the details:
-Claude Opus 4.5 -- four scaffolds, 15 points span
-Claude Haiku 4.5 -- two scaffolds, 22 points span
-Claude Opus 4.6 -- two scaffolds, 8 point span
-Qwen3-Coder-480B -- three scaffolds, 6 point span
-Gemini 3 Pro -- the only one that was scaffold-insensitive (four scaffolds, 4 point span)
The point gaps between the top agents, on the other hand, tend to be only a few points -- potentially dwarfed by the choice of scaffold.
(Limitation: I couldn't investigate the effect of varying the agents with a single scaffold. The data has multiple scaffolds per agent but only one agent per scaffold.)
So, instead of investing time in choosing between, for example, the top two agents (which are a statistical tie) we need to focus on choosing the best scaffold!
Methodology:
Choices were fixed prior to viewing results. Pass rates weight each task equally, not each trial (trial counts per task vary wildly). CIs and pairwise tests used task-clustered bootstrap. Multiple comparisons are FDR controlled (q=0.05). For tasks with enough trials reliability was ICC(1). Everything computed on public Terminal-Bench 2.0 leaderboard data (tbench.ai) and accessed through yoonholee/terminalbench-trajectories (Apache 2.0). Terminal-Bench and Harbor are released under Apache 2.0. Numbers won't perfectly match the official leaderboard because it uses its own averaging, but my findings are the same regardless and replicate what I saw on the earlier Terminal-Bench archive.
Terminal-Bench 2.0, one of the best agent benchmarks, runs agents on 89 tasks with at least 5 tries per task. The aggregate rankings are then published along with a confidence interval. What I wanted to know, however, was whether or not the point gap between any two given agents represented a statistically significant difference (or if it was just noise).
My first finding was that if you look at pairs with adjacent rankings then 24/25 of them differ by such a small amount that it's essentially statistical noise (if, that is, you take into account that the tasks are a sample and you do the appropriate uncertainty clustering). This is particularly evident towards the top of the leaderboard where ranks 5-13 have only a 5 point range (from 67% to 62%).
I looked at all 325 pairs of agents using task-clustered bootstrap with Benjamini-Hochberg correction for multiple comparisons and found that 23% were statistically equivalent. A minority, yes, but a sufficiently large enough minority to potentially throw a wrench into the agent selection process. My first thought was that perhaps this was being driven by agents' individual inconsistencies across repeated trials; I found, however, that their reliability in this regard is actually pretty high (ICC 0.5-0.8). There just aren't enough tasks to yield statistical significance. A naive analysis may miss this because if you treat all 52,104 trials as independent (which they are not, multiple attempts on the same task are highly correlated) then the required confidence intervals are only half the size.
My second finding had to with the effect of the scaffold used for each agent. It turns out that the harness that manages each agent/model can be of far more importance than the chosen agent: if you hold the agent fixed and vary the scaffold you see more significant differences than if you, for example, switch out agent rank #2 for agent rank #3 (see chart).
Here are the details:
-Claude Opus 4.5 -- four scaffolds, 15 points span
-Claude Haiku 4.5 -- two scaffolds, 22 points span
-Claude Opus 4.6 -- two scaffolds, 8 point span
-Qwen3-Coder-480B -- three scaffolds, 6 point span
-Gemini 3 Pro -- the only one that was scaffold-insensitive (four scaffolds, 4 point span)
The point gaps between the top agents, on the other hand, tend to be only a few points -- potentially dwarfed by the choice of scaffold.
(Limitation: I couldn't investigate the effect of varying the agents with a single scaffold. The data has multiple scaffolds per agent but only one agent per scaffold.)
So, instead of investing time in choosing between, for example, the top two agents (which are a statistical tie) we need to focus on choosing the best scaffold!
Methodology:
Choices were fixed prior to viewing results. Pass rates weight each task equally, not each trial (trial counts per task vary wildly). CIs and pairwise tests used task-clustered bootstrap. Multiple comparisons are FDR controlled (q=0.05). For tasks with enough trials reliability was ICC(1). Everything computed on public Terminal-Bench 2.0 leaderboard data (tbench.ai) and accessed through yoonholee/terminalbench-trajectories (Apache 2.0). Terminal-Bench and Harbor are released under Apache 2.0. Numbers won't perfectly match the official leaderboard because it uses its own averaging, but my findings are the same regardless and replicate what I saw on the earlier Terminal-Bench archive.