bhalstead

If I were Anthropic, I'd be worried about distillation attacks via this route. It seems to make their CoT obfuscation moot.

c.f. this tweet from Greg Burnham, showing how Opus 4.5 barely suffers on FrontierMath from having CoT disabled:

In the No Thinking setting, Opus 4.5 repurposes the Python tool to have an extended chain of thought. It just writes long comments, prints something simple, and loops! Here's how it starts one problem:

This is such a cool method! I am really curious about applying this method to Anthropic's models. Would you mind sharing the script / data you used?

the method of estimating hardware vs software share seems biased in the direction of exaggerating hardware share. (because training compute and software efficiency both tend to increase over time, so doing a univariate regression on training compute will include some of the effect from software improvements. so subtracting off that coefficient from the total will underestimate the effect of software improvements / “algorithmic progress”.)

From the paper:

Comparing these two estimates allows us to decompose the total gain into hardware and software components. By subtracting the compute effect (0.048) from the total effect (0.083), we isolate a residual of 0.035. This residual represents algorithmic progress, an economic catch-all for improvements in model architecture, software optimization, and user learning—effectively the Solow residual of AI production. In percentage terms, this decomposition suggests that compute scaling drives approximately 56% of the total reduction in time, while algorithmic advancements account for the remaining 44%.

Yep! Here's an example where the 50% horizon and 80% horizon can be lower for an agent whose success profile dominates another agent (i.e. higher success rate at all task lengths), even for
(1) monotone nonincreasing success rates (i.e. longer tasks are harder)
(2) success rate of 1 at minimum task length
(3) success rate of 0 at maximum task length

before points are
[(0,1), (1, 1/15), (2, 0), (3,0)]
after points are
[(0,1), (1, 0.1), (2, 0.1), (3, 1/15)]
https://www.desmos.com/calculator/nqwn6ofmzq
 

Progress probably has sped up in the past couple of years. And training compute scaling has, if anything, slowed down (it hasn't accelerated, anyway). So yes, I think "software progress" probably has sped up in the past couple of years. 

I haven't looked into whether you can see the algorithmic progress speedup in the ECI data using the methodology I was describing. The data would be very sparse if you e.g. tried to restrict to pre-2024 models for greater alignment with the Algorithmic Progress in Language Models paper, which is where the 3x per year number comes from.

Also, that 3x per year number is only measuring pre-training improvements. Post-training (1) didn't really exist before 2022 and (2) was notably accelerated in 2024 by the introduction of RLVR. I wouldn't be confident in whether pre-training algorithmic progress alone is much faster than 3x per year today. (as rumor would have it, there's substantial divergence between the different AGI companies on the rate of pretraining progress.)

Registering that I don't expect GPT-5 to be "the biggest model release of the year," for various reasons. I would guess (based on the cost and speed) that the model is GPT-4.1-sized. Conditional on this, the total training compute is likely to be below the state of the art.

For what it's worth, the model showcased in December (then called o3) seems to be completely different from the model that METR benchmarked (now called o3).