For what it's worth, I would expect the behavior you described, and suspect a malicious explanation overall. 

Anecdotally, Claude lies/cheats an enormous amount (far more than comparable frontier models).

From the published cases, I could not find a clear example of Sonnet 4.5 following that particular pattern, but maybe I missed it. Could you share?

What apps have you tried for this, and how recently?

Most of my usage is multi-turn in a 200k line codebase, for what it's worth. It's extremely rare that GPT-5 (via Codex CLI) breaks anything.

What app were you using? This sounds very similar to my experience using GPT-5 in Cursor.

Codex CLI is much much better -- night and day difference.

I suppose this is good evidence that harness-specific RL was important for GPT-5.

Anecdotally, GPT-5 seems way above trend for real-world agentic coding.

I think METR's constrained task-set overstated the capabilities of previous models, and the "real-world" performance of GPT-5 in e.g. Codex CLI seems to be much much higher than e.g. Claude Code.

Sonnet/Opus 4 was almost never able to test, debug, and fix end-to-end in our real codebase, and GPT-5 usually can.

I don't work with RL, but I predict that if you created an RL environment with 
1. GPT-5 (released <1 month ago) via Codex CLI
2. Claude Opus 4 (released 4 months ago) via Claude Code
you would see a dramatic difference in robustness/quality/functionality. Perhaps someone could test this.

I'm not sure what drove GPT-5 seeming so much better for agentic coding (plausibly a big RL scale-up + other improvements?), but I do expect recent and upcoming advancements to drive an explosion of RL environment quality/quantity/robustness.

The prediction is correct on all counts, and perhaps slightly understates progress (though it obviously makes weak/ambiguous claims across the board).

The claim that "coding and research agents are beginning to transform their professions" is straightforwardly true (e.g. 50% of Google lines of code are now generated by AI). The METR study was concentrated in March (which is early 2025). 

And it is not currently "mid-late 2025", it is 16 days after the exact midpoint of the year.

Early: January, February, March, April

Mid: May, June, July, August

Late: September, October, November, December

If you extrapolate further, do you think the one-dimensional scale works well to describe the high-level trend (surpassing human abilities broadly)?

Trying to determine if the disagreement here is "AI probably won't surpass human abilities broadly in a short time" or "even if it does, the one-dimensional scale wasn't a good way to describe the trend".

I agree that AI capabilities are spiky and developed in an unusual order. And I agree that because of this, the single-variable representation of intelligence is not very useful for understanding the range of abilities of current frontier models.

At the same time, I expect the jump from "Worse than humans at almost everything" to "Better than humans at almost everything" will be <5 years, which would make the single-variable representation work reasonably well for the purposes of the graph.

I think these "examples of silly mistakes" have not held up well at all. This was often blamed on "training around the limitations"; however, in the case of the linked post, we got a model the next day that performed much better.

And almost every benchmark and measurable set of capabilities has rapidly improved (in some cases beyond human experts).

"We too often give wrong answers to questions ourselves to be justified in being very pleased at such evidence of fallibility on the part of the machines. Further, our superiority can only be felt on such an occasion in relation to the one machine over which we have scored our petty triumph."
Alan Turing, Computing Machinery and Intelligence
1950

I've generally found it much harder over time to find "examples where LLMs fail in surprising ways". If you test o3 (released the day after that post!) for the examples they chose, it does much better than previous models. And I've just tried it on your "269 words" task, which it nailed.