Here's a simple, concrete example of how frontier benchmark scores don't always track practical utility.
Until very recently, if you were to navigate to the Artificial Analysis homepage[1], the first thing you'd see was something like this:
* 59 | Claude 5.5 Opus (max)
* 57 | GPT-6 Astra (max)
* 55 | GPT-6.1 Sol (max)
* 55 | Claude Sonnet 5.5 (max)
* ...a bunch of lower scores for other models...
If you take this at face value, you'll draw a conclusion like: "Opus 5.5 is the most capable model in existence, although Astra is a close second, and other OAI/Ant models aren't far behind."
Notably, though, every one of these scores was determined using max reasoning effort. Indeed, the default view served by the AA site only shows scores at max reasoning effort, or a model's highest effort setting if it doesn't have one named "max."
This might sound reasonable if the goal is to capture the state of the frontier. And yet... as far as I can tell, no one actually uses max reasoning effort in practice.
I mean, maybe someone out there does, for something? But I've never found it useful myself, and my sense of the overall vibe among people who use agents is that max reasoning effort is considered kind of a joke -- a setting that gets you obsessive overthinking at great expense, for little-to-no practical benefit.
Consider, for instance:
* Anthropic's documentation steers users toward medium or high effort as a default for routine use, and tells the user to "reserve xhigh and max for work where you’ve measured a quality gain" (see e.g. here).
* With Opus 5.5 specifically, the API default is medium (not high), and the docs recommend "starting at" medium.
* Jackson Kernion, who works on post-training at Anthropic, recommends using "Opus 5.5 medium for everything" and says that with Opus 5.5 "medium usually gets you similar preformance as high."
* I recently saw a "Claude reasoning effort tier list" on twitter that placed max effort at "B tier," below high and m
