Here's a simple, concrete example of how frontier benchmark scores don't always track practical utility.

Until very recently, if you were to navigate to the Artificial Analysis homepage^[1], the first thing you'd see was something like this:

• 59 | Claude 5.5 Opus (max)
• 57 | GPT-6 Astra (max)
• 55 | GPT-6.1 Sol (max)
• 55 | Claude Sonnet 5.5 (max)
• ...a bunch of lower scores for other models...

If you take this at face value, you'll draw a conclusion like: "Opus 5.5 is the most capable model in existence, although Astra is a close second, and other OAI/Ant models aren't far behind."

Notably, though, every one of these scores was determined using max reasoning effort. Indeed, the default view served by the AA site only shows scores at max reasoning effort, or a model's highest effort setting if it doesn't have one named "max."

This might sound reasonable if the goal is to capture the state of the frontier. And yet... as far as I can tell, no one actually uses max reasoning effort in practice.

I mean, maybe someone out there does, for something? But I've never found it useful myself, and my sense of the overall vibe among people who use agents is that max reasoning effort is considered kind of a joke -- a setting that gets you obsessive overthinking at great expense, for little-to-no practical benefit.

Consider, for instance:

• Anthropic's documentation steers users toward medium or high effort as a default for routine use, and tells the user to "reserve xhigh and max for work where you’ve measured a quality gain" (see e.g. here).
  • With Opus 5.5 specifically, the API default is medium (not high), and the docs recommend "starting at" medium.
• Jackson Kernion, who works on post-training at Anthropic, recommends using "Opus 5.5 medium for everything" and says that with Opus 5.5 "medium usually gets you similar preformance as high."
• I recently saw a "Claude reasoning effort tier list" on twitter that placed max effort at "B tier," below high and medium (A tier) and xhigh (S tier). This ordering would probably sound bizarre to someone who's never used a coding agent before, but when I saw it I just kind of nodded along and thought "yeah, sounds about right."

OK, so what are the AA scores like at these more-commonly-used effort levels?

At medium effort, Opus 5.5's score is 48.

That's a whole 11 points lower than its score at max effort. An 11-point difference on AA is huge -- it's about the same size as the gap between Fable 5.2 (max) and Sonnet 5.5 (max), or between Astra (max) and Kimi 3.1 Pro.

Kernion says that "medium usually gets you similar preformance as high". But on AA, Opus 5.5 (high) is 4 points above Opus 5.5 (medium). That's twice as large as the Opus(max)-vs.-Astra(max) gap that jumps out at the viewer immediately on AA's headline chart, and the same size as that chart's Opus(max)-vs.-Sonnet(max) gap.

Well, but lower effort levels are faster and less expensive, right? In the abstract, I could imagine this explaining the disconnect: people are more price-sensitive when it comes to routine use where frontier intelligence is usually not needed.

But, although that's certainly true in the abstract, it doesn't fully dispel the sense that something weird is happening here.

Recall, again, that Opus 5.5 (max) scored 59 on AA and this was a record-breaking score at the time, narrowly edging out Astra (max) for first place. But this dominance is entirely about Opus 5.5 (max), a configuration which (as far as I can tell) basically no one uses outside of benchmarks.^[2]

If you pretend that Opus 5.5 (max) doesn't exist and compare every other Opus 5.5 and Astra effort level on AA's cost-vs.-intelligence chart, Astra looks no worse than Opus and indeed arguably better -- e.g. Opus (xhigh) and Astra (xhigh) both score 57 at nearly identical cost, and Astra (medium) scores 4 points above Opus (medium) while costing only 11% more. Meanwhile, the GPT-6.1 family Pareto dominates (non-max) Opus 5.5 by quite a lot, with e.g. 6.1 Sol (xhigh) achieving the same score as Opus 5.5 (high) while costing only ~1/5 as much.

As for Fable 5.2, it doesn't crater on AA at lower effort levels the way Opus 5.5 does, but it has its own smaller version of the phenomenon we saw with Opus where the use of max effort makes a nontrivial impact on the eye-catching narrative about its dominance at the frontier. At max effort, Fable 5.2 scores 65, which is 5 points above the next-highest model (Carbon) and 6 points about Opus 5.5 (max). But 3 of those points are solely due to the gap between Fable 5.2 (max) and lower effort levels: Fable 5.2 (xhigh) scores 62, as does Fable 5.2 (high).

All of this would be less weird if max effort were more commonly used outside of benchmark runs. But I don't think I've ever seen a single positive review of max effort from a real user -- and I have spent much more time than I like to admit reading tweets (and Discord messages, etc.) by coding agent power users, many of whom were doing fairly difficult or esoteric things.

It's not like this user base is especially cost-sensitive. After all, these are people who adopt the latest frontier model as soon as they possibly can (...but then use it at effort levels far below max!). Or consider the perverse fad for reporting token usage as though it's a benefit rather than a cost (I guess because it's a proxy for "am I using agents enough?"): you'd think that people on this bandwagon would readily embrace max effort, since it reliably uses more tokens and gets higher benchmark scores. But again: I cannot remember ever having seen someone recommend max effort, with any model, for any purpose.

If you trust the benchmarks, the choice to not use max effort with recent frontier models is roughly equivalent to not using a new frontier model that edges ahead of all the competition -- not using Astra on release, say, or Opus 5.5, or [any given Fable version]. But these two choices are received very differently by real users!

1. ^{^}

   In this note I'll use the Artificial Analysis intelligence index as a proxy for "overall capabilities as measured on benchmarks." AA isn't perfect, but it's a popular composite measure and (unlike, say, ECI) it reports scores at multiple reasoning effort levels, which is convenient for my purposes here.

2. ^{^}

   If you're wondering how this relates to non-AA composites: if I understand correctly, ECI and AECI both run each model at several effort levels and then use whichever score happens to be highest. I expect this will yield fairly similar results to using max for everything, since max tends to score highest on benchmarks (although only as an aggregate trend, not in every case, so it's possible that the distinction here matters more than I'm imagining it does).