andreasrobinson — LessWrong

LESSWRONG
LW

Replying toClaude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins

Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins

The flat 80%-horizon while 50%-horizon climbs might be evidence for intercept-dominated progress.

@Michaël Trazzi Actually, it's the opposite, the Claude progress was dominated by slope () improvement, and intercept actually got a bit worse: Is METR Underestimating LLM Time Horizons?

Replying toIs METR Underestimating LLM Time Horizons?

andreasrobinson25d

Is METR Underestimating LLM Time Horizons?

Your link to chatgpt's analysis is not loading for me, but I don't really trust chatgpt to do this without mistakes anyway; when I run this regression in python statsmodels with METR's official $β$ values and release dates from their github (plus the recent gpt-5 models), it is stat sig for this OpenAI instruct-model trend (p ~ .04) and it gets more stat sig once we include Claude 4.5. I don't see what dates you are using, but your beta values have a fair amount of (rounding?) error compared to METR's data, and the model list isn't the same, e.g. they didn't include non-frontier mini models, etc. (btw, note that one-sided p-values are... (read more)

-1

•••

Replying toIs METR Underestimating LLM Time Horizons?

andreasrobinson25d*

Is METR Underestimating LLM Time Horizons?

If you run a linear regression on $β$ versus time, then the regression line does have a positive slope estimate (even pre-claude-4.5), and I used that regression line in section 2.2.1.1 to provide an alternative estimate for when LLMs will catch up with human $β$ (at which point the denominator goes to zero, and the LLM's horizon blows up). That said, the $β$ trend is quite noisy; and while the OpenAI trend from that estimate was already stat sig, the overall $β$ trend across companies did not hit stat sig until the Claude Opus 4.5 datapoint. Also, as I argued in the post (partly due to the noisy $β$ trend): "I suspect it makes more sense to directly extrapolate the overall time-horizon... (read more)

Replying toIs METR Underestimating LLM Time Horizons?

andreasrobinson1mo

Is METR Underestimating LLM Time Horizons?

Could you actually provide a citation for Claude already being a supercoder?

I did not claim that Claude is a "supercoder" or even human-level at coding; rather, the Claude addendum continued with: "to be clear, we shouldn't over-interpret this specific 444 billion figure" and "Realistically, this highlights that to really make accurate projections of the time to catch up with human horizons based on METR data, we need better human baselines." In my view, the natural takeaway is that that Claude has now basically caught up with METR's existing human baselines, which they have acknowledged were not that well incentivized, which does not mean that it is better than properly incentived software engineers.... (read more)

Is METR Underestimating LLM Time Horizons?

andreasrobinson

1mo

TL;DR

Using METR human-baseline data, I define an alternate LLM time-horizon measure, i.e. the longest time horizon over which an LLM exceeds human baseline reliability (or equivalently the intersection point of the human and LLM logistic curves), and this measure shows a much faster growth-trend than METR's fixed-threshold trends: doubling every 1.9 months, versus 6.8 months for the 50% METR-trend over the same time period. Also, since this metric is directly comparing to human baseline reliabilities (unlike the METR fixed-reliability estimates), we can use it in a more principled way to assess time to human-level horizons, which suggests roughly 2026-2027, with substantial uncertainty.
METR has generally deemphasized their human reliability baselines, on the grounds

... (read 5060 more words →)

•••

Replying toWhat does 10x-ing effective compute get you?

andreasrobinson6mo

What does 10x-ing effective compute get you?

Regarding the 3x/1.25x copy/speedup multipliers, do you have any kind of back-of-the-envelop justification for why those are plausible, e.g. based on the trend of algorithmic-progress, such that you might expect about this much inference-efficiency gain paralleling the assumed 10x (or so) training-efficiency gain?