This is a special post for quick takes by LouisMegatron. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Some (over) simplified explanation to the METR anthropic article
first I suspect ECI progress rate is constant against compute. i.e. k = FLOP/eci.
supporting evidence: Mimo's live RL run demonstrates that benchmarks points went up as RL steps went up. this is a clean RL run, no idea change, no model arch change, only compute input.
mimo RL run
then what is needed to let the GPUs run? there are 1) ideas 2) code. 3) let them run on GPU. they are serialised.
for code there are two types of code: a researcher would write codes to test their ideas, and adopt/discard it. (exp-code), and the training-related infrastructure code (infra-code) , like the distributed training framework, GPU kernels, etc.
Exp-codes and infra-codes are different. in exp-code, it needs to be written-from-scratch most of the time, since ideas are new. and for infra-code, it needs to be built, then to be maintained. companies rarely write infra from scratch every few months.
I now suspect three things changed after the release of Opus 4.5, i.e. the slope change in METR post.
1. FLOPs went up. Most post-oct-2025 models are trained on Blackwell GPUs, and the massive deployment of Vera Rubins may change that number. since the FLOPs for VR GPUs are higher than Blackwells. (it needs fewer seconds to achieve the FLOP needed for raising one ECI point). So GPU Flops could be the dominent reason.
2. Opus 4.5 is capable of boosting exp-code writing 10-100x, which eliminates the coding bottleneck completely. from metr's interview, and as a software engineer myself, I assign high confidence to this argument.
3. the infra code is built. and only needs to be maintained. I am not that confident, though, as I am not an insider. If most infra bugs are eliminated, running large pretraining runs would be easier, even without superhuman ai coders.
FP8 flops for Hopper, Blackwell and Rubin
by combining 1) 2) and 3), it is possible that a ECI step change is explainable, by completely removing the coding part. This does not necessary means that anthropic's internal model can write the infra codes from scratch, as maintaining codes are easier than building a new one, also they are still hiring.
Next question is, could it accelerate more? I am not sure, but I have some thoughts here:
1. Rubins are coming. if FLOPs is the dominant factor, more speedups incoming.
2. how fast u could propose idea is also a crux. This openai infra engineer's podcast suggested that idea may not be that important. but I am not sure.
Open source Chinese models can achieve similar performance in METR's time horizon as western SOTAs, by simply running twice, and still being cheaper.
I did a small research when waiting my flight at SFO, using METR's TH 1.0 dataset (unfortunately, TH 1.1 does not have any Chinese model).
The math behind this is simple: View the success/fail of a task as an independent event. For a certain task, given the success rate of a Chinese LLM (p) and a SOTA LLM (p_sota), the try out number n for a Chinese LLM can be derived as:
Or, by trying n times, the overall success rate r = 1 - (1-p)^n.
I picked these 'rival pairs' for Chinese and then-SOTA US LLMs:
- DeepSeek R1-0528 (May 28) vs o3 (Apr 16, -42d), Claude 4 Opus (May 22, -6d)
- Kimi K2-thinking (Nov 6) vs GPT-5.1-Codex-Max (Nov 19, +13d), Claude Opus 4.5 (Nov 24, +18d)
- Two 'Bridge models': GPT-5 and Claude 4.1 Opus (both Aug) connect the two groups.
since METR have already provided time horizons for p_sota=50% and p_sota=80%, the p for Chinese models can be derived by fitting the curves.
For most rival pairs, EXCEPT kimi-k2-thinking v. Opus 4.5 at p50, it only requires 2 independent runs to match US SOTA model performance.
METR provides token usage for each task. By multiplying the token price and the average token usage for tasks within a certain time horizon, the average price count can be calculated.
Given Chinese models are significantly cheaper than OAI or Anthropic's models, it is not surprising for this chart:
Please keep in mind that this evaluation is naive since a) we only have two Chinese models. b) best-of-n is only valid in easy-to-verify domains, like METR's software engineering evaluations. and c) METR benchmark tasks are not production workloads
I did this research because i saw someone on X saying (sorry i cannot find the source rn) Chinese models are bs since they are less intelligent. Yes, but in easy-to-verify domains, one can simply run a 'swarm' of Chinese models in parallel and get things done eventually. Just like how Shein products flood the US market!
Thanks for pointing out! i re-evaluated the data and my methodology is flawed.
METR provides task-level success rate. They run a model several time (4-8 typically) on one task. Here is the per-task success rate graph for tasks > 1h human completion time, using Opus 4.5, codex-5,1-max and kimi-k2 (made by sonnet 4.6)
the X-axis is ordered by human completion time (time horizon). beyond some certain point, kimi almost always fails. yet SOTA models could succeed (red blocks in heat map). For these tasks, running n times would not help much since they are likely beyond a model's capabilities.
However, for tasks that both models can complete (the green-ish blocks), i still lean to assume the structural independence of runs, (e.g. one could run a task several times with a clean context window each time). My mental model for these tasks is like this slot machine meme. best-of-n sampling will improve the overall success rate on these tasks IMO.