Aaron_Scher's Shortform

Aaron_Scher

This is a special post for quick takes by Aaron_Scher. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

People often ask whether GPT-5, GPT-5.1, and GPT-5.2 use the same base model. I have no private information, but I think there's a compelling argument that AI developers should update their base models fairly often. The argument comes from the following observations:

The cost of inference at a given level of AI capability has been dropping quickly. A reasonable estimate is 10× per year, or a halving time of 3.6 months (edit: but 3× is also reasonable, it's hard to be sure).
The cost of new near-frontier AI training runs is relatively small, on the order of tens or hundreds of millions of dollars, according to this Epoch data insight based on public reporting for 2024 (which I expect is directionally correct but I wouldn't take too literally).
By contrast, frontier AI developers were spending single digit billions of dollars on AI inference in 2024 (per the same Epoch data insight) and likely high-single digit billions in 2025.

Therefore, it is economically sensible to train entirely new AI models fairly often because their lower inference costs will compensate for the relatively small training costs. "Fairly often" seems like it could be every 2-6 months depending on the exact details.

As a hypothetical example, let's say that OpenAI is considering training a new base model to become GPT-5.1 which will be deployed for only one month before GPT-5.2 is released. Maybe it's 40% cheaper to serve than GPT-5 due to being smaller and using more efficient KV caching^[1]. The cost of serving GPT-5 for that month, assuming it's half of all inference by cost would be ($6B (total inference cost in the year) /2/12)= $250 million; at 40% cheaper, the cost of serving GPT-5.1 would be $150m, saving $100m. If it costs less than $100m to develop GPT-5.1 (in additional marginal costs, because e.g., R&D is amortized across models), then it would be economically sensible to do so.

A big reason to be skeptical of this argument is that there could be large non-compute costs to training, such as lots of staff time—this just pushes training costs up but the overall argument still goes through with a less frequent update rate. Another related reason is that constantly training new models might split the focus of an organization and thus be really costly.

My overall confidence in this take is low, and I would be curious to hear what others think.

^{^}
GPT-5.1 being 40% cheaper than GPT-5 is reasonable given halving times of 3.6 months; the GPT-5 was released August 7, 2025 and GPT-5.1 was released around 3 months later on November 12, 2025; GPT-5.2 was released December 11, 2025.

In this recent interview, Greg Brockman says April 2024 with GPT4o to December 2025 (lines up best with GPT-5.2 released December 11, 2025) was squeezing the juice out of GPT4o. "fundamentally we were on the same underlying base model, the same underlying pretrain". This was largely because producing a new pretrain is very hard, requiring the company to work together very well.

4moozooh9mo

Accuracy being halved going from 5.1 to 5.2 suggests one of the two things: 1) the new model shows dramatic regression on data retrieval which cannot possibly be the desired outcome for a successor, and I'm sure it would be noticed immediately on internal tests and benchmarks, etc.—we'd most likely see this manifest in real-world usage as well; 2) the new model refuses to guess much more often when it isn't too sure (being more cautious about answering wrong), which is a desired outcome meant to reduce hallucinations and slop. I'm betting this is exactly what we're looking at, and your Sonnet graph also suggests the same. So if your methodology counts refusal as lowering accuracy, then it doesn't necessarily prove the base model or the training data mix is different. Teaching a model to refuse on low-signal data is in the domain of SFT and reinforcement learning, and investing into that heavily on the same pretrain would result in something similar to the graph you've posted. 4o and 5 almost certainly have different base models since 4o is natively omnimodal and 5 and its derivatives are not, taking that into account you have to make a lot of weird assumptions to reconcile this discrepancy. 5 and 4.1, on the other hand... Everything seems to fall into place neatly when looking in that direction.

2Daniel Paleka9mo

I think 1 is true. This is only a single, quite obscure, factual recall eval. It's certainly possible to have regressions on some evals across model versions if you don't optimize for those evals at all. Wrt point 2 -> here is the plot of how often the models guess, versus say they do not know, on the same dataset. My understanding is that the theory in point 2 would have predicted a much more dramatic drop in GPT-5.2?

Aaron_Scher's Shortform

3

Aaron_Scher's Shortform

3