Aaron_Scher's Shortform

Aaron_Scher

This is a special post for quick takes by Aaron_Scher. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I initially shared this on Twitter. I'm copying over here because I don't think it got enough attention. Here's my current favorite LLM take (epistemic status is conspiratorial speculation).

We can guesstimate the size of GPT-5 series models by assuming that OpenAI wouldn't release the gpt-oss models above the cost-performance pareto curve. We get performance from benchmarks, e.g., ArtificialAnalysis index.

That gives us the following performance ordering: 20B < nano < 120B < mini < full. But the ordering depends on where you get #s and what mix. From model cards: Nano < 20B < 120B ≤ mini < full. We get cost from model size, recall the oss models:

20b: 21B-A3.6B

120b: 117B-A5.1B

Now time for the wild speculation. I think this implies that Nano is order 0.5-3B. Mini is probably in the 3B-6B range, and GPT-5 full is probably in the 4-40B range. For active parameters. This would be consistent with the 5x pricing diff between each, if e.g., 1B, 5B, 25B.

This is super speculative obviously. We don't even know model architecture, and something totally different could be happening behind the scenes.

One implication of this analysis pointing to such small model sizes is that it indicates GPT-5 *really* wasn't a big scale-up in compute, maybe even a scale down vs. 4, almost certainly a scale down vs. 4.5.

API pricing for each of these models at release (per 1m input/output): GPT-4: 30/60 (rumored to be 1800B-A280B) GPT 4.5: 75/150 GPT-5: 1.25/10 If you're curious about the pricing ratio from 4 to 5, it might imply active parameter counts in the range of 12B-47B for 5.

Thanks to ArtificialAnalysis who is doing a public service by aggregating data about models. Also here's a fun graph.

There's a DeepMind textbook (the estimate for output tokens is in equation (1) of the inference chapter), though it focuses of TPUs and doesn't discuss issues with low total HBM in Nvidia's older 8-chip servers (which somewhat go away with GB200 NVL72). (I previously tried explaining output token generation a bit in this comment.)

Basically, all else equal chips compute much faster than they can get input data for that computation (even from HBM), but if they are multiplying sufficiently large matrices, then the number of necessary operations gets much grea... (read more)

4[anonymous]1y

I suspect the models' output tokens become input tokens when the conversation proceeds to the next turn; certainly my API statistics show several times as many input tokens as output despite the fact that my responses are invariably shorter than the models'.

3eggsyntax1y

Thanks, that's a really great point, and I feel silly for not considering it. I'm not entirely sure offhand how to model that. Naively, the entire conversation is fed in as context on each forward pass, but I think KV caching means that's not entirely the right way to model it (and how cached tokens are billed probably varies by model and provider). I'm also not sure to what extent those caches persist across conversational turns (and even if they are, presumably they're dropped once the user hasn't responded again for some length of time).