(Responding to a deleted reply, which raised good questions.)

I'm trying to guess what the largest models will be like, that are barely practical to run on their contemporary hardware. For example, they are not going to be close to compute constrained on decode, and NVFP4 is likely too risky to use in pretraining (for FFNs). I've mostly updated to FP8 (over BF16) in pretraining based on Nvidia's FP8 bet in Rubin (where the FP8 to BF16 performance ratio is 4.4, which used to be 2), more examples of using FP8 in large-ish models, and the practicality of using many active routed experts in an MoE layer (8+ rather than 1-2), which reduces FFN matrix dimensions.

But for a 44T active param model, with say 100 layers and 512 experts with a 3-matrix GLU nonlinearity, that's inner dimensions in matmuls of at least 17K, so I'm less sure that NVFP4 works very well there. Maybe for inference it's fine, but pretraining is more delicate, especially when entering a new level of scale for the first time. Similarly for sparsity preference, I assume higher sparsity is more difficult to get right at an unfamiliar scale (and 2030 is still in rapid scaling; this consideration changes by 2035). Maybe 30x sparsity is an overestimate for what will be risked in 2030 for the largest models.

For the 2-4 scale-up systems disclaimer that I'm including lately, there's an update (mostly based on Mythos in the context of revenue growth from Opus, rather than Reiner Pope; Pope mostly suggests the anchor of stopping at 4, rather than going further). But also it's to preempt the interpretation (evident in some comments to my posts) that the estimates of practical model sizes I previously used directly imply (or follow from) what I expect as shapes of inference deployments when serving to users (rather than what I expect RLVR to want). Any multiplier for scale-up HBM capacity would work to compare what's practical on a 1 TB scale-up system to what becomes practical on a 14 TB scale-up system, which is roughly 14x larger models by total params. Though possibly less than 14x since the 1 TB systems were too small, and not fitting in one scale-up system was more forced than with a 20 TB GB300 NVL72. In particular, if inference compute is scarce, it can make sense to sacrifice some speed to save on model weight storage even when a model does fit in one scale-up system, or to use FP4 in decode even if it does a bit worse. But since the question is what remains practical for the largest models, the assumption of FP4 in the estimates would have the FP8 version of the model twice as large where quantization is less advisable (if there are such situations at all), so the practicality is going to be constrained by what happens in FP8, even if FP4 is often used in user-facing inference. More crucially, RLVR would still want fast tokens to get more training steps done within some fixed wall-clock time (even if they cost more, meaning need more hardware), and it might want to use less-quantized models, and so inference deployments for RLVR might actually want to fit everything in one scale-up world, with model FFNs still in FP8.

I've mostly updated to the largest models giving up on trying to fit within one scale-up system (even for RLVR) based on Mythos size rumors and greater capabilities over Opus. If Opus is only 3T total params (to fit in one Trainium 2 Ultra, with FP8 FFNs), then anything less than 10T for Mythos probably wouldn't work (unless Opus has surprisingly few active params compared to compute optimal, while Mythos is much closer to compute optimality). If Opus is more than 3T total params, the argument for Mythos being at least 10T total params is only stronger. But that would need GB300 NVL72 under my assumptions about wants of RLVR, and Anthropic doesn't have too many of those as first-party compute. So if even Mythos, which had to be pretrained in 2025, already broke the heuristic of fitting in one scale-up system in FP8, probably it's not a good assumption going forward, considering Opus's commercial impact and Mythos's even greater capabilities. Thus I now expect models of 2027 to somewhat ignore the HBM capacity of GB300/VR NVL72 and get to 20-40T of total params. But not yet to 100-300T total params that Kyber will enable.