Nvidia confirms 8x Oberon scale-up systems for Rubin Ultra, so the feasibility of 2028 models with about 200T total params is getting less precarious (despite the bigger Kyber rack likely being delayed until Feynman; Kyber wasn't mentioned in this Rubin Ultra talk, for example). It's based on CPO NVLink switches (so the possible issues with CPO for NVLink are probably getting resolved). If individual chips hold 192 GB of HBM (down from 288 GB for non-Ultra Rubin), this is 110 TB per scale-up system (8 racks with 72 chips each), which can fit models of 2028 with 110T-330T[1] total params (with pipelining; depending on if they use FP8 or FP4 FFN weights).
To reiterate the argument for model param count, HBM4's bandwidth makes pipelines with 3-4 stages practical (while maintaining 60+ tokens/s per request in decode; unlike much slower HBM3E of Blackwell), which is enough capacity for about 110-165 TB of model weights (2-3x half-capacity of a scale-up system). (I'm assuming pipelining as the more straightforward constraint that gives a soft upper bound on model sizes; in principle WideEP enables even bigger models, but it's unclear to me if the resulting cost or time per output token in decode would be worth making models that are even bigger.) Judging from Jalapeno's bet on FP4 (which is a 27 TB HBM4 scale-up system, with a 3.9 FP4/FP8 FLOP/s ratio), deployment of even very big models in FP4 could be a serious possibility. And in FP4, 110-165 TB become 220T-330T total params. In FP8, the lower end of the estimate remains 110T total params. ↩︎