SemiAnalysis claims HBM for Rubin Ultra was downgraded from 16-Hi to 12-Hi stacks. The 16-Hi HBM was going to be HBM4E, with 4 GB per DRAM die, so 48 GB for 12-Hi stacks. In Rubin Ultra, there are 4 stacks of HBM per compute die, and in Kyber racks there are 144 packages with 4 compute dies each. Thus 192 GB instead of 256 GB per compute die, 768 GB instead of 1024 GB per package, and 110 TB instead of 147 TB per Kyber Rubin Ultra rack. This has implications for practical model sizes of 2028-2029 (for the largest models), they just got 25% smaller, all else equal. In the rest of the post, I discuss a likely change in Feynman that halves HBM per rack compared to what I recently estimated, giving merely 737 TB per 8x Kyber scale-up system in the second year of Feynman. This also makes it more likely that the 8x Kyber scale-up systems (589 TB HBM) are available in the first year of Feynman, rather than only in the second year.

First year Feynman (on the roadmap for 2028, most of the buildout completes over 2029) will probably get back to 16-Hi HBM4E stacks. But Feynman might use a different arrangement of compute dies (original slide deck), so that there are only 2 HBM stacks per compute die instead of 4. It's unclear if this change happens with the first year Feynman, since it's listed as a 2029 design for HBM5 in the KAIST (Joungho Kim) HBM roadmap (from Jun 2025), while the first year of Feynman is 2028. So either the first year Feynman already has 2 HBM stacks per compute die, but it's HBM4E, or the first year Feynman still has 4 HBM stacks per compute die (as in Rubin Ultra). Another question is whether the version of Feynman with 2 HBM stacks per compute die still has 576 compute dies per Kyber rack (in 144 packages), or moves to 1152 compute dies per Kyber rack (in 288 packages). The latter might ask for infeasible power density per rack, but the former reduces the number of HBM stacks per Kyber rack from 2304 (for Rubin Ultra) to 1152 (for Feynman that adopts 2 HBM stacks per compute die).

Thus first year Feynman (2028) might have 147 TB per Kyber rack if it's 4 16-Hi HBM4E stacks per compute die or 73 TB if it's 2 stacks per compute die (576 compute dies in 144 packages per rack, in both cases). And second year Feynman (2029) that uses HBM5 (5 GB per DRAM die) might have 92 TB per Kyber rack if it's 2 16-Hi stacks per compute die and 144 packages per rack (4 compute dies and 8 HBM stacks per package), or 184 TB per rack if it's 288 packages.

Since reducing HBM capacity relative to the previous generation doesn't look good in a product, the largest practical scale-up offering should have at least 2x the number of Kyber racks compared to the previous generation when the move to 2 HBM stacks per compute die happens, if it doesn't simultaneously double the number of compute dies per rack. So either there's a 16x Kyber system for second year Feynman (this wasn't announced), there are 2x more compute dies per Kyber in second year Feynman (making the power density even more alarming), or the first year Feynman will already offer the 8x Kyber system and have 2 HBM stacks per compute die.

My guess is the last possibility, meaning 73 TB per Kyber rack (rather than 147 TB, and down from the 110 TB of Kyber Rubin Ultra), but with 589 TB per 8x Kyber system (thus overall it's still up from 110 TB of Kyber Rubin Ultra) for first year Feynman. And then the second year of Feynman (2029, buildout completes mostly over 2030) merely upgrades from HBM4E to HBM5, keeping 8x Kyber as the largest practical scale-up offering, meaning 92 TB per Kyber rack and 737 TB per 8x Kyber system.

(This is 2x lower than the 1.18 PB and 1.47 PB per system I previously expected, but 10 GW of pretraining compute still predicts 1 quadrillion total params, and that remains borderline practical with these systems. Though it probably won't tolerate FP8 for inference, might really want the contemporary TPUs for RLVR, and sparsity for the largest models of 2030-2031 might need to remain below 30x.)