Peter Wildeford: I’d expect the 10GW OpenAI cluster becomes operational around 2027-2028.
There are no 10 GW OpenAI clusters, there is a conditional investment by Nvidia for every additional 1 GW in total across all their datacenters, which won't be forming a single training system. Inference and smaller training experiments need a lot of compute, and OpenAI is now building their own even for inference, so most of what they are building is not for frontier model training.
Zvi Mowshowitz: these announcements plus the original cite in Abilene, Texas cover over $400 billion and 7-gigawatts over three years
Depending on OpenAI growth, this is more of a soft upper bound on what gets built. This is evidence that 5 GW training systems likely aren't going to be built by (end of) 2028. So there's going to be a slowdown compared to the trend of 12x in compute (and 6x in power) every 2 years for the largest frontier AI training systems, which held in 2022-2026[1]. In 2027-2028, the largest training systems are likely going to be merely 2 GW instead of the on-trend 5 GW. Though for the 2024 systems, FP8 is likely relevant, and for 2026 systems maybe even FP4, which turns the 12x in compute every 2 years in 2022-2026 into 24x in compute every 2 years (5x per year in pretraining-relevant raw compute for a single training system).
This 24x every 2 years is even less plausible to remain on-trend in 2028, so 2027+ is going to be the time of scaling slowdown, at least for pretraining. Though the AIs trained on 2026 compute might only come out in 2027-2028, judging by how 2024 training compute is still held back by inference capabilities, and some AIs enabled by 2024 levels of compute might only come out in 2026. So the slowdown in the scale of frontier AI training systems after 2026 might only start being observable in scaling of deployed AIs starting in 2028-2029.
Perhaps some of these sites will be connected with sufficient bandwidth, and training with RLVR at multiple sites doesn't need a lot of bandwidth. Actual plans for larger training runs should urge them to build larger individual sites, as this ensures optionality for unusual training processes. So the fact that this isn't happening suggests that there are no such plans for now (except perhaps for RLVR-like things specifically).
24K A100s in 2022 (7e18 BF16 FLOP/s, 22 MW), 100K H100s in 2024 (1e20 BF16 FLOP/s, 150 MW), 400K chips in GB200/GB300 NVL72 racks in 2026 (1e21 BF16 FLOP/s, 900 MW). The power estimates are all-in, for the whole datacenter site. ↩︎
Thanks for your thoughtful pushback.
I want to withdraw my prediction "I’d expect the 10GW OpenAI cluster becomes operational around 2027-2028." I spoke too quickly in my Twitter thread and this was based on a confusion on my part.
I will have a more detailed article soon to give more thoughtful updated predictions. I apologize for this error.
3Forgive if a naive question, but what about distributed training runs — any view on whether progress on that front will result in training runs larger than what you describe here?
PS thank you for the extremely useful analysis as always — I would 100% subscribe if you had a newsletter or something (with a strictly positive willingness-to-pay even!).
There doesn't necessarily need to be algorithmic progress to get there, sufficient bandwidth enables traditional pretraining across multiple sites. But it might be difficult to ensure it's available across the geographically distributed sites on short notice, if you aren't already a well-established hyperscaler building near your older datacenter sites.
In 2028, targeting inference on Rubin Ultra NVL576 (150 TB of HBM in a scale-up world) might want a MoE model with 80 TB of total params (80T params if in FP8, 160T in FP4). If training uses the same precision for gradients, that's also 80 TB of gradients to exchange. If averaged gradients use more precision, this could be 2x-8x more data.
If training is done using 2 GW of some kind of Rubin GPUs, that's about 2e22-3e22 FP4 FLOP/s, and at 30% utilization for 4 months it produces 8e28 FP4 FLOPs. At 120 tokens/param (anchoring to 40 tokens/param for the dense Llama 3 405B and adjusting 3x for 1:8 sparsity), this system might want about 10T active params (so we get 1:16 sparsity, with 160T total FP4 params, or about 1:8 for FP8). This needs 1,200T tokens, maybe 250T unique, which is a problem, but not yet orders of magnitude beyond the pale, so probably something can still be done without needing bigger models.
With large scale-up worlds, processing sequences of 32K tokens with non-CPX Rubin NVL144 at 30% utilization would take just 2.7 seconds (for pretraining). A 2 GW system has 9K racks, so that's a batch of 300M tokens, which is already a lot (Llama 3 405B used 16M token batches in the main phase of pretraining), so that should be the target characteristic time for exchanging gradients.
Moving 80 TB in 2.7 seconds needs 240 Tbps, or 500-2,000 Tbps if averaged gradients use 2x-8x more precision bits (even more if not all-to-all, which is likely with more than 2 sites), and this already loses half of utilization or asks for even larger batches. A DWDM system might transmit 30-70 Tbps over a fiber optic pair, so this is 4-70 fiber optic pairs, which seems in principle feasible to secure for overland fiber cables (which hold hundreds of pairs), especially towards the lower end of the estimate.
As a naive follow-up: let's say GPT-6 could be trained in 3 months on a 3GW cluster. Could I instead train it in 9 months on a 1GW cluster?
Depending on OpenAI growth, this is more of a soft upper bound on what gets built.
I'm confused: the announcement indicates that the $400B has been committed, and is not dependent on OpenAI's growth (although perhaps you're implying that there's no way they actually spend the $400B unless OpenAI revenue continues to rapidly grow)?
Also, why would this $400B / 7GW be an upper bound? A recent WSJ article suggests they are planning to surpass that, although details are super light.
Do you think that pretraining is currently bottlenecked by compute, or clean text data? How do you forecast this will change?
A key worry has long been that OpenAI depends on Nvidia, so OpenAI may be reluctant to speak about AI risk lest they anger Nvidia and endanger their chip allocations. OpenAI has not been a great political actor, but Sam Altman appreciates many of the dangers and issues that will be at stake.
Is there any evidence that he still believes this? E.g. his gentle singularity essay suggests otherwise.
They also show us the chips, and the data centers.
It is quite a large amount of money, and chips, and some very large data centers.
Nvidia Invests Up To $100 Billion In OpenAI
Nvidia to invest ‘as much as’ $100 billion (in cash) into OpenAI to support new data centers, with $10 billion in the first stage, on the heels of investing $5 billion into Intel.
New Street Research estimates that OpenAI will spend $35 on Nvidia chips for every $10 that Nvidia invests in OpenAI. Nice work if you can get it.
Nvidia was up about 4.5% on the announcement. This project alone is a full year’s worth of Nvidia chips, which should further put to bed any question of Nvidia running out of Western customers for its chips, as discussed later in the post.
Note that the details of this deal are not finalized, which means the deal isn’t done.
OpenAI’s Stargate Program Announces New Sites And Big Spending
OpenAI paired this with progressing its Stargate project with Oracle and SoftBank, expanding into five new AI data center cites and claiming they are on track to complete their $500 billion, 10-gigawatt commitment by the end of 2025, as these announcements plus the original cite in Abilene, Texas cover over $400 billion and 7-gigawatts over three years.
The new sites are Shackelford County, Texas, Dona Ana County, New Mexico, ‘a cite in the midwest to be announced soon,’ the existing Oracle cite in Lordstown, Ohio, and next year a cite in Milam County, Texas developed in partnership with SB energy.
Berber Jin at WSJ described the full plan as a $1 trillion buildout of computing warehouses across the nation and beyond.
To celebrate all this, Sam Altman wrote Abundant Intelligence.
Very Large Training Runs Are Coming
What about what this means for training runs? It means there is no physical cap that is likely to bind, it will be more about what makes economic sense since compute will be in high demand on all fronts.
What about for other purposes? Ambitions are running high.
That seems like quite the timeline for either curing cancer or providing free global education. But yes, compute is going to be like any other limited resource, you’re going to have to make tradeoffs. Spending on health care and education cannot be unlimited, nor does it automatically ‘win in a fight’ against other needs.
Eyes are on the prize. That prize is superintelligence.
Should we have plans and policies designed around the idea that superintelligence is coming? Well, we certainly shouldn’t build our plans and policies around the idea that the AI companies do not expect superintelligence, or that they won’t try to build it.
If that statement does not terrify you, either you did not understand it, you did not think through its consequences, or you have become numb to such things.
In case it is not obvious, no, they are not racing for ‘market share’ of inference tokens.
A central argument for going forward with AI, that was often cited by OpenAI CEO Sam Altman, used to be ‘compute overhang.’ As in, if we didn’t build AI now, then when we did built it we would have more compute available and progress faster, which would be more dangerous, so better to make progress now. Ah, simpler times.
Nvidia Does Not Have Too Much Money
Does Nvidia have, as Dan Gallagher claims in the Wall Street Journal, Too Much Money? As in so much money they don’t know what to do with it and it loses meaning.
Nvidia is set to generate $165 billion or so in annualized free cash flow by the end of 2028 according to consensus projections:
In the worst case you can always return value to shareholders. Nvidia wisely only pays the minimum $0.01 dividend (so it can be included in various financial products), but has already repurchased $50 billion of its own stock over the past year and plans to buy another $60 billion more.
The argument iby Gallagher is largely that Nvidia cannot make a major acquisition without Chinese approval? I do not understand why they would care, given China is already actively telling companies not to buy Nvidia’s products? Seems like none of their damn business.
But also Nvidia can do further strategic investments and projects instead that can easily eat up this level of free cash flow, if they have no better use for it. Stock market investors would be happy to get more exposure to private companies like OpenAI.
Thinking around Nvidia on Wall Street has for a long time been rather silly and my only regret is I did not buy in far sooner (for those who don’t know, consider this my disclosure that I am overweight Nvidia and have been for a while, quite profitably). For example, a real newspaper like the WSJ will post that the ‘wow’ factor is wearing off because Nvidia is now beating analyst revenue projections by smaller amounts:
If a company keeps beating your expectations, that’s a failure of expectations, with that failure at least getting smaller over time.
Nvidia And The Circular Balance Sheet Trick
Many noticed that when Nvidia invests in OpenAI which uses the money to buy chips from Nvidia, and you move $100 billion back and forth, this can inflate valuations without obviously creating real value.
Such things have happened before.
You do have to watch out for the de facto Ponzi scheme, but this is not that, from a pure business perspective it is a clear win-win agreement. You also do have to watch out for a naively multiplying profits into a fixed P/E ratio, but the stock market is smarter than that. The only losers are other AI labs, Microsoft and humanity’s chances of survival.
Nvidia is a bet on AI happening, but in a safe fashion as they are doing it via reinvesting their profits rather than leverage. If AI valuations collapse, Nvidia only goes down roughly proportionally, and they remain a great company.
Partnership Potentially Preempts Policy Preferences
What about the implications of the deepening of the OpenAI-Nvidia partnership for policy, and willingness to speak out about existential risks?
A key worry has long been that OpenAI depends on Nvidia, so OpenAI may be reluctant to speak about AI risk lest they anger Nvidia and endanger their chip allocations. OpenAI has not been a great political actor, but Sam Altman appreciates many of the dangers and issues that will be at stake.
It is also fortunate that OpenAI’s interests in many ways align with America’s and with those of humanity, far more so than do the incentives of Nvidia. Nvidia wants to make and sell as many chips as possible to the widest variety of customers, whereas OpenAI has a strong interest in America beating China across the board and keeping things under control.
This situation could get far worse. Nvidia may even have made this part of the deal, with Altman agreeing to pursue or not pursue various political objectives, including around export controls. There is even the danger Nvidia ends up with a board seat.
It is likely not a coincidence that Anthropic has found a way to get its compute without Nvidia.
There is a matching bull case, on two fronts.
In terms of Nvidia impacting OpenAI, it is plausible that Nvidia already had maximal leverage over OpenAI via controlling chip allocations, especially if you add that it has the ear of the Administration. At some point, things can’t get any worse. And indeed, locking in these contracts could make things much better, instead, via locking in large commitments that reduce Nvidia’s real leverage.
Then there’s the question of OpenAI impacting Nvidia. Nvidia has a market cap of ~$4.3 trillion, but $100 billion is still ~2.3% of that, and the baseline scenario (since OpenAI is riskier and has more room to grow) is OpenAI equity grows in value faster than Nvidia equity, and Nvidia engages in stock buybacks. So even if no further investments are made, Nvidia could quickly be 5% or more exposed to OpenAI, on top of its exposure to xAI and other Western AI companies, where it makes a variety of strategic investments.
That starts to matter in the calculus for Nvidia. Yes, they can do better short term in their core chip business via selling to China, but that hurts their other investments (and America, but they clearly don’t care about America), and long term the model of compute-intensive top-quality closed models is better for Nvidia anyway.
Usually the incentives point the way you would think. But not always.