Musings on Reported Cost of Compute (Oct 2025)

Vladimir_Nesov

109 Musings on Reported Cost of Compute (Oct 2025)

24th Oct 2025

3 min read

109

There are many ways in which costs of compute get reported. A 1 GW datacenter site costs $10-15bn in the infrastructure (buildings, cooling, power), plus $30-35bn in compute hardware (servers, networking, labor), assuming Nvidia GPUs. Useful life of the infrastructure is about 10-15 years, and with debt financing a developer only needs to ensure it's paid off over those 10-15 years, which comes out at $1-2bn per year. For the compute hardware, the useful life is taken as about 5 years, which gives $6-7bn per year. Operational expenses (electricity, maintenance) are about $2.0-2.5bn per year.

In total, 1 GW of compute costs about $9-11bn per year, but whoever paid the compute hardware capex needs to ensure the payments continue for 5 years, so a contract for 1 GW of compute will be 5 years long, which makes it a single contract for at least $45-55bn, which might become $55-65bn to allow a profit for the cloud provider.

Thus 1 GW of compute could get reported as $10bn (infrastructure capex; or alternatively the compute costs for the AI company in a calendar year), as $30bn (compute hardware capex without labor costs), as $45bn (infrastructure plus compute hardware capex), or as $60bn (the total cost of the contract between the AI company and the cloud provider over 5 years).

A $300bn contract then might mean about 5 GW of total capacity, while a $27bn datacenter site could simultaneously mean 2 GW of total capacity. And a $300bn contract doesn't mean that the 5 GW of capacity will be built immediately, for example if only 2 GW is built initially, that requires the AI company to be capable of paying about $25bn per year (for 5 years), with the other 3 GW being contingent on the AI company's continuing growth.

Non-Nvidia Hardware

The Nvidia servers (1 GW all-in is about 5500 GB200 NVL72 servers with 72 chips each, or 400K chips in total) take up about $20bn of capex, so if Nvidia's margin of about 70% applies to this part (it's probably less, since GPUs are not all of the server), it comes out to $14bn per GW, or $2.8bn per year in a 5-year contract between an AI company and a cloud provider, about 25% of it. This suggests that non-Nvidia compute might only cost up to 25% less, all else equal (which it isn't), even though GPUs are usually portrayed as the majority of the cost of compute, and Nvidia's margin as the majority of the cost of the GPUs.

Thus a TPU contract for "tens of billions of dollars" and "over a gigawatt of capacity" admits an interpretation where it's a ~5-year contract at ~$12bn per year for ~1.2 GW of compute (in total power, not just IT power).

Model Sizes in 2026

If the new contract is for TPUv7 Ironwood, in 2026 Anthropic will have 1 GW of compute with 49 TB of HBM per 256-chip pod. This is comparable to OpenAI's Abilene site, which is 1 GW of compute with 14-20 TB of HBM per GB200/GB300 NVL72 rack, and will also be ready at this capacity in 2026. Currently Anthropic has access to Trainium 2 Ultra servers of AWS's Project Rainier with 6 TB of HBM per rack, while OpenAI's capacity is probably mostly in 8-chip Nvidia servers with 0.64-1.44 TB of HBM per server.

Feasible model size scales with HBM per server/rack/pod that serves as a scale-up world (especially for reasoning models and their training), so 2026 brings not just 10x more compute than 2024 (400K chips in GB200/GB300 NVL72 servers instead of 100K H100s), but also 10x larger models (in total parameter count). As the active params for compute optimal models scale with the square root of compute, this enables more sparsity in MoE models than 2024 compute did, and the number of active params for the larger models will no longer be constrained in practice by the 8-chip Nvidia servers (with 100K H100s, compute optimality asked for about 1T active params, which is almost too much for servers with 1 TB of HBM, and MoE models ask for multiple times more total params).

ComputeAI

Frontpage

109

New Comment

Rendering 0/11 comments, sorted by

top scoring

(show all) Click to highlight new comments since: Today at 6:17 AM

Moderation Log

109 Musings on Reported Cost of Compute (Oct 2025)

by Vladimir_Nesov

24th Oct 2025

3 min read

109

Non-Nvidia Hardware

Model Sizes in 2026

ComputeAI

Frontpage

109

New Comment

Rendering 0/11 comments, sorted by

top scoring

(show all) Click to highlight new comments since: Today at 6:17 AM

Moderation Log

More from Vladimir_Nesov

Curated and popular this week

11Comments

Comment Permalink

Łukasz Chajdaś1y120

Do we have reason to assume that MoE models will stretch to the entire rack memory capacity minus KV cache in batched inference? I ruminated earlier that Rubin Ultra could theoretically run 300T sparse models, but it looks recently like it's possible to squeeze much more from far fewer parameters and iterate much faster.

This data about compute spend at OpenAI suggests this trend: https://epoch.ai/data-insights/openai-compute-spend

At the same time, some Chinese open source models are downscaling in parameters. The most recent MiniMax M2 is much smaller than MiniMax M1. Qwen-Next 80B-A3B is close in performance to Qwen-3 235B-A22B. I wouldn't be surprised if the same happened with DeepSeek-V4.

It's not that models won't grow, but it feels like intelligence density per parameter and non-degrading sparsity thresholds will be tested in the search for maximum training and inference efficiency.

8Vladimir_Nesov1y

Experiments on smaller models (and their important uses in production) will continue, reasons they should continue don't affect the feasibility of there also being larger models. But currently there are reasons that larger models strain the feasibility of inference and RLVR, so they aren't as good as they could be, and cost too much to use. Also, a lot of use seems to be in input tokens (Sonnet 4.5 via OpenRouter processes 98% of tokens as input tokens), so the unit economics of input tokens remains important, and that's the number of active params, a reason to still try to keep them down even when they are far from being directly constrained by inference hardware or training compute. Prefill (input tokens) and pretraining are mostly affected by the number of active params, adding more total params on top doesn't make it worse (but improves model quality). For generation (decoding, output tokens) and RLVR, what matters is the time to pass total params and KV cache through compute dies (HBM in use divided by HBM bandwidth), as well as latency for passing through the model to get to the next token (which doesn't matter for prefill). So you don't want too many scale-up worlds to be involved, or else it would take too much additional time to move between them, and you don't care too much if the total amount of data (total params plus KV cache) doesn't change significantly. So if you are already using 1-2 scale-up worlds (8-chip servers for older Nvidia chips, 72-chip racks for GB200/GB300 NVL72, not-too-large pods for TPUs), and ~half of their HBM is KV cache, you don't lose too much from filling more of the other half with total params. It's not a quantitative estimate, as the number of scale-up worlds and the fractions used up by KV cache and total params could vary, but when HBM per scale-up world goes up 10x, this suggests that total param counts might also go up 10x, all else equal. And the reason they are likely to actually go there is that even at 5e26 FLOPs (1

1Łukasz Chajdaś1y

Thanks for the detailed response. If we assume that continued progress requires using much more compute on RL than pretraining, this could favor model sizes much smaller than Chinchilla optimal. It's possible that future models will be trained with 99% RL and will employ some sort of continuous learning architecture. I believe the number of active parameters will be determined by achieving similar load on compute and memory, which will depend on the current level of attention efficiency and the context necessary to hit agent ability targets.I don't know how sparse very large (>20T) models could be, but they could probably be extremely sparse (maybe 0.1% or less, where most of the volume is taken by a router). I think it's possible that current and planned AI compute designs are not even close to optimal for training methods that may be developed within a year or so. Assuming something like DeepSeek DSA is relatively close to maximum efficiency, and RL pathways produced by agents will stretch to many millions of tokens, it may be the case that HBM will finally be too slow, and using SRAM will be the only way forward. There's also a completely different consideration. For example, inclusionAI/Ring-1T has 1T total and 50B active parameters, and the creators claim it can achieve silver in IMO using a multi-agent framework, with plans to train it further to achieve gold, it's only 50b. As you said, we're probably entering compute ranges that enable training models with many T active parameters (assuming RL will be in roughly one-to-one proportion to pretraining). The question is whether pattern density in our environment is currently large enough to make use of it."

Vladimir_Nesov1y50

Compute optimality makes sense for RL as much as for pretraining. A model that's too large won't see much data, and a model that's too small won't be able to learn well even from the correspondingly larger amount of data. So it's a quantitative question, where compute optimality for RLVR happens to be (compared to pretraining). The papers are still hashing out stable/scalable RL training, rather than the tradeoffs of RL training constrained by fixed total compute (under varying model size specifically).

See in context