I estimate that DeepSeek-R1-Zero's RL training used ~2e22 (1.5e22-3e22) FLOPs, 30x less than Ege Erdil's estimate of 6.1e23 in the Epoch AI newsletter, mainly due to a batch size correction in the estimate.
I derive a more precise formula for RL compute, accounting for long-context attention costs.
DeepSeek reports that the training run cost ~$200k of GPU time, only 5x less than the previous estimate of $1M.
RL ran at a significantly lower MFU than pre-training (~3% vs 16% respectively). If future estimates don't account for this, FLOP counts during RL may understate the actual costs of training.
I conclude by discussing the implications for RL compute forecasts and thresholds, and outline a research agenda to better track RL compute and investigate scaling laws.
Overview
On January 20th, 2025, the Chinese AI lab DeepSeek released their flagship open-weight reasoning model, DeepSeek-R1. Its performance was comparable to OpenAI’s o1 — a groundbreaking reasoning model announced 4 months prior. As a result, DeepSeek attracted widespread media attention and discussion around the gap between open and closed-weight models and how much its training cost.
A key enabler for DeepSeek-R1’s performance over its base DeepSeek-V3 was the additional resources spent post-training the model with Reinforcement Learning (from Verifiable Rewards), RL(VR). In this stage, the model learns to reason by completing tasks and updating its weights based on which tasks it successfully completed or not.
Shortly after release, a post by Ege Erdil on Epoch AI’s newsletter attempted to estimate the compute used in training DeepSeek-R1. It calculates that pre-training used 3e24 FLOPs at a cost of $5.3 million, while RL required 6.1e23 FLOPs[1]at a cost of $1 million.
However, Erdil’s post borrows a hyperparameter from the earlier DeepSeekMath paper — batch size, which it noted as a large source of uncertainty — which resulted in overestimating RL FLOPs by 64x due to a notational discrepancy. Further, the DeepSeek team has since provided more information about R1’s training in an updated Nature paper released September 2025, enabling more up-to-date estimates to be made.
In this post — using an improved methodology and newer data — I estimate that DeepSeek-R1-Zero used ~2e22 FLOPs for its RL training, 30x less than the original estimate. DeepSeek have also reported training costs: R1-Zero’s RL used 101k GPU hours ($202k at $2/GPU-hour), 5x less than the original $1M estimate. When later stages are included, R1’s full post-training cost about $300k of GPU time, only ~3x lower than Erdil’s estimate.
This gap in FLOP vs cost revisions (~30x vs ~5x) exists because the original estimate assumed RL ran at pre-training utilisation ranges (~16%), whereas I find it was ~3%, 5-6x lower. In practice, this means FLOP counts will systematically underestimate the cost of training as RL becomes dominant, with effects on how we interpret compute trends and set compute-based regulatory thresholds.
I end by discussing the implications of this revision and the broader state of knowledge on the compute scales of RL training.
K: Data reuse factor — number of optimisation steps per batch of rollouts.
T: Average rollout length (in tokens).
B: Batch size — number of questions sampled during each RL step.
G: GRPO group size — number of rollouts generated per question.
They take Nₛₜₑₚₛ = 8000, K = 1, T = 4000 based on graphical estimates in the original paper. Since batch sizes weren’t released at the time, they set the parameters B = 1024 and G = 64 according to a related DeepSeekMath paper (one of DeepSeek’s earlier forays into reasoning models).
Batch Size Mismatch
DeepSeekMath paper
The main difference is in the value of the ‘batch size’ (B) chosen. In the DeepSeekMath paper, they appear to adopt the convention where the batch size refers to the total number of rollouts per RL step (reported as a value of 1024), which using Erdil’s notation equals the value of B × G (no. questions per step × no. rollouts per question). This convention stands in contrast with the post’s, where the batch size is intended to mean the number of questions per RL step.[2]
As a result of this difference, Erdil used the paper’s reported batch size value of B = 1024 in its calculations, rather than the corresponding B = 16.[3]
This is what leads to the 64x overestimate, and once corrected, their formulas would estimate that 9.5e21 FLOPs were used in RL training.
Further notes on the original estimate can be found in Appendix B.
Improved Estimates
To improve on these estimates, we can first begin by using the information detailed in the updated DeepSeek-R1 paper (Sec 2.1) to gather more accurate hyperparameter values.
We have that Nₛₜₑₚₛ = 10400, K = 1, B = 32, G = 16. Based on their Figure 1b, I also assume a higher mean response length, Tᵣ, of 7000 tokens[4]and treat prompt tokens as negligible.
To calculate the FLOPs for DeepSeek R1-Zero’s RL training, I propose the following formula:
Here, N is the number of active parameters, and Cₐ and Cₙ are the FLOPs used in attention costs per token, for the “absorbed” and “naive” implementations of DeepSeek’s MLA attention mechanism. At our chosen response length, attention contributes a third of the total FLOPs.
A derivation and justification of this formula can be found in Appendix A below this post.
Updated Figures
With this new formula, we can use the aforementioned hyperparameter values to produce new FLOP estimates.
Using Nₛₜₑₚₛ = 10400, K = 1, B = 32, G = 16, T = 7000 tokens, N = 37 billion, we get:
Overall, this revised estimate of 2e22 FLOPs is 30x lower than their original estimate of 6.1e23 FLOPs.[5]
Hardware Utilisation and Costs
Another important piece of information provided in the updated R1 paper was how many GPU-hours were used in training the models.
The DeepSeek-R1 paper reported the use of 512 (64×8) H800 GPUs for 198 hours in training DeepSeek-R1-Zero (Appendix B.4.4). Assuming the model was trained in FP8 precision to match the pre-training of DeepSeek-V3, the H800 SXM GPUs DeepSeek use have a reported computing power of ~1900 TeraFLOPS.[6]Multiplying the figures, we can calculate the theoretical compute capacity used in training as 6.9e23 FLOPs:
They also report lower compute costs of such training at $202k (101k GPU hours, at $2 an hour) compared with Erdil’s original estimates of $1M.
As a sanity check, this implies that the RL training process had a ~3% Model FLOP Utilisation (MFU)[7][8], which isn’t entirely unrealistic for RL runs — more on this below.
This also strongly supports the idea that the original 6.1e23 FLOP estimate was an overestimate as it would imply an unrealistic ~88% MFU.
Note: Further compute and GPU-hours went into SFT data generation and later post-training to produce the final R1 model. I’m not accounting for this here as it contributes to a smaller fraction of the reported post-training compute than the initial RLVR stage, and is more difficult to count FLOPs for. It was reported that this stage increased the GPU hours required by about 50% (with post-training costs totalling at 147K GPU-hours or $294k).
Implications
To summarise, here’s a table of the RL compute estimates and methods discussed in this post so far.
Calculation
RL FLOPs
Changes
Original estimate
6.1e23
Original article’s estimate
Batch size correction
9.5e21
Removed factor of 64
Original formula with updated parameters
1.1e22
Updated training steps, batch size, and response length
New estimates
2e22 (1.5e22–3e22)
Factored in detailed attention calculations
Effects on Reinforcement Learning and Compute Allocation Forecasts
Considering the amount of compute used to pre-train the base DeepSeek V3 model, 3e24, only 0.7% of the model’s compute (in FLOPs) was required for R1-Zero’s RL training. This is a much smaller fraction than the ~20% implied by the original estimates.
If you instead consider the fraction of overall GPU-hours or actual training costs the RL training required, RL accounted for a higher 3.5% of the compute budget/cost.[9]Once you include the cost of other RL/post-training stages, this rises to 5% of the compute budget.[10]
These figures are important to update since forecasts, like this Epoch newsletter by Josh You, “How Far Can Reasoning Models Scale,” rely on them to make projections on how they expect the compute used for RL to grow over time and reach comparable scale to pre-training’s compute.
As DeepSeek is one of few models with this degree of transparency about their post-training setup, it serves as a significant anchor point for these projections and relevant sources might consider revisiting their estimates. That said, because the actual GPU-hours/training cost — which is the more real-world relevant constraint than FLOPs — was revealed to be ~5x lower than initially estimated (despite a 30x decrease in computations) it may not substantially affect projections. In the aforementioned newsletter, this correction just shifts back their projections for RL matching pre-training compute scales by ~2–4 months.[11]
Low Model FLOP Utilisation
Where hyperparameters and information about the training dataset aren’t available, another way to approach estimates for RL compute is with knowledge of how many GPUs were used and how long they were run for — this is occasionally used to estimate pretraining compute for closed models. To calculate the FLOPs, you need to guess what fraction of the computing power went to useful operations (that is, the MFU).
In pre-training, the MFU of a training run typically varies between15–40%, with DeepSeek-V3’s pre-training having an estimated 16% MFU[12]. However, RL training is less hardware efficient due to factors such as data transfer during inference or idle periods waiting for rollouts. RL MFUs aren’t widely reported and measurement methods vary but typical values may lie between 1% (Olmo3, Table 23, inference MFU) and 20% (veRL on DeepSeek-671B), depending on how well-optimised the RL setup is. From the data I’ve been able to gather for large runs I believe it generally lies in the single digits, but more robust data is necessary.
When estimating the cost of GPUs for R1’s training, the original post assumes GPU utilisation similar to pre-training, treating 5% as an unlikely case:
The MFU during reinforcement learning would have to be 5x worse than in pretraining, around 5%, for the cost of RL to even be comparable to the cost of pretraining when measured in GPU hours.
However, my estimates and the reported GPU hours imply the MFU during R1-Zero’s training is only around 3%. This said, Erdil’s conclusion remains valid. Since the FLOPs were significantly overestimated, the errors partly cancel out, leaving RL to account for only 3.5% of overall training costs.
Overestimating the MFU results in inflated compute/FLOP estimates when only the GPU-hours are known.
Josh You’s “How Far Can Reasoning Models Scale” post, makes a similar optimistic assumption when estimating the RL training compute for Llama-Nemotron Ultra 253B. It assumes “a 30% utilization rate (large-scale training tends to be 30-40% but utilization rates for RL reasoning could be different).” To check how sensitive these figures are to MFU, if you choose a utilisation rate closer to DeepSeek’s calculated 3%, their compute estimate directly decreases by an order of magnitude.
Conversely, this leads to underestimates of the compute costs of an RL run just given its FLOP count. To use DeepSeek as an example, note how only 0.7% of the FLOPs required for training DeepSeek-R1-Zero were used during RL, whereas this accounted for a larger 3.5% of the GPU runtime. Another case where I’d speculate this applies is the recently announced Beam model from Reflection AI; there, significantly (1.7x) more GPUs were used for the RL training (10,500) than pre-training (6,144) despite the RL stage seeming to train the model on fewer tokens.[13]
This MFU gap between training stages casts into doubt the assumed correspondence between the FLOPs required and the actual costs of a training stage, and is important to keep in mind when reporting or collecting data about runs. My recommendation is that when reporting information about the scale of an RL run, the GPU/accelerator-hours used, and the estimated MFU or FLOPs required to complete the run should both be reported.
The problem is that lower MFUs mean you gain fewer useful FLOPs per GPU-hour or dollar spent on RL. Using DeepSeek’s figures as an example (where there’s a 5x MFU gap between pre-training and RL), a run whose GPU-hours are equally split between the two stages executes 60% of the FLOPs of a pre-training only run on the same GPU-budget, reaching as low as 20% when RL dominates. So as RL takes a growing share of compute budgets, FLOP counts will increasingly underestimate the cost of training runs. This should cause us to reconsider how we report training costs,interpret compute trends, and set regulatory thresholds as RL becomes more dominant.
Future work: A Reinforcement Learning Compute Research Agenda
We have well established correlations between the scale of pre-training and model performance — scaling laws — along with the emergence of test-time compute trends. As RL accounts for an increasing fraction of training compute and model capabilities, it may be useful to develop a similar understanding there. So far, there has been a lack of experiments and compiled datasets to investigate this, but in this section I aim to lay out some steps that may help.
Two key problems with attempts to understand how far RL can scale and subsequently how this affects AI progress and compute-based regulations, are:
A lack of transparency from labs, both closed and open-weight developers about their RL setups and scale. From a regulatory perspective, it’s also important to decide what factors (model size, architecture, training steps, total tokens) about RL runs are worth sharing to inform the public.
No consistent methodology to rely on to make grounded estimates with the evidence we have. As RL is new and has significant room for setups to vary, I haven’t encountered attempts to lay out a standardised and sufficiently descriptive (accounting for long-context attention mechanisms) compute formula, unlike the standard 6ND formula used for pre-training compute.[14]
We Need a Reinforcement Learning Compute Database
Once a methodology has been decided, it would be useful to compile a database of all open (and occasionally closed) source models which have released sufficient data to make reasonable estimates of their RL compute budgets. The dataset for this isn’t large, but there is a growing number of open source model releases with a high degree of transparency about their RL training such as the Olmo/Tulu models from AllenAI, MiMo v2.6 from Xiaomi, or Beam by Reflection AI.
An Epoch-style visualisation or article tracking RL compute scales of reported models would be an ideal outcome of this step.
We Lack Robust Scaling Laws for RL
As mentioned at the start of the section, we have a poor public understanding of how scaling up an RL training run improves a model’s capabilities. Ideally, we’d have something similar to pre-training scaling laws: a validated relationship between RL compute and model capability across a range of tasks and training setups.
At the moment we can only rely on very noisy signals. Frontier labs have strongly indicated they observed clear scaling trends with their models, but don’t reveal specifics. We can extrapolate trends based on public data, such as in Toby Ord’s analysis of OpenAI’s o-series and RL scaling, but it’s hard to make robust conclusions given limited RL compute data with models reporting on different benchmarks. Finally, large-scale RL experiments are potentially our best sources of data, but can be prohibitively expensive to investigate with frontier-scale models, and the few examples of these so far find results are highly dependent on training recipes and don’t draw explicit comparisons to pre-training scaling laws (see: ScaleRL, [2509.25300], and [2603.12151]). Open post-training labs like Trillium or AllenAI may be well positioned to do useful work on these questions.
In a series of future posts, I intend to contribute to these questions by:
Breaking down RL and its associated hardware demands in detail, then laying out a potential starting point for a set of RL compute estimation methodologies.
Starting to compile a dataset of RL compute based on what’s available from open-source models.
Discussing the publicly available data on how scaling RL improves performance (initial data collection here). Then scope out what a set of more informative experiments and investigations into this topic looks like, and potentially run some preliminary experiments.
If you have any interest in supporting, collaborating, or advising me with any of these research directions; have ideas for helpful sources of model data; or otherwise want to chat and have further questions about this post, please reach out! Feel free to leave your thoughts on this post in the comments, or email me at obayomijo@gmail.com.
Acknowledgements
Thank you to Josh You at Epoch AI for double-checking my early calculations, initial encouragement to write this post, and helpful comments on my drafts and framing.
Thanks to Sean Fillingham and Robi Rahman for helpful discussions and comments on my drafts.
AI Usage: All writing in this post is mine. LLMs (mainly Claude Opus 5/5.5) were used to proofread for errors in my calculations/text, read chart data for the precise estimates, and format the post’s LaTeX and summary table.
Appendix A: Formula Derivation
The standard formula for the FLOPs used in matrix multiplications (matmuls) during a rollout (excluding dot-product attention) is given by 2NT, where N is the number of active parameters. The attention FLOPs calculated below are an addition to this baseline formula to give the total FLOPs during rollout generation.
For this derivation, I’m primarily relying on my own analysis (some details I haven’t fully explained here and will leave to a future post) and otherwise refer the reader to DeepSeek’s analysis of their MLA algorithm as a sanity check. I’ll introduce the following notation and their corresponding values for DeepSeek-V3/R1:
Tₚ: Prompt length in tokens.
Tᵣ: Response length in tokens.
T: Rollout length, defined as Tₚ + Tᵣ.
S: Sequence length (number of tokens in context before the current token).
L: Number of transformer layers. 61.
H: Number of query heads. 128.
rₖᵥ: Dimension of compressed KV vectors. 512.
Dₖ: Dimension of key vectors/heads. 128.
Dᵥ: Dimension of value vectors/heads. 128.
Dᵣ: RoPE embedding dimension. 64.
Naive vs Absorbed MLA FLOPs
For a standard (“naive”) MLA implementation, the attention FLOPs per token (across all layers) are:
The last term slightly changes for an “absorbed” MLA, a slight variation to the algorithm where matrices are multiplied in different orders to save memory at the cost of additional computation.
This “absorbed MLA” algorithm is what DeepSeek reports to have used for their inference / rollout generation[15]. With this the formula becomes:
For the rest of this derivation, I’ll continue referring to this formula, although the former remains relevant for putting together a final calculation.
If we let the product of constants be represented by Cₐ = 2LHDₐ (and likewise Cₙ = 2LHDₙ), and substitute the hyperparameter values DeepSeek-V3/R1 uses, this gives the values stated earlier:
The formula for attention FLOPs per token (at position S) becomes S · Cₐ.
Note: Since S · Cₐ is the number of attention FLOPs per generated token at position S and 2N is the number of matmul FLOPs per token, you can calculate at what context length the dot-product attention FLOPs exceed the other matrix multiplications during decoding. This happens when:
This aligns with a Twitter/X post made around the time of release.
Summing FLOPs Over a Rollout
To get the FLOPs for a single completed response/rollout you could naively multiply the above expression by T, and set S = T. However, this will lead to a 2x overestimate for FLOPs since the value of S increases from Tₚ to T − 1 as the rollout is generated.
Instead, if we multiply the expression by T, and set S = T/2 — treating S as the average sequence length over the whole rollout generation process — this closely approximates the number of FLOPs a more careful calculation with prompt and response lengths summed over the generated tokens would give.
Overall, this gives us the following formulas for the number of dot-attention FLOPs per rollout, and for the total FLOPs per rollout including dot-product attention:[16]
Trainer / Backpropagation FLOPs
The calculations are very similar for backpropagation with two changes.
For weight updates, each matrix multiplication must be done thrice to calculate activations then gradients for backpropagation as a result of the chain rule. Multiplying the baseline and attention FLOPs by 3x.
During backpropagation, the growing KV cache memory and loading is less of a constraint as it is during inference. So the DeepSeek team could afford to use the more memory-intensive, lower compute “naive” MLA attention calculations to train more quickly.
Overall, this gives the FLOPs per update on a rollout as:
Reference Model
One addition is that in DeepSeek-R1’s training setup, they run a reference model (an older copy of the model’s weights) to calculate the KL penalty for the model being trained. This requires an additional forward pass on the model, and although it isn’t specified, I assume it similarly uses the absorbed MLA implementation on inference nodes. However changing this assumption to a naive MLA implementation would reduce the overall estimate by just under 10%.
By summing the total FLOP counts, and multiplying by the number of rollouts per batch, group size and number of RL steps we arrive at the earlier formula:
where Cₐ = 16,990,208 and Cₙ = 4,997,120.
Note: Oversampling of rollouts may result in more FLOPs being used than estimated here. Typically, more rollouts are generated than are actually used for weight updates, as a result of low-scoring/quality rollouts being filtered out.
One would also want to sum the FLOPs calculated over segments of an RL run for precision, as response lengths tend to increase during training in a way not represented by the mean.
Overall this is still an approximate formula with uncertainties I haven’t fully quantified in either direction.
In a future post, I aim to provide a more granular break-down of the formulas behind calculating FLOPs across various attention mechanisms and lay out a methodology for RL compute estimation.
Appendix B: Further Notes on the Original Formula
Besides batch sizes, here are some of my criticisms/notes on the formula used to compute RL FLOPs.
In general, RL can be broken down into two phases: the rollout (or inference) phase, and the update (or trainer) phase.
By default, we tend to assume models use 2 FLOPs/param/token during rollout generation, and 6 FLOPs/param/token during weight updates, giving a total 8 FLOPs/param/token during each RL step.
We add an additional 6 FLOPs/param/token when more than one gradient update/epoch is performed within an RL step on the same data (when K > 1).
This motivates a formula of:
As described, this formula only includes the core matrix multiplications during a forward/backward pass of a LLM. It excludes the cost of the dot-product attention calculations, which across long prompts/sequences (>5,000 tokens for DeepSeek-V3/R1 with MLA – a particularly compute-intensive attention algorithm) can double the FLOPs/token during rollout generation. The post acknowledges this and bakes it into their calculations such that rollouts are 4 FLOPs/token.
The post uses 4 FLOPs/param/token during the update step of RL, which accounts for the core backpropagation steps. This doesn’t include the additional forward pass required during the update step to compute the necessary activations for backpropagation, which increases update compute to 6 FLOPs/param/token.[17]
Further, the post considers the cost of attention during backprop “not substantial” by leveraging a less compute-intensive attention mechanism (at the cost of more memory). However, as derived in Appendix A, the compute required is significant enough to contribute ~1–2 additional FLOPs at 5–10k sequence lengths, raising updates to 8 FLOPs/param/token.
It also leaves out the additional forward passes done by the reference model in the RL setup. Adding an additional 2-4 FLOPs/param/token.
Baking in these additional costs, we’d get that the formula’s first term should be closer to 16 FLOPs/param/token, compared with the original 8. This change leads to a near doubling of the compute estimates relative to the original values.
FLOP stands for Floating Point Operation. In essence, FLOPs count the number of basic operations (multiplications, additions) required to perform a calculation.
DeepSeek’s updated R1 paper explicitly adopts the former convention, stating: “Each training step consists of 32 unique questions, resulting in a training batch size of 512.”
I originally obtained this figure by visual estimation. For a more precise mean, we can integrate under the graph of tokens against training steps provided in the paper and divide by the number of steps, obtaining a mean of 6100 tokens.
For simplicity, I’ll stick with the original value as the effective response length. The FLOPs with this length match those of a more precise method that accounts for increasing response lengths during the training run (addressed in the next footnote).
To account for the changing response lengths during the run, I calculated the FLOPs on each 500 step interval and summed them together. This gave a final estimate of 1.9e22 FLOPs (LLM-assisted; see this document for the breakdown).
Some sources of uncertainty in my estimates are: the variance in response lengths within a batch (with coefficient of variation of up to 1 from my experiments), oversampling, recomputing off-policy probabilities, and other assumptions in my formula. Overall, I expect the true value to lie in a range between 1.5e22-3e22.
The exact precision at which RL was performed in the paper wasn’t given, so it may be the case the model was trained in FP16/BF16, or mixed precision. If it is, this halves the computing power of the GPUs and decreases the theoretical compute capacity to 3.5e23, increasing the MFU during RL to 5.8%.
MFU is a measure of how well the training run is utilising the actual hardware resources it’s given. In its simplest form, it’s the estimated useful FLOPs for training / theoretical max FLOPs the hardware can perform during the training run.
DeepSeek V3’s technical report stated they used 2.66 million H800 hours or $5.3M, with overall training compute costs totalling at 2.8 million H800 hours or $5.6M.
The share of the compute budget used is significantly greater than the share of actual calculations required for RL due to the lower MFU / compute utilisation of RL when compared with pre-training.
Josh You’s post assumes a 10x growth in RL training compute every 3–5 months. Assuming a 5x lower baseline when considering training costs, this sets their estimate back log₁₀(5) × 3–5 = 2–3.5 months. If instead the projections were anchored on the increase in FLOPs, this would be a log₁₀(30) × 3–5 ~= 4-7 months setback. To give an example of things being on trend: based on livestream charts, in 2025, xAI seemingly trained Grok-4 with an equivalent amount of RL compute as pre-training compute, both being the largest publicly estimated runs at the time.
Assuming 3e24 FLOPs were used in pre-training DeepSeek-V3. Given 2.66 million GPU-hours were used, then we have MFU = 3e24 / (2.66×10⁶ × 3600 × 1.9e15) ~=16% This differs from Erdil’s 23% estimate as I assume DeepSeek use the SXM chips rather than the PCIe, which have different throughputs.
Reflection AI reported running 6144 vs 10500 GB300 GPUs for 4 weeks to complete pre-training and RL respectively. They also report that Beam was pre-trained on 23.8T tokens. During RL they report 100m rollouts were generated and if I assume a long 100k token mean response length, ~10T tokens were generated during RL. So pre-training used 2.4x the tokens. The actual compute required in both stages will be closer given the costs of long-context attention in RL, but the general point about lower RL MFUs (even after Beam made substantial improvements to the compute efficiency of their RL setup) resulting in the use of more hardware than pre-training seems reasonable (more careful calculations are needed to confirm this).
Lacking a shared methodology and replication procedure makes mistakes harder to notice and correct. It also increases the difficulty of compiling reliable datasets and building on previous estimates. Discussion of some benefits of this here.
I will note that for rollout generation, DeepSeek reports using self-speculative decoding. As this involves additional forward passes, it increases the FLOPs required, although I don’t account for this and don’t expect it to notably affect the FLOPs.
Because the FLOPs depends on T quadratically, a more thorough calculation would quantify the uncertainty introduced by variance in response lengths within a batch, as this would increase the estimates. However no such measurements were reported and I decided to exclude this from my formula, but account for it in the uncertainty ranges.
The activations generated during the rollout phase often can’t be reused to reduce this computation. This is due to greater memory demands, or mismatches between the trainer and rollout specs, where rollouts from previous versions of the model may be used for weight updates.
[Crossposted from my Substack.] An update to Ege Erdil’s “What went into training DeepSeek-R1?”
TL;DR
Overview
On January 20th, 2025, the Chinese AI lab DeepSeek released their flagship open-weight reasoning model, DeepSeek-R1. Its performance was comparable to OpenAI’s o1 — a groundbreaking reasoning model announced 4 months prior. As a result, DeepSeek attracted widespread media attention and discussion around the gap between open and closed-weight models and how much its training cost.
A key enabler for DeepSeek-R1’s performance over its base DeepSeek-V3 was the additional resources spent post-training the model with Reinforcement Learning (from Verifiable Rewards), RL(VR). In this stage, the model learns to reason by completing tasks and updating its weights based on which tasks it successfully completed or not.
Shortly after release, a post by Ege Erdil on Epoch AI’s newsletter attempted to estimate the compute used in training DeepSeek-R1. It calculates that pre-training used 3e24 FLOPs at a cost of $5.3 million, while RL required 6.1e23 FLOPs[1]at a cost of $1 million.
However, Erdil’s post borrows a hyperparameter from the earlier DeepSeekMath paper — batch size, which it noted as a large source of uncertainty — which resulted in overestimating RL FLOPs by 64x due to a notational discrepancy. Further, the DeepSeek team has since provided more information about R1’s training in an updated Nature paper released September 2025, enabling more up-to-date estimates to be made.
In this post — using an improved methodology and newer data — I estimate that DeepSeek-R1-Zero used ~2e22 FLOPs for its RL training, 30x less than the original estimate. DeepSeek have also reported training costs: R1-Zero’s RL used 101k GPU hours ($202k at $2/GPU-hour), 5x less than the original $1M estimate. When later stages are included, R1’s full post-training cost about $300k of GPU time, only ~3x lower than Erdil’s estimate.
This gap in FLOP vs cost revisions (~30x vs ~5x) exists because the original estimate assumed RL ran at pre-training utilisation ranges (~16%), whereas I find it was ~3%, 5-6x lower. In practice, this means FLOP counts will systematically underestimate the cost of training as RL becomes dominant, with effects on how we interpret compute trends and set compute-based regulatory thresholds.
I end by discussing the implications of this revision and the broader state of knowledge on the compute scales of RL training.
Original Estimates
In the post by Ege Erdil, RL compute is estimated with the following calculations:
Where the variables denote the following:
They take Nₛₜₑₚₛ = 8000, K = 1, T = 4000 based on graphical estimates in the original paper. Since batch sizes weren’t released at the time, they set the parameters B = 1024 and G = 64 according to a related DeepSeekMath paper (one of DeepSeek’s earlier forays into reasoning models).
Batch Size Mismatch
DeepSeekMath paper
The main difference is in the value of the ‘batch size’ (B) chosen. In the DeepSeekMath paper, they appear to adopt the convention where the batch size refers to the total number of rollouts per RL step (reported as a value of 1024), which using Erdil’s notation equals the value of B × G (no. questions per step × no. rollouts per question). This convention stands in contrast with the post’s, where the batch size is intended to mean the number of questions per RL step.[2]
As a result of this difference, Erdil used the paper’s reported batch size value of B = 1024 in its calculations, rather than the corresponding B = 16.[3]
This is what leads to the 64x overestimate, and once corrected, their formulas would estimate that 9.5e21 FLOPs were used in RL training.
Further notes on the original estimate can be found in Appendix B.
Improved Estimates
To improve on these estimates, we can first begin by using the information detailed in the updated DeepSeek-R1 paper (Sec 2.1) to gather more accurate hyperparameter values.
Updated DeepSeek-R1 paper
We have that Nₛₜₑₚₛ = 10400, K = 1, B = 32, G = 16. Based on their Figure 1b, I also assume a higher mean response length, Tᵣ, of 7000 tokens[4]and treat prompt tokens as negligible.
To calculate the FLOPs for DeepSeek R1-Zero’s RL training, I propose the following formula:
Here, N is the number of active parameters, and Cₐ and Cₙ are the FLOPs used in attention costs per token, for the “absorbed” and “naive” implementations of DeepSeek’s MLA attention mechanism. At our chosen response length, attention contributes a third of the total FLOPs.
A derivation and justification of this formula can be found in Appendix A below this post.
Updated Figures
With this new formula, we can use the aforementioned hyperparameter values to produce new FLOP estimates.
Using Nₛₜₑₚₛ = 10400, K = 1, B = 32, G = 16, T = 7000 tokens, N = 37 billion, we get:
Overall, this revised estimate of 2e22 FLOPs is 30x lower than their original estimate of 6.1e23 FLOPs.[5]
Hardware Utilisation and Costs
Another important piece of information provided in the updated R1 paper was how many GPU-hours were used in training the models.
The DeepSeek-R1 paper reported the use of 512 (64×8) H800 GPUs for 198 hours in training DeepSeek-R1-Zero (Appendix B.4.4). Assuming the model was trained in FP8 precision to match the pre-training of DeepSeek-V3, the H800 SXM GPUs DeepSeek use have a reported computing power of ~1900 TeraFLOPS.[6]Multiplying the figures, we can calculate the theoretical compute capacity used in training as 6.9e23 FLOPs:
They also report lower compute costs of such training at $202k (101k GPU hours, at $2 an hour) compared with Erdil’s original estimates of $1M.
As a sanity check, this implies that the RL training process had a ~3% Model FLOP Utilisation (MFU)[7][8], which isn’t entirely unrealistic for RL runs — more on this below.
This also strongly supports the idea that the original 6.1e23 FLOP estimate was an overestimate as it would imply an unrealistic ~88% MFU.
Note: Further compute and GPU-hours went into SFT data generation and later post-training to produce the final R1 model. I’m not accounting for this here as it contributes to a smaller fraction of the reported post-training compute than the initial RLVR stage, and is more difficult to count FLOPs for. It was reported that this stage increased the GPU hours required by about 50% (with post-training costs totalling at 147K GPU-hours or $294k).
Implications
To summarise, here’s a table of the RL compute estimates and methods discussed in this post so far.
Calculation
RL FLOPs
Changes
Original estimate
6.1e23
Original article’s estimate
Batch size correction
9.5e21
Removed factor of 64
Original formula with updated parameters
1.1e22
Updated training steps, batch size, and response length
New estimates
2e22 (1.5e22–3e22)
Factored in detailed attention calculations
Effects on Reinforcement Learning and Compute Allocation Forecasts
Considering the amount of compute used to pre-train the base DeepSeek V3 model, 3e24, only 0.7% of the model’s compute (in FLOPs) was required for R1-Zero’s RL training. This is a much smaller fraction than the ~20% implied by the original estimates.
If you instead consider the fraction of overall GPU-hours or actual training costs the RL training required, RL accounted for a higher 3.5% of the compute budget/cost.[9]Once you include the cost of other RL/post-training stages, this rises to 5% of the compute budget.[10]
These figures are important to update since forecasts, like this Epoch newsletter by Josh You, “How Far Can Reasoning Models Scale,” rely on them to make projections on how they expect the compute used for RL to grow over time and reach comparable scale to pre-training’s compute.
As DeepSeek is one of few models with this degree of transparency about their post-training setup, it serves as a significant anchor point for these projections and relevant sources might consider revisiting their estimates. That said, because the actual GPU-hours/training cost — which is the more real-world relevant constraint than FLOPs — was revealed to be ~5x lower than initially estimated (despite a 30x decrease in computations) it may not substantially affect projections. In the aforementioned newsletter, this correction just shifts back their projections for RL matching pre-training compute scales by ~2–4 months.[11]
Low Model FLOP Utilisation
Where hyperparameters and information about the training dataset aren’t available, another way to approach estimates for RL compute is with knowledge of how many GPUs were used and how long they were run for — this is occasionally used to estimate pretraining compute for closed models. To calculate the FLOPs, you need to guess what fraction of the computing power went to useful operations (that is, the MFU).
In pre-training, the MFU of a training run typically varies between 15–40%, with DeepSeek-V3’s pre-training having an estimated 16% MFU[12]. However, RL training is less hardware efficient due to factors such as data transfer during inference or idle periods waiting for rollouts. RL MFUs aren’t widely reported and measurement methods vary but typical values may lie between 1% (Olmo3, Table 23, inference MFU) and 20% (veRL on DeepSeek-671B), depending on how well-optimised the RL setup is. From the data I’ve been able to gather for large runs I believe it generally lies in the single digits, but more robust data is necessary.
When estimating the cost of GPUs for R1’s training, the original post assumes GPU utilisation similar to pre-training, treating 5% as an unlikely case:
However, my estimates and the reported GPU hours imply the MFU during R1-Zero’s training is only around 3%. This said, Erdil’s conclusion remains valid. Since the FLOPs were significantly overestimated, the errors partly cancel out, leaving RL to account for only 3.5% of overall training costs.
Overestimating the MFU results in inflated compute/FLOP estimates when only the GPU-hours are known.
Josh You’s “How Far Can Reasoning Models Scale” post, makes a similar optimistic assumption when estimating the RL training compute for Llama-Nemotron Ultra 253B. It assumes “a 30% utilization rate (large-scale training tends to be 30-40% but utilization rates for RL reasoning could be different).” To check how sensitive these figures are to MFU, if you choose a utilisation rate closer to DeepSeek’s calculated 3%, their compute estimate directly decreases by an order of magnitude.
Conversely, this leads to underestimates of the compute costs of an RL run just given its FLOP count. To use DeepSeek as an example, note how only 0.7% of the FLOPs required for training DeepSeek-R1-Zero were used during RL, whereas this accounted for a larger 3.5% of the GPU runtime. Another case where I’d speculate this applies is the recently announced Beam model from Reflection AI; there, significantly (1.7x) more GPUs were used for the RL training (10,500) than pre-training (6,144) despite the RL stage seeming to train the model on fewer tokens.[13]
This MFU gap between training stages casts into doubt the assumed correspondence between the FLOPs required and the actual costs of a training stage, and is important to keep in mind when reporting or collecting data about runs. My recommendation is that when reporting information about the scale of an RL run, the GPU/accelerator-hours used, and the estimated MFU or FLOPs required to complete the run should both be reported.
The problem is that lower MFUs mean you gain fewer useful FLOPs per GPU-hour or dollar spent on RL. Using DeepSeek’s figures as an example (where there’s a 5x MFU gap between pre-training and RL), a run whose GPU-hours are equally split between the two stages executes 60% of the FLOPs of a pre-training only run on the same GPU-budget, reaching as low as 20% when RL dominates. So as RL takes a growing share of compute budgets, FLOP counts will increasingly underestimate the cost of training runs. This should cause us to reconsider how we report training costs, interpret compute trends, and set regulatory thresholds as RL becomes more dominant.
Future work: A Reinforcement Learning Compute Research Agenda
We have well established correlations between the scale of pre-training and model performance — scaling laws — along with the emergence of test-time compute trends. As RL accounts for an increasing fraction of training compute and model capabilities, it may be useful to develop a similar understanding there. So far, there has been a lack of experiments and compiled datasets to investigate this, but in this section I aim to lay out some steps that may help.
We Need a Compute Estimation Methodology for RL
My aim here is essentially making it easy to “Report Your Compute” for RL.
Two key problems with attempts to understand how far RL can scale and subsequently how this affects AI progress and compute-based regulations, are:
We Need a Reinforcement Learning Compute Database
Once a methodology has been decided, it would be useful to compile a database of all open (and occasionally closed) source models which have released sufficient data to make reasonable estimates of their RL compute budgets. The dataset for this isn’t large, but there is a growing number of open source model releases with a high degree of transparency about their RL training such as the Olmo/Tulu models from AllenAI, MiMo v2.6 from Xiaomi, or Beam by Reflection AI.
An Epoch-style visualisation or article tracking RL compute scales of reported models would be an ideal outcome of this step.
We Lack Robust Scaling Laws for RL
As mentioned at the start of the section, we have a poor public understanding of how scaling up an RL training run improves a model’s capabilities. Ideally, we’d have something similar to pre-training scaling laws: a validated relationship between RL compute and model capability across a range of tasks and training setups.
At the moment we can only rely on very noisy signals. Frontier labs have strongly indicated they observed clear scaling trends with their models, but don’t reveal specifics. We can extrapolate trends based on public data, such as in Toby Ord’s analysis of OpenAI’s o-series and RL scaling, but it’s hard to make robust conclusions given limited RL compute data with models reporting on different benchmarks. Finally, large-scale RL experiments are potentially our best sources of data, but can be prohibitively expensive to investigate with frontier-scale models, and the few examples of these so far find results are highly dependent on training recipes and don’t draw explicit comparisons to pre-training scaling laws (see: ScaleRL, [2509.25300], and [2603.12151]). Open post-training labs like Trillium or AllenAI may be well positioned to do useful work on these questions.
In a series of future posts, I intend to contribute to these questions by:
If you have any interest in supporting, collaborating, or advising me with any of these research directions; have ideas for helpful sources of model data; or otherwise want to chat and have further questions about this post, please reach out! Feel free to leave your thoughts on this post in the comments, or email me at obayomijo@gmail.com.
Acknowledgements
Thank you to Josh You at Epoch AI for double-checking my early calculations, initial encouragement to write this post, and helpful comments on my drafts and framing.
Thanks to Sean Fillingham and Robi Rahman for helpful discussions and comments on my drafts.
AI Usage: All writing in this post is mine. LLMs (mainly Claude Opus 5/5.5) were used to proofread for errors in my calculations/text, read chart data for the precise estimates, and format the post’s LaTeX and summary table.
Appendix A: Formula Derivation
The standard formula for the FLOPs used in matrix multiplications (matmuls) during a rollout (excluding dot-product attention) is given by 2NT, where N is the number of active parameters. The attention FLOPs calculated below are an addition to this baseline formula to give the total FLOPs during rollout generation.
For this derivation, I’m primarily relying on my own analysis (some details I haven’t fully explained here and will leave to a future post) and otherwise refer the reader to DeepSeek’s analysis of their MLA algorithm as a sanity check. I’ll introduce the following notation and their corresponding values for DeepSeek-V3/R1:
Naive vs Absorbed MLA FLOPs
For a standard (“naive”) MLA implementation, the attention FLOPs per token (across all layers) are:
The last term slightly changes for an “absorbed” MLA, a slight variation to the algorithm where matrices are multiplied in different orders to save memory at the cost of additional computation.
This “absorbed MLA” algorithm is what DeepSeek reports to have used for their inference / rollout generation[15]. With this the formula becomes:
For the rest of this derivation, I’ll continue referring to this formula, although the former remains relevant for putting together a final calculation.
If we let the product of constants be represented by Cₐ = 2LHDₐ (and likewise Cₙ = 2LHDₙ), and substitute the hyperparameter values DeepSeek-V3/R1 uses, this gives the values stated earlier:
The formula for attention FLOPs per token (at position S) becomes S · Cₐ.
Note: Since S · Cₐ is the number of attention FLOPs per generated token at position S and 2N is the number of matmul FLOPs per token, you can calculate at what context length the dot-product attention FLOPs exceed the other matrix multiplications during decoding. This happens when:
This aligns with a Twitter/X post made around the time of release.
Summing FLOPs Over a Rollout
To get the FLOPs for a single completed response/rollout you could naively multiply the above expression by T, and set S = T. However, this will lead to a 2x overestimate for FLOPs since the value of S increases from Tₚ to T − 1 as the rollout is generated.
Instead, if we multiply the expression by T, and set S = T/2 — treating S as the average sequence length over the whole rollout generation process — this closely approximates the number of FLOPs a more careful calculation with prompt and response lengths summed over the generated tokens would give.
Overall, this gives us the following formulas for the number of dot-attention FLOPs per rollout, and for the total FLOPs per rollout including dot-product attention:[16]
Trainer / Backpropagation FLOPs
The calculations are very similar for backpropagation with two changes.
Overall, this gives the FLOPs per update on a rollout as:
Reference Model
One addition is that in DeepSeek-R1’s training setup, they run a reference model (an older copy of the model’s weights) to calculate the KL penalty for the model being trained. This requires an additional forward pass on the model, and although it isn’t specified, I assume it similarly uses the absorbed MLA implementation on inference nodes. However changing this assumption to a naive MLA implementation would reduce the overall estimate by just under 10%.
By summing the total FLOP counts, and multiplying by the number of rollouts per batch, group size and number of RL steps we arrive at the earlier formula:
where Cₐ = 16,990,208 and Cₙ = 4,997,120.
Note: Oversampling of rollouts may result in more FLOPs being used than estimated here. Typically, more rollouts are generated than are actually used for weight updates, as a result of low-scoring/quality rollouts being filtered out.
One would also want to sum the FLOPs calculated over segments of an RL run for precision, as response lengths tend to increase during training in a way not represented by the mean.
Overall this is still an approximate formula with uncertainties I haven’t fully quantified in either direction.
In a future post, I aim to provide a more granular break-down of the formulas behind calculating FLOPs across various attention mechanisms and lay out a methodology for RL compute estimation.
Appendix B: Further Notes on the Original Formula
Besides batch sizes, here are some of my criticisms/notes on the formula used to compute RL FLOPs.
FLOP stands for Floating Point Operation. In essence, FLOPs count the number of basic operations (multiplications, additions) required to perform a calculation.
DeepSeek’s updated R1 paper explicitly adopts the former convention, stating: “Each training step consists of 32 unique questions, resulting in a training batch size of 512.”
The paper’s batch size = 1024 = B × G, and G = 64 as stated. Hence B = 1024 / 64 = 16.
I originally obtained this figure by visual estimation. For a more precise mean, we can integrate under the graph of tokens against training steps provided in the paper and divide by the number of steps, obtaining a mean of 6100 tokens.
For simplicity, I’ll stick with the original value as the effective response length. The FLOPs with this length match those of a more precise method that accounts for increasing response lengths during the training run (addressed in the next footnote).
To account for the changing response lengths during the run, I calculated the FLOPs on each 500 step interval and summed them together. This gave a final estimate of 1.9e22 FLOPs (LLM-assisted; see this document for the breakdown).
Some sources of uncertainty in my estimates are: the variance in response lengths within a batch (with coefficient of variation of up to 1 from my experiments), oversampling, recomputing off-policy probabilities, and other assumptions in my formula. Overall, I expect the true value to lie in a range between 1.5e22-3e22.
The exact precision at which RL was performed in the paper wasn’t given, so it may be the case the model was trained in FP16/BF16, or mixed precision. If it is, this halves the computing power of the GPUs and decreases the theoretical compute capacity to 3.5e23, increasing the MFU during RL to 5.8%.
MFU is a measure of how well the training run is utilising the actual hardware resources it’s given. In its simplest form, it’s the estimated useful FLOPs for training / theoretical max FLOPs the hardware can perform during the training run.
(2e22 / 6.9e23 × 100 ≈ 3%)
DeepSeek V3’s technical report stated they used 2.66 million H800 hours or $5.3M, with overall training compute costs totalling at 2.8 million H800 hours or $5.6M.
The share of the compute budget used is significantly greater than the share of actual calculations required for RL due to the lower MFU / compute utilisation of RL when compared with pre-training.
Josh You’s post assumes a 10x growth in RL training compute every 3–5 months. Assuming a 5x lower baseline when considering training costs, this sets their estimate back log₁₀(5) × 3–5 = 2–3.5 months.
If instead the projections were anchored on the increase in FLOPs, this would be a log₁₀(30) × 3–5 ~= 4-7 months setback.
To give an example of things being on trend: based on livestream charts, in 2025, xAI seemingly trained Grok-4 with an equivalent amount of RL compute as pre-training compute, both being the largest publicly estimated runs at the time.
Assuming 3e24 FLOPs were used in pre-training DeepSeek-V3. Given 2.66 million GPU-hours were used, then we have MFU = 3e24 / (2.66×10⁶ × 3600 × 1.9e15) ~=16%
This differs from Erdil’s 23% estimate as I assume DeepSeek use the SXM chips rather than the PCIe, which have different throughputs.
Reflection AI reported running 6144 vs 10500 GB300 GPUs for 4 weeks to complete pre-training and RL respectively.
They also report that Beam was pre-trained on 23.8T tokens. During RL they report 100m rollouts were generated and if I assume a long 100k token mean response length, ~10T tokens were generated during RL. So pre-training used 2.4x the tokens.
The actual compute required in both stages will be closer given the costs of long-context attention in RL, but the general point about lower RL MFUs (even after Beam made substantial improvements to the compute efficiency of their RL setup) resulting in the use of more hardware than pre-training seems reasonable (more careful calculations are needed to confirm this).
Lacking a shared methodology and replication procedure makes mistakes harder to notice and correct. It also increases the difficulty of compiling reliable datasets and building on previous estimates. Discussion of some benefits of this here.
I will note that for rollout generation, DeepSeek reports using self-speculative decoding. As this involves additional forward passes, it increases the FLOPs required, although I don’t account for this and don’t expect it to notably affect the FLOPs.
Because the FLOPs depends on T quadratically, a more thorough calculation would quantify the uncertainty introduced by variance in response lengths within a batch, as this would increase the estimates. However no such measurements were reported and I decided to exclude this from my formula, but account for it in the uncertainty ranges.
The activations generated during the rollout phase often can’t be reused to reduce this computation. This is due to greater memory demands, or mismatches between the trainer and rollout specs, where rollouts from previous versions of the model may be used for weight updates.