The Chinchilla scaling law says that for a given number of FLOPs C, the optimal amount of data D to train on is proportional to (C to the power of b), that is you'll get a less intelligent model if you use either more data or less data than that while training for C FLOPs. The optimal model size (number of parameters) N is then whatever would require C FLOPs when trained on D tokens of data, and C is usually about 6ND, so optimal N is proportional to (C to the power of (1-b)). See Figure 4 in the paper. Turns out that b is close to 0.5, so N and D increase with C at a similar pace, and their ratio D/N (tokens per parameter) remains approximately the same across multiple orders of magnitude for available compute C.

In the Chinchilla paper, the exponent b is estimated as 0.50-0.54 using different methods^[1] (see Table 3), and D/N at 3e21 FLOPs is about 20 tokens per parameter. If b is taken to be 0.50, then D/N doesn't change with compute. But if b is 0.54, then D/N increases proportionally to (C to the power of 0.08). At 2e25 FLOPs of original GPT-4 that would result in D/N of 40 tokens/parameter, at 5e26 FLOPs of the next year's models it becomes 52, and at 7e27 of late 2026 models from 1 gigawatt training systems it becomes 64 tokens/parameter. So even the original Chinchilla paper doesn't quite promise 20 tokens/parameter for 5e28 FLOPs with some of its estimates of b, you'd need to stick to the estimate of 0.50, and there isn't enough data to confidently make that decision, not 7 orders of magnitude beyond the scale of the actual experiments that only go up to 3e21 FLOPs (Chinchilla itself is 6e23 FLOPs, at 200 times more compute, where we get D/N of 30 for b of 0.54; but they went with b of 0.50).

The picture of steady 20 tokens/parameter is disrupted further with more recent papers. Llama 3 report measures the exponent b of 0.53 with experiments of up to 1e22 FLOPs. At 3.8e25 FLOPs of Llama-3-405B, this predicts a D/N ratio of 41 tokens per parameter (see Figures 2 and 3). Back at 3e21, this predicts a D/N ratio of 23^[2], consistent with the Chinchilla experiments at that scale. For 5e26 FLOPs, this predicts D/N of 48, for 7e27 FLOPs a D/N of 56.

There is also the DeepSeek report where the estimate of exponent b is 0.47 (see Figure 4) from experiments of up to 3e20 FLOPs. This again predicts a D/N of about 20 at 3e21 FLOPs, even though the extrapolation to...

O O's Shortform

1

O O's Shortform

1