Setup

The model is the current speedrun record (#360, Sept 2026), with the parts that don't make sense for full-batch training removed:

• The hashed n-gram embedding table. It's a 65B-parameter lookup table keyed on the previous one or two tokens, and it memorizes any data that repeats: validation loss explodes the moment a token is seen twice. Full batch on a fixed set repeats every token every step, so it has to go.
• The sampled softmax, which makes the loss biased and stochastic.
• Every schedule that changes the loss function during training (sequence length, attention window, multi-token-prediction weights, batch size). The whole run uses the record's final-stage settings.

What remains is an 11-layer, width-768 model with the record's FP8 and FlashAttention-3 kernels and its optimizer: a Muon-family optimizer (ANVIL) for the matrices and Adam for everything else. These changes cost about 40% more steps to reach 3.28 than the record (about 1,625 instead of 1,190, at 393K tokens per step).

The dataset is the first 800M tokens of FineWeb10B. I picked this size because small-batch training on it barely overfits: 2,372 steps at 393K tokens per step, about 1.2 passes over the data, reaches 3.248 validation loss with a train/val gap of 0.06.

For full batch, each step accumulates the exact gradient of the mean loss over all 800M tokens (16,272 microbatches of 49K tokens, with fp32 accumulators), then takes one optimizer step. I ran 350 steps. The budget comes from a sweep of large batches on fresh data, which extrapolates to roughly 300 to 330 steps to reach 3.28 at infinite batch; a fresh-data run at 50M tokens per step reaches 3.23 in 350 steps. The full-batch run took about 100 GPU-hours on 4 H200s on preemptible compute, surviving 106 preemptions by checkpointing both between and partway through steps.

Result

"Training loss" here is loss on a fixed 10M-token sample of the training set. At equal training loss, full batch's validation loss is about 0.2 nats worse than small batch's. Full batch's validation loss is flat from about step 260 to the end, while its training loss keeps falling. Intermediate batch sizes sampled from the same 800M set (4.7M to 200M tokens per step) fall progressively further behind fresh data at the same batch size as each token gets seen more times.

What the dynamics look like

The run sits at the edge of stability the whole way through:

• Consecutive full gradients point in nearly opposite directions: the median cosine between them is −0.945. On 98% of steps, every single microbatch gradient is anti-aligned with the previous step's full gradient.
• Measuring the loss along each step's actual update, the step overshoots the minimum in its own direction by about 2× and lands at about the same loss on the far side.
• The exact loss over all 800M training tokens goes up on about a third of steps.
• The top Hessian eigenvalue keeps growing: 48, 82 and 113 at steps 100, 200 and 300.

This is the picture from Cohen et al. on edge-of-stability, here with a Muon-family optimizer on a 2026 speedrun architecture.

Caveats

• One run per configuration. Seed-to-seed noise in validation loss for this model is about 0.003, so the gap isn't noise, but its exact size isn't pinned down.
• Full batch on a fixed set means 350 passes over the same tokens, so this doesn't separate "no gradient noise" from "too many repetitions". Full batch plus artificially injected minibatch-shaped noise would separate the two; I haven't run that.
• The learning rate was tuned on fresh data, not on the fixed set.
• I also tried a schedule ten times as long on fresh data as a convergence reference. In that run the model's FP8 training forward pass and its bf16 evaluation forward pass drifted apart (3.15 against 4.62 on the same unseen data), so I don't trust its number. The full-batch run doesn't have this problem: its two forward passes agree within 0.015 at the end.

All runs

Every run that contributed data to this post, grouped by purpose. "Tokens processed" counts every token in every step, including repeats. "Train" is loss on a fixed 10M-token sample of the 800M training set for runs on that set, and on the training subset for the ablation subset runs; "—" means it wasn't measured. "Val" is the standard speedrun validation set (10.5M tokens) at the end of the run, with the best value in parentheses where it differs. Unless noted, runs use the fixed-objective model described above, on 4 H200s or 8 H100s; metrics for all of them are in the Hugging Face dataset.

Main comparison, on the fixed 800M-token set:

Batch-size ladder on the same 800M set (minibatches drawn without replacement within each pass), with fresh-data runs at the same batch size, learning rate and schedule. Runs marked "halted" were stopped by a rule comparing them against fresh data at matched steps:

Large-batch sweep on fresh data, used to estimate how many steps full batch would need (about 300 to 330 at infinite batch). Steps to 3.28 is where the validation curve crosses 3.28, for runs that got there:

Full-batch pilots on smaller fixed sets (all stopped early once they clearly overfit):

Convergence references on fresh data, at 1×, 3× and 10× the schedule length that reaches 3.28:

Ablation of the original record toward a full-batch-compatible configuration, using the record's own code on 8 H100s (each change is cumulative with the ones above it; "subset" runs train on only the first N tokens, repeated):