If should be trivially simple to recognize if model pretraining runs have been chained togther, looking just at the perplexity graph or hyperparameter schedule settings.
Running multiple pretraining runs in parallel is the same problem as distributed training, which has quite high bandwidth requirements between locations, so this should also be readily apparent with only quite basic monitoring of incoming/outgoing bandwidth.
However, doing this sort of stunt for RL post training runs is a lot more feasible.
Your first bullet sounds challenging to me, but this is not my area of expertise. Your second is even less my area of expertise.
Both my previous suggestions require some accurate access to the training run and cluster — just not anything particularly sophisticated.
[distributed training] has quite high bandwidth requirements between locations
No it doesn't. https://arxiv.org/html/2605.29359v1
I think defenses against this are possible, and I think that we can bound the worst case possibility of it.
In order of difficulty:
1) If the optimizer has some sort of exploit that allows directly setting weights, we would need the model to be able to alter at least its own size in bytes of training data. While it's theoretically possible for the model to be heavily compressed, it would be ~incredibly unlikely~ for the optimizer to have a bug which also happens to operate along the same representation as the compressed model (unless you let the LLM write its own optimizer, in which case you're basically voting for the Lions eating your face party).
The absolute worst case scenario here would be smuggling an optimally distilled submodel of some size, and then doing something like boosting around it. Still not ideal.
2) If the optimizer doesn't have any useful bugs, but the model is able to change its own training data and is able to perfectly predict both seed and hyperparametres for a future run; then we're limited by both the number of bits of training data, batch size, and learning rate.
We still need at least X bits of control over the training data, but now we face pretty difficult computational constraints. If we're able to alter our learning rate, then a greedy algorithm should suffice (just iteratively refine and set lr~=0), but if we're on a fixed learning schedule (like cosine), then it's roughly equivilent to knapsack to construct the gradients needed to epsilon approximate the model weights (someone with more money then me, please check my logic with Fable/Astra/Scott Aaronson). Given the dimensionality of the problem, this imposes serious compute constraints. To make this less-np-hard, you need to first simulate the effects of X-rounds of optimization (where X is the index of your first viable learning rate), and even this assumes the best case scenario of absolute control over a given region of training data (instead of small modifications all around the place).
If training data is optimized over >1 time in a short period of time, this becomes even more difficult. Similarly, tokenized training data makes this harder than continuous training data (like patched time series).
The worst case scenario here probably looks something like existing coreset refinement literature, which can steer behaviour more effectively but can't directly replicate model weights quickly.
3) If we have no way to derive the random seed, I don't think it's viable at all; but even if it is the bounds for 1 and 2 still apply.
Is there anything I'm radically wrong about?
I agree that it's likely there are bounds and countermeasures to mitigate the issue if we restrict to today's algorithms and architectures, but we still need to verify and audit the training algorithm itself to some extent [which is more restrictive than just counting FLOPs!]. In that regime, the questions could be more "how bad can it get? how should we adjust constraints?".
However, for a general statement, robust to attacks by Mythos 7, GPT-8 and the like, a formal model and proof would be necessary (and maybe not sufficient). For instance, when you say "absolute worst case scenario", I fear you are not being maximally creative [not sure if work in this direction would border on info hazard].
Underrated indeed! And I think this is directionally much more true than any even oh so smart 'exactly this way we can control exactly that because of exactly this and this and this' type arguments that we encounter regularly.
While I love to be very specific, I think here we ought exactly not.
It will be evolutionary-type (or whatever) mechanisms through which capitalism will find ways we don't yet think about, to circumvent even the smartest policies we'll have, indeed.
History tells us over and over and over and over again.
Even much of this circumvention might - once we observe what it is - theoretically be prevented, but every new step requires new coordination and its solution may be fraught with ever new delicacies and trade-offs not easy to agree on.
Surely dumb in many ways, but I still think directionally correct, is a comparison to financial markets. Try to regulate any elements in them to avoid a particular problem, say of risk or int'l tax competition or what have you - and they find ever and ever new ways, loopholes. The parallel isn't fully wrong also in the sense that in addition to say 'technical' and sheer cognitive difficulty for smart regulation - which the voting commoners never understand, comes what the big money will be able to do to influence the regulation
With all due respect to anyone confidently putting out 'we can control it, and it needs this many of this powerful and tightly amalgamated Nvidia ...' - simply FORGET about it. At least: admit chances are significantly higher than we can now easily explain, that ways will be found to create the intelli thingy exactly in ever more different ways once we restrict some of the currently known way to SOTA intelli machines.
There is a good verification system against this based on scaling laws, specifically, it is obvious when you are making a model that is over parameterized for the amount of FLOPS you allowed, and yet shows that it has converged if stepped by third parties. The only way to get that is by beating the restrictions, and so any way of hiding pretrain FLOPS is detectable. This does not work for RL, however.
Epistemic status: >90% confidence in the principle, >70% confidence that mitigating these would be hard in practice, no full implementation yet.
tldr: it seems difficult for verification mechanisms to prevent chaining runs together or aggregating parallel ones; per-run FLOP caps could thus be covertly bypassed.
1. If governments want to regulate frontier AI training, one might want to cap individual training runs, e.g. putting bounds on the number of FLOPs per training run, and making sure each run starts either from scratch (i.e. random initialization at pre-training) or from a known model with accounted FLOPs (e.g. post-training, continuous pre-training, etc).
In the former we could have the initial weights generated by the verifier in some way, or ask for a proof that the weights were initialized with a verifier-controlled random seed. In the later we could ask for a proof that the weights corresponds to unaltered registered weights[1] from a previous training run.
This approach would allow to enforce agreements or regulation targeting a specific model (or model generation) based on FLOPS thresholds: triggering specific evals beyond a specific threshold and possibly proving that a hard threshold has not been reached.
2. However, it seems that there exist many techniques the trainer might use to covertly (a) chain sequential runs together or (b) aggregate parallel ones.[2]
Chaining works by finding a way to inject, at the beginning of the training, the weights of a previously trained model, either to replace a random initialization (pre-training case) or to replace declared weights with ones from bigger training run (post-training case).[3] One way to approach this could be to encode the smuggled weights in the training data[4], decode them when the run begins, then continuing with "normal" training. Some of those techniques might be computationally very similar to regular training.
Aggregation works by combining the results of several parallel runs a posteriori, such as through model souping.[5]
3. Thus, the effectiveness of regulatory regimes which rely on capping the number of FLOPs per training run is in question.[6]
4. A potential solution to avoid this type of attacks consists in not relying on per-run compute budget, but instead on a unique global training budget covering the entirety of the training runs. While preventing that type of attacks, global training budgets approaches still require the questions of global budget thresholds estimation and allocation to be addressed.
GPAI Policy Lab is building a research community tackling open problems in frontier AI verification. If you're interested in contributing, reach out!
Registration does not imply revealing the weights. A cryptographic signature (e.g. hash) can be publicly registered with a traceability certificate linking to the training run that produced it.
Such attacks would technically respect the letter of the FLOPs bounds, not the spirit. One might be able to mitigate these by adding restrictions, checking stronger properties of training (beyond literal FLOPs bounds), and putting strict controls on model weights; we're not sure these mitigations can be effective against very competent adversaries.
This injection does not require necessarily to inject the exact weights: getting close to previous weights could be enough.
Smuggling the full parameters suite at full precision is likely to require a very big part of the training data, questioning the efficacy of this solution, but a LoRa approximation would make it much more easy to conceal, and unfortunately very tractable.
Maiti, Shalini, et al. “Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance.” arXiv:2511.13254, arXiv, 17 Nov. 2025. arXiv.org, https://doi.org/10.48550/arXiv.2511.13254.
Let us emphasize that these attacks do not depend on the technical way one verifies FLOPs bounds, unless the technical mechanisms are explicitly designed to try and mitigate the issue, as gestured at in the note above.