beyond the inevitable 3-letter agency training "ghostsites", I think commercial operations will just as inevitably expand the definition of inference to include conventional training. we've already seen some of that occurring with CoT, but I think this generalizes almost completely. the necessary advances in model building techniques aren't so very far away as seems to be assumed.
Overview
These are my non-expert notes on the compute verification section of AIFP’s Plan A. I cover interconnect limits, memory wipes, network taps + replay, and ZKPs. For the most part, the sections can be read independently. I restrict my attention to inference-only verification: ensuring that compute is used for inference, not training. For each method suggested by AIFP, I ask:
AIFP estimates that the fraction of the world’s compute that is unmonitored might be kept as low as 0.1% (this is the optimistic, low end of their 80% confidence interval). So my target for inference-only verification is to slow down training by 1000x – any more hits diminishing returns as unmonitored compute dominates – with much less than 1000x overhead on inference and little sharing of secrets.
I won’t discuss how much a 1000x reduction in effective training compute would actually benefit humanity. The answer depends greatly on algorithmic progress rates; I wish labs would publish the rates they’re seeing internally.
Interconnect Limits
In a datacenter, accelerator racks are connected to two networks. The backend network is used to send gradients between racks during training, and to send some large objects like KV caches during inference. The frontend network is used to send prompts and responses, and occasionally to load model checkpoints. AIFP proposes physically removing the backend network. Inference would be mostly unaffected, since the inference stack can probably be rewritten so that KV caches never cross rack boundaries.[1] But standard training relies heavily on the backend network, so the hope is that removing it would massively slow training.
Just removing the backend network would likely not slow training by much, though, since the frontend network’s bandwidth per rack is not that much lower than the backend network’s: ChatGPT estimates ~1TB/s and ~4TB/s, respectively, with big error bars.[2] Therefore, one should additionally enforce per-rack bandwidth limits on the frontend network, calibrated to allow sending tokens but not gradients. ChatGPT estimates that, after compression, a typical LLM inference rack sends and receives a few MB/s, so a 10MB/s limit may be acceptable. At this rate, sending or receiving a trillion-parameter gradient in fp8 would take around a day, slowing standard training massively.
The obvious attack is low-communication training via algorithms such as DiLoCo or SparseLoCo, which let each rack train its own replica of the model, only occasionally syncing updates. There isn’t much data on how well these algorithms work at frontier scale. Covenant-72B was pretrained using SparseLoCo, communicating pseudo-gradients that are 146x compressed relative to fp32 every 30 steps. If we naively apply those numbers to a trillion-parameter model, assume that each backward pass takes 10s, and limit bandwidth to 10 MB/s per rack, then for each 5 minutes’ worth of computation, a rack must spend ~50 minutes sending the update: a 10x slowdown. But Covenant-72B is far below frontier scale, and used only 20 replicas, whereas a frontier datacenter would need thousands. Also, although the model is supposedly around Llama-2-70B level, it is not benchmarked against the same model trained with standard high-bandwidth AdamW.[3] So we have our first open question:
I don’t think you should directly work on OQ1, but it seems important to keep track of the SoTA and forecast future progress. Another, lower-priority open question:
Summary: Removing the backend network, and imposing bandwidth limits on the frontend network, comes with little downside: it requires no secret sharing between adversaries and imposes minor overhead on inference. Training with only 10MB/s bandwidth per rack sounds difficult, but it’s hard to estimate the exact slowdown it would impose, since little research has been done on super-low-communication training across many racks (at least, not compared to the amount of work that has gone into optimizing frontier training stacks).
Memory Wipes
Another technique that seems useful: periodically wipe each rack’s memory, with a proof that the wipe was done correctly, then reload it only with precommitted stuff (no new artifacts like weight updates). AIFP mentions memory wipes as a method to ensure “that the only outputs from the cluster are verified inference tokens” (“verified” here means “reproducible via replay” – see next section). But I don’t understand this: why does it matter what a rack has in memory, provided every bit it emits is verified?
Perhaps memory wipes could help prevent the sort of low-communication training that evades interconnect limits. If the interval between wipes is short, not much gradient information can be accumulated by a rack (note that although SparseLoCo syncs are small and infrequent, each node still must accumulate a large amount of information over the course of training – the net parameter update is dense). We saw that, with a 10MB/s bandwidth limit per rack, a full trillion-parameter update in fp8 would take about a day to send. So, in order to be useful, the period between wipes should be less than a day. Unfortunately, in an initial test, Amodo Design found that provably wiping a rack took ~24h and, if I'm reading their results correctly, left more than 100TB of memory unwiped.[4]
Summary: I mainly see memory wipes as a technique to combine with interconnect limits. They force the attacker to use algorithms that are not only low-communication, but that also keep each rack's state close to the initial checkpoint: a rack can only accumulate as much extra state as fits through the bandwidth limit in one wipe interval. I expect that this could be a really powerful way to slow training, and it does not require any secret sharing. The only issue is that current memory wipe methods take too long (adding a lot of overhead to inference) and leave too much residual memory unwiped.
Network Taps and Replay
Network taps record accelerator racks’ ingress and egress. This traffic goes to a replay server which recomputes egress from ingress. Since the replay server contains much less compute than the full datacenter, only a small fraction of traffic can be replayed. Concretely, the prover should provide[5] a partition
Plan A says that replay is used to “check the correctness of the packets”. I find this phrasing confusing: how are the packets produced by training any less “correct” than those produced by inference? My understanding is that what replay buys us depends on whether the replay server hardware is trusted by the verifier or by the prover (I assume that manufacturing a mutually-trusted replay server is impractical). I sketch the sorts of properties one could verify in each case below.
Trusted Replay
If replay takes place on verifier-trusted hardware, the verifier can be sure of exactly what program P the replay server runs (e.g. it can see model weights, kernels, and computational graphs). This feels like it should make the verifier’s job easy: all it needs to do is read P, then decide whether it is training or inference. But the code might be obfuscated, and the verifier must be able to detect unseen future training algorithms.
Rice’s theorem rules out a perfect classifier for arbitrary programs, so one should restrict the program language (e.g. consider StableHLO computational graphs of bounded depth) or put up with some false positives.
Trusted replay adds two forms of overhead to inference. First, replay must either use a deterministic stack or record enough execution metadata to reproduce the original computation exactly. Cankaya (2026a) suggests this may sometimes be possible without sacrificing inference performance, but I expect that in general this adds overhead. Second, some compute must be set aside to be used for replay, rather than for directly useful work. But due to the power of sampling, the replay fraction can stay small. If the verifier samples a small fraction
The main downside of building the replay server from verifier-trusted hardware is that the server might leak the prover’s secrets to the outside world. This issue can be partly mitigated by placing a strict egress limit on the replay server. Cankaya (2026b) estimates that physical isolation techniques can reduce covert egress to kilobits per second, at a cost of less than 1% of total datacenter cost. At 1kbps, a trillion-parameter model in fp8 would take 250 years to exfiltrate – too long to worry about.[6] More worrying is the leakage of algorithmic secrets, which can have short description length, as well as sensitive information contained in prompts and responses.
Summary: I think trusted replay is what AIFP had in mind in Plan A, and I get the sense that it’s what people generally mean when they talk about recomputation, re-execution etc. I think people have this idea that replay lets us check that the “correct” program ran, but haven’t yet proposed a way to tell whether or not that program is training (OQ4). Without such a proposal in hand, I can’t confidently estimate inference and training overhead, but I weakly expect that classifying code as training vs inference is tractable, and very large training overhead (>1000x) can be imposed without affecting inference too much. The trust issues in this proposal seem quite bad to me: algorithmic secrets and sensitive user information are probably leaked.
Untrusted Replay
If the replay server is not trusted by the verifier, there is no worry about secrets leaking. All the verifier must see is commitments to network traffic records, which it checks against commitments to the inputs and outputs of the replay server.[7] But unlike with trusted replay, the verifier cannot read the code the replay server is running; it can only check “externally observable” properties like:
Reuter et al (2026) propose a solution: they call a workload
Summary: Untrusted replay doesn’t suffer the secret-leaking issue of trusted replay. Again, without an answer to OQ5, I can’t estimate the inference and training overhead. I may write an accessible summary of Reuter et al’s proposal soon, with a discussion of whether it fully solves OQ5. In the meantime I encourage others to read their paper.
Zero-knowledge Proofs
Zero-knowledge proofs (ZKPs) offer the visibility of trusted replay without the trust issues. As before, we tap all inter-rack traffic, and that traffic is somehow partitioned into input-output pairs
Is this too good to be true? AIFP gives the standard warning that ZKPs are currently too expensive to be practical. I don’t think I’ve ever seen ZKPs mentioned in a compute verification context without this warning attached. In my experiments, proving “this output token was obtained by running a committed model on these inputs and sampling from the logits” took over a million times longer than running the original forward pass, which sounds like a lot. (Caveats: I used tiny models and input sizes; I only hid weights, not architecture, as seems to be standard in zkML).[8] But say 0.1% of a datacenter’s compute is set aside for producing ZKPs. Then, at the 1e6 overhead I measured, the prover could prove one in every 1e9 tokens, which seems totally fine to me. Say the verifier uniformly samples one in every 1e9 tokens, and tells the prover to produce ZKPs for those tokens. The prover can only get away with on the order of a billion incorrect tokens (i.e. ones that weren’t actually the output of running
My conclusion here seems to be wildly different to everyone else’s that I’ve seen, including people with far more expertise than me. So you should be appropriately skeptical, and I think it’s likely I’ll be corrected in the comments. I guess I’ll state the obvious open question regardless:
This is a math problem with a crisp success metric: efficiency. So it seems particularly amenable to autonomous research using LLMs. Someone should throw tokens at it.
Summary: ZKPs let us slow down training by the same amount as trusted replay, which I expect to be a lot, without leaking any secrets. ZKPs are currently very expensive to construct, but, if my logic is to be trusted, we can get away with sampling a tiny fraction of computations
Appendix: Notable Omissions
Here I briefly mention a few proposals that I’ve seen floating around in the verification literature but which were notably absent from Plan A. My notes here are particularly low-effort – sorry if I don’t give appropriate time of day to your favorite proposal.
Trusted execution environments (TEEs) let chips prove what code they are running, which seems like it should be useful for inference verification. But TEEs keep getting broken. Also, if I were China, I’d worry that Nvidia might collude with an American datacenter operator, providing them secret keys with which to forge proofs. FlexHEGs aim to solve these problems. They put the chip in a tamper-proof box alongside a “guarantee processor” which issues proofs that the chip ran such-and-such a computation, and everyone can trust the proofs because the processor was built fully open-source (along with fab inspections, etc). Unfortunately, as far as I can tell, not much progress has been made on FlexHEGs since the initial paper. Overall, I think AIFP was probably correct not to focus on these sorts of hardware-based workload attestation methods.
Rahman and Tajdari (2026) detect training via telemetry signals: GPU utilization, memory utilization, memory used, power draw, temperature, SM clock, memory clock, and PCIe TX/RX bandwidth. All of these quantities are self-reported by the chip, so a prover that owns the chip could simply lie about them. So telemetry-based verification should only use signals that can be measured by trusted, tamper-proof devices – things like power draw and temperature, which can be measured off-chip. I expect that it’s easy enough for the prover to spoof those signals. For example, I told Fable to train a model in such a way that the power draw traced out the silhouette of a reclining man, and it did a decent (though not perfect) job:
Fable's explanation of what it did
The GPU's power draw is controlled by exploiting a simple fact: a matrix multiplication's power consumption depends on its shape. We train a neural network — a 14-layer residual MLP — but instead of calling a standard training library, we hand-write the forward and backward passes ourselves as a long sequence of small matrix-multiply tiles. Each tile computes one block of the real gradient, and the width of that block sets how many of the GPU's cores are active, and therefore how much power it draws at that instant: a narrow tile lights up few cores and draws ~100 W, a wide one draws ~250 W. By calibrating the tile-width-to-watts relationship on the specific GPU and then choosing each tile's width to match the target curve at that moment, the power trace draws whatever shape we want. Nothing is faked: every multiplication is part of a genuine gradient computation, the gradient matches PyTorch's autograd bit-for-bit, and the network's loss really falls. The only trick is the order and shape in which the arithmetic is scheduled. A feedback loop (iterative learning control) replays the pattern a dozen-odd times, measuring the delivered power each round and correcting the command, to converge on a tight match.
Resource exhaustion strategies may be able to establish completeness – that no undeclared computations were run – without using network taps. For example, by using techniques like this, the prover could produce a ZKP that it ran a program
Alternatively, the compute could be arranged into islands of a few backend-connected racks to allow the movement of KV caches, with no backend connections between islands.
So e.g. sending a trillion-parameter gradient over the frontend network would only take seconds: fast enough to be done concurrently with the gradient computation itself, which ChatGPT estimates as ~10s. Larger models bump up against communication limits regardless of which network is used, requiring clever, low-communication parallelism.
The original SparseLoCo paper performs a controlled comparison, and finds that a 2B model trained over 16 replicas, communicating 16x compressed gradients every 50 steps, matches the perplexity of the equivalent densely-trained model. But again, it’s very hard to tell how these results will scale.
They use a Proof of Secure Erasure: flood memory with hash labels the prover could only produce by actually filling memory, then challenge it to recite random locations back within a time limit. Wiping persistent storage leaves some residual unwiped memory: a prover willing to reflash its own SSD controllers gets the “non-wipeable” column of their Table 1, which is over-provisioned flash the host cannot normally address. That column appears to be per device rather than per rack, so the full 24h wipe leaves ~110TB unwiped across the full rack.
The partition might be fixed ahead of time (e.g.
Brown et al (2026) demonstrate 16x weight compression with minimal capability degradation (provided a small amount of finetuning is performed after decompression) and guess that 100x is feasible. Even then, exfiltration would take several years.
I think this requires a mutually-trusted device that computes hashes. Amodo Design did some experiments that look very relevant, but it’s way outside my expertise and I can’t really comment.
Ideally we’d prove a prefill forward pass all at once – this would essentially prove a whole batch of decode steps all at once, but more cheaply since prefill is fast. But annoyingly, prefill and decode use different kernels, so the logits they produce differ slightly. Perhaps once could rewrite the inference stack so that prefill and decode match exactly, but I expect this would incur a lot of overhead.
I proved the exact floating point arithmetic done by the chip, as given by Hawkeye. ZKPs play nicer with exact matmuls than with float arithmetic, since they can use Freivald’s algorithm. Proving an integerized version of the model was around 10x faster (I expect the gap would widen for larger models, due to fixed costs). But the integerized model produces slightly different logits to the float model, so the prover can occasionally choose to sample a different token than the one the integerized model would’ve sampled. Across several models, I find that this typically allows about 0.3 bits of covert information to be sent per token. A token carries a few bits of information in total, so the covert channel might have around 10% of the capacity of the full token channel. In other words, the net effect of the integerized ZKP is to effectively reduce interconnect limits by, say, 10x.