A baseline overhead measurement and the hash-problem
This is an exploration into the current state of ZKP for LLM inference to use in a high-stakes environment like international agreements. It’s the first post in a current project I’m working on that will result in a paper that goes into all of the details of this question; “When does ZKP become more cost-effective than an auditing environment?” in this context. I got inspired after reading Naci Cankaya’s post here on LW and am motivated by a general “Don’t trust, verify”-mentality.
Motivation
I won’t go into the motivation behind the need for low-trust verification in general since that is completely covered by Naci Cankaya in the original post. The motivation for this comparison is to identify the areas in which the ZKP path of verification has to improve to be more cost-competitive and generally preferable to the auditing environment. The auditing environment comes with some inherent issues regarding security and trust that would ideally be solved by the ZKP path, such as needing to either invite hardware approved by an adversary into your secure facility and handle proprietary data or, for the verifier - trust the prover to run the auditing themselves on selected hardware (which seems dead in the water to begin with), or trusting a third party with the same. The auditing environment also comes with the additional cost of running a secure facility within a secure facility, while a ZKP should ideally be executable on an untrusted device.
The situation today is that ZKP takes much more times the computation to run than the inference it’s proving. In my own comparison between the zkLLM run in the paper by Sun, Li, Zhang (2024) and my own run of the same model sizes, token length and hardware gives 1,694 - 3,777 x compute. In the context of using ZKP for regular inference proof for everyday usage, that is arguably too much, but in the context of international agreement on AI-compute verification, this has to be compared to the alternative of the proposed auditing environment. This is what I aim to do with this project (- see next steps).
I’m creating this first post before the whole project is done so I can get some early feedback and gather inspiration from others, and potentially find collaborators to make it even more rigorous. That said - it is meant to be self-contained and be valuable on its own as well.
Comparison
This is the main calculation the project aims to fill in. I don’t have much detail for each variable yet, but I lay it out here so that others can poke holes in it and possibly point me to relevant literature.
Cost per year
Number of verifications/audits per year
Native compute for one forward pass
How much more times the native compute it takes to run one verification on one forward pass
Price per unit of compute
Model commitments per year
Cost per model commitment
Fixed costs (guards, facility, TCB)
Mainly circuit auditing; proving can run on ordinary datacenter GPUs, since proof validity doesn’t depend on trusting the hardware.
ZKP side
Computation calculations
in the calculation needs to take several factors into consideration, so I’m breaking it down.
Of course all of the variables in this calculation won’t be so easy to derive into a precise number-range, I’m just trying to capture all of the different moving factors that in the end will affect the compute cost. If you have any feedback on this calculation, please share! Did I miss something?
Measured baseline from model size up to 13B overhead.
Hardware improvements: How much newer hardware speeds up native inference relative to proving. I assume it’ll be over 1, since FP8/FP4 tensor cores help native inference but not field arithmetic. ZKP-specific hardware is the counterweight.
Model size above 13B (as in the baseline).
Token length beyond 2,048 (as in the baseline).
Need for bit-exact verification on floating-point calculations.
Difference in model architecture between the simpler models measured and the target frontier models, such as for example “mixture-of-expert routing”.
Stronger requirement for security to verify in a high-stakes environment.
Progress to the ZKP. Only makes sense to trend in a decreasing direction.
Model Size
I take the prover time (s) for zkLLM on OPT and LLaMa-2 from Sun, Li, Zhang’s (2024) paper (zkML is the older less optimized one). This was done on Nvidia A100 (40GB), 124GB RAM, model sizes up to 13B, token length of 2,048.
To check the times it takes to run forward passes on each model with the same setup as the paper, I run a script on Modal and measure each model's time, and compare with the proving time.
The smallest model (OPT-125M) is an outlier, I think it doesn’t really utilize the full GPU and that is probably why the overhead for the proof is so much higher on that one, and that’s why it’s left out of the final estimate. There could be a small increase (or even decrease but that seems unlikely in the long run) in overhead for the rest. As we see with the LLaMA-2 models, there might be a big difference between models for other reasons than size as well. It’s important not to draw any certain conclusions from such a small dataset, this is just an illustrative example of what would be needed on a bigger scale. Further benchmark tests are required.
As seen in the data, the larger overhead for LLaMA-2 7B mostly depends on the shorter inference time; the proving time isn’t that much higher than OPT-6.7B, so this obviously also affects the overhead, and since frontier models are much more optimised than HF transformers that means the real overhead might be much higher. Note that in this transformers version, LLaMA likely uses fused SDPA attention while OPT uses eager attention, so part of this gap may be implementation rather than architecture, which further illustrates how sensitive k₀ is to baseline optimization.
GPUs
Larger models also require more GPU memory for the proofs, while the models tested in the paper by Sun, Li, Zhang’s (2024) were small enough to all fit under 40 GB on A100, when the size gets large enough that you need multiple GPUs, there’s a cost in communication between them and more importantly; distributed ZKP is not yet solved. Frontier models don’t run on A100, and the specialized GPUs that run inference are not suitable for ZKP (LLMs can for example tolerate the imprecision those GPUs deliver to accelerate speed while ZKP cannot). With a similar effort to build hardware specifically for ZKP, it’s possible that similar increases in efficiency can be made there.
Hash functions
For the auditing environment, the hash function is pretty straight forward since it will just be used by the tap. But for ZKP, the hash function needs to be compatible with the ZKP as well, and the speed at which it can be processed needs to be taken into consideration.
The computational problem
Whatever hash is run on the tap, the ZKP has to recompute. SHA-256 and SHA-3 are built for CPUs and use bitwise operations like XOR, AND, bit rotations and additions. ZKP runs code by turning the computations into a big set of equations over a finite field. The only operations that are supported in there are additions, multiplications and mod p. So if SHA is used as the hash, ZKP has to simulate bitwise operations inside using only addition and multiplication. Since XOR is (a + b − 2ab), this means that one XOR of two 32-bit words naively costs 32 operations (the others are pretty straightforward), not counting the decomposition. The decomposition and recomposition also come with major costs but modern proof systems reduce this cost with lookup arguments (e.g. Plookup, LogUp), which replace bit decomposition with table lookups over small chunks, but the SHA-family remains substantially more expensive in-circuit than ZK-native hashes like Poseidon. The SHA-family also works in another number system than ZKP; it works in mod 2³², so it wraps much earlier than ZKP’s mod p. So for the result, ZKP has to calculate the wrapping as well.
So why not just use Poseidon (Grassi et. al, 2021) then? It runs natively in the ZKP, but it would be much slower on the tap, exactly where you need an efficient function that can run fast and efficient “for the high line rate, ideally without software” Naci Cankaya (2026), which is exactly why they propose SHA-3.
The political problem
Another thing to consider, since this will live in a setting where there already is very little trust is that the ZKP-compatible hash functions are much newer than the standard and tested SHA-256 and SHA-3, so the trade-off in what technologies countries will be willing to trust is a real issue here. So computationally, you want to use something like Poseidon for a hash function, since that was created specifically for ZKP (Grassi et. al, 2021), but for political trust, you want to use something like SHA-256 or SHA-3 since they’ve been around for much longer and have earned the trust.
Discussion
The technologies are developing, some STARK-style proof systems have smaller fields, like Plonky3, and a hash function like Poseidon2 could work with this on a tap. But since the ZKP is going to be a new thing regardless, it’s more advantageous to keep the tap conservative by using SHA-3 and using another ZKP that is more bitfriendly. An example here is Binius. But then we run into the problem of the model not being binary. The ZKP has to run over a huge number of matrix multiplications and although Binius has some techniques to handle this, it’s not nearly good enough to be competitive with the existing tools where such operations are field-native. Ideally, we would want to use SHA-3, and a field-operating ZKP for the proof but then have a layer in between that specialises in translating the SHA-3 hash into something that the ZKP can verify with only field-native operations.
Next steps
Benchmarks: I will gather and read up on the current state of ZKP on LLM inference, collect all of the different benchmarks and run corresponding inference tests to measure the overheads, as well as run my own ZKPs to collect more data in the areas that are least represented.
Hash: The problem regarding compatibility between the hash-function and the ZKP was more interesting and complex than I first imagined, so I will look into the current state and the techniques in this area as well and map the different possible combinations and list their pros and cons.
Model privacy: “Standard zkML systems (ZKML, zkLLM) hide model weights from the verifier but require the model architecture (layer count, kernel sizes, attention heads) to be public, because the architecture is encoded into the circuit structure.” Naci Cankaya (2026). This is a major point that needs solving, I will look into the development in this area.
Variables: I will go through each of the variables in the ZKP compute calculation and expand on them with what the current state is, and in what directions they’re likely to develop, including what the uncertainties are. I’m mostly interested here to find the biggest blockers, but there are also possible positive developments. At GPT-2 scale, reported proving times fell from about an hour to under 25 seconds within roughly two years, though on differing hardware and setups - but this points to how progress in the ZKP can be a significant factor.
Auditing Environment: I will create a similar calculation for the auditing environment. Admittedly, this will include some variables that are less technical and more operational, so the costs will be monetary rather than computational.
GPU: NVIDIA A100-SXM4-40GB Torch: 2.4.0+cu121 Precision: fp16, batch size 1, one forward pass over 2,048 tokens, median of 10 runs after 3 warm-up runs.
Note: LLaMA-2 timings use shape-matched, randomly initialized models (same architecture and dimensions).
A baseline overhead measurement and the hash-problem
This is an exploration into the current state of ZKP for LLM inference to use in a high-stakes environment like international agreements. It’s the first post in a current project I’m working on that will result in a paper that goes into all of the details of this question; “When does ZKP become more cost-effective than an auditing environment?” in this context. I got inspired after reading Naci Cankaya’s post here on LW and am motivated by a general “Don’t trust, verify”-mentality.
Motivation
I won’t go into the motivation behind the need for low-trust verification in general since that is completely covered by Naci Cankaya in the original post. The motivation for this comparison is to identify the areas in which the ZKP path of verification has to improve to be more cost-competitive and generally preferable to the auditing environment. The auditing environment comes with some inherent issues regarding security and trust that would ideally be solved by the ZKP path, such as needing to either invite hardware approved by an adversary into your secure facility and handle proprietary data or, for the verifier - trust the prover to run the auditing themselves on selected hardware (which seems dead in the water to begin with), or trusting a third party with the same. The auditing environment also comes with the additional cost of running a secure facility within a secure facility, while a ZKP should ideally be executable on an untrusted device.
The situation today is that ZKP takes much more times the computation to run than the inference it’s proving. In my own comparison between the zkLLM run in the paper by Sun, Li, Zhang (2024) and my own run of the same model sizes, token length and hardware gives 1,694 - 3,777 x compute. In the context of using ZKP for regular inference proof for everyday usage, that is arguably too much, but in the context of international agreement on AI-compute verification, this has to be compared to the alternative of the proposed auditing environment. This is what I aim to do with this project (- see next steps).
I’m creating this first post before the whole project is done so I can get some early feedback and gather inspiration from others, and potentially find collaborators to make it even more rigorous. That said - it is meant to be self-contained and be valuable on its own as well.
Comparison
This is the main calculation the project aims to fill in. I don’t have much detail for each variable yet, but I lay it out here so that others can poke holes in it and possibly point me to relevant literature.
Cost per year
Number of verifications/audits per year
Native compute for one forward pass
How much more times the native compute it takes to run one verification on one forward pass
Price per unit of compute
Model commitments per year
Cost per model commitment
Fixed costs (guards, facility, TCB)
Mainly circuit auditing; proving can run on ordinary datacenter GPUs, since proof validity doesn’t depend on trusting the hardware.
ZKP side
Computation calculations
Of course all of the variables in this calculation won’t be so easy to derive into a precise number-range, I’m just trying to capture all of the different moving factors that in the end will affect the compute cost. If you have any feedback on this calculation, please share! Did I miss something?
Measured baseline from model size up to 13B overhead.
Hardware improvements: How much newer hardware speeds up native inference relative to proving. I assume it’ll be over 1, since FP8/FP4 tensor cores help native inference but not field arithmetic. ZKP-specific hardware is the counterweight.
Model size above 13B (as in the baseline).
Token length beyond 2,048 (as in the baseline).
Need for bit-exact verification on floating-point calculations.
Difference in model architecture between the simpler models measured and the target frontier models, such as for example “mixture-of-expert routing”.
Stronger requirement for security to verify in a high-stakes environment.
Progress to the ZKP. Only makes sense to trend in a decreasing direction.
Model Size
I take the prover time (s) for zkLLM on OPT and LLaMa-2 from Sun, Li, Zhang’s (2024) paper (zkML is the older less optimized one). This was done on Nvidia A100 (40GB), 124GB RAM, model sizes up to 13B, token length of 2,048.
To check the times it takes to run forward passes on each model with the same setup as the paper, I run a script on Modal and measure each model's time, and compare with the proving time.
The smallest model (OPT-125M) is an outlier, I think it doesn’t really utilize the full GPU and that is probably why the overhead for the proof is so much higher on that one, and that’s why it’s left out of the final estimate. There could be a small increase (or even decrease but that seems unlikely in the long run) in overhead for the rest. As we see with the LLaMA-2 models, there might be a big difference between models for other reasons than size as well. It’s important not to draw any certain conclusions from such a small dataset, this is just an illustrative example of what would be needed on a bigger scale. Further benchmark tests are required.
As seen in the data, the larger overhead for LLaMA-2 7B mostly depends on the shorter inference time; the proving time isn’t that much higher than OPT-6.7B, so this obviously also affects the overhead, and since frontier models are much more optimised than HF transformers that means the real overhead might be much higher. Note that in this transformers version, LLaMA likely uses fused SDPA attention while OPT uses eager attention, so part of this gap may be implementation rather than architecture, which further illustrates how sensitive k₀ is to baseline optimization.
GPUs
Larger models also require more GPU memory for the proofs, while the models tested in the paper by Sun, Li, Zhang’s (2024) were small enough to all fit under 40 GB on A100, when the size gets large enough that you need multiple GPUs, there’s a cost in communication between them and more importantly; distributed ZKP is not yet solved. Frontier models don’t run on A100, and the specialized GPUs that run inference are not suitable for ZKP (LLMs can for example tolerate the imprecision those GPUs deliver to accelerate speed while ZKP cannot). With a similar effort to build hardware specifically for ZKP, it’s possible that similar increases in efficiency can be made there.
Hash functions
For the auditing environment, the hash function is pretty straight forward since it will just be used by the tap. But for ZKP, the hash function needs to be compatible with the ZKP as well, and the speed at which it can be processed needs to be taken into consideration.
The computational problem
Whatever hash is run on the tap, the ZKP has to recompute. SHA-256 and SHA-3 are built for CPUs and use bitwise operations like XOR, AND, bit rotations and additions. ZKP runs code by turning the computations into a big set of equations over a finite field. The only operations that are supported in there are additions, multiplications and mod p. So if SHA is used as the hash, ZKP has to simulate bitwise operations inside using only addition and multiplication. Since XOR is (a + b − 2ab), this means that one XOR of two 32-bit words naively costs 32 operations (the others are pretty straightforward), not counting the decomposition. The decomposition and recomposition also come with major costs but modern proof systems reduce this cost with lookup arguments (e.g. Plookup, LogUp), which replace bit decomposition with table lookups over small chunks, but the SHA-family remains substantially more expensive in-circuit than ZK-native hashes like Poseidon. The SHA-family also works in another number system than ZKP; it works in mod 2³², so it wraps much earlier than ZKP’s mod p. So for the result, ZKP has to calculate the wrapping as well.
So why not just use Poseidon (Grassi et. al, 2021) then? It runs natively in the ZKP, but it would be much slower on the tap, exactly where you need an efficient function that can run fast and efficient “for the high line rate, ideally without software” Naci Cankaya (2026), which is exactly why they propose SHA-3.
The political problem
Another thing to consider, since this will live in a setting where there already is very little trust is that the ZKP-compatible hash functions are much newer than the standard and tested SHA-256 and SHA-3, so the trade-off in what technologies countries will be willing to trust is a real issue here. So computationally, you want to use something like Poseidon for a hash function, since that was created specifically for ZKP (Grassi et. al, 2021), but for political trust, you want to use something like SHA-256 or SHA-3 since they’ve been around for much longer and have earned the trust.
Discussion
The technologies are developing, some STARK-style proof systems have smaller fields, like Plonky3, and a hash function like Poseidon2 could work with this on a tap. But since the ZKP is going to be a new thing regardless, it’s more advantageous to keep the tap conservative by using SHA-3 and using another ZKP that is more bitfriendly. An example here is Binius. But then we run into the problem of the model not being binary. The ZKP has to run over a huge number of matrix multiplications and although Binius has some techniques to handle this, it’s not nearly good enough to be competitive with the existing tools where such operations are field-native. Ideally, we would want to use SHA-3, and a field-operating ZKP for the proof but then have a layer in between that specialises in translating the SHA-3 hash into something that the ZKP can verify with only field-native operations.
Next steps
Data
Inference vs. ZKP proving time
Ran on Modal (numbers rounded), zkLLM proof times from Sun, Li, Zhang’s (2024)
GPU: NVIDIA A100-SXM4-40GB
Torch: 2.4.0+cu121
Precision: fp16, batch size 1, one forward pass over 2,048 tokens, median of 10 runs after 3 warm-up runs.
Note: LLaMA-2 timings use shape-matched, randomly initialized models (same architecture and dimensions).
Model
Inference time (s)
zkLLM proof time (s)
Overhead
facebook/opt-125m
0.02606
73.9
2836
facebook/opt-350m
0.06278
111
1768
facebook/opt-1.3b
0.12510
221
1767
facebook/opt-2.7b
0.17898
352
1967
facebook/opt-6.7b
0.25211
548
2174
facebook/opt-13b
0.42097
713
1694
meta-llama/Llama-2-7b-hf
0.16413
620
3777
meta-llama/Llama-2-13b-hf
0.27887
803
2879