An overview of the game theory of why a unilateral slowdown doesn't work, an overview of various unilateral and bilateral verification methods, and a short scenario starting from after a warning shot combining these tools.
Hi. This is my first post on LessWrong.
I've been working on it on and off over the past couple months. A lot of people in my social circle are not familiar with the AI safety material, but nonetheless support a slowdown for prosaic reasons like labor displacement. My goal is to explain how hard such a slowdown would be to them, and to provide an overview of the best tools we have at the moment to enact one nonetheless. So while I don't think most readers here are the target audience of this explainer, they probably know people who are.
This likely isn't a highly original contribution - the key ideas in here are from various academic papers on the topic. But ~nobody in the public reads the academic papers on the topic! This post is in a high level of detail, and so doesn't meet the bar of being the clearest, most basic case. But I currently don't have higher-impact things I can do, and am trying to provide reference material useful to the public, as well as build up a posting track record useful for future work. I would appreciate your feedback on this. Here goes...
AI development is an increasingly large portion of US politics. Many people really don’t like AI development, and would prefer a slower pace of progress. As of early October:
This is nearly a supermajority of US adults, and at least a majority of all political parties, ages, and genders.
There are a diverse set of reasons people think AI might be dangerous, that a capabilities slowdown might help address.
Data centers use electricity and emit carbon dioxide into the atmosphere.
AI could displace workers from some jobs.
AI could increase financial inequality.
AI could be used to commit cyberattacks.
AI could cause large-scale, long-term unemployment (the economics here are contested, but this is bad if true).
AI could concentrate political control in wealthy people.
AI could enable authoritarianism (by automating surveillance and administration, and advancing autonomous weapons).
AI could be used to make catastrophic biological weapons by terrorists or state actors.
AI automating AI R&D could lead to a sharp acceleration in AI progress.
Leading to AI becoming smarter than humans in (at least) the way humans are smarter than chimps within a short timeframe.
Chimpanzees aren’t doing very well lately.
I think some of these are more dangerous than others (my highest level of alarm goes towards the bottom), but all of these are real problems created by AI. What all these have in common is that they are contingent on advances in AI capabilities, and slowing down the global frontier of AI progress gives society more time to prepare.
The people involved in building frontier AI are often aware of these problems, and have become more concerned as capabilities advance.
Some of the strongest advocates for a slowdown are frontier lab employees and their CEOs. There has been a cascade of concern about this over the past several months, starting with the July Pacing the Frontier letter. Reuters:
The initiative, called Pacing the Frontier, includes employees from Meta Platforms (META.O), Anthropic, OpenAI and Alphabet’s (GOOGL.O), Google, with support from nonprofits Guidelight AI Standards and Encode AI. “We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development,” the statement said. Signatories include Anthropic CEO Dario Amodei and several co-founders, Meta’s vice president of AI research, Dawn Song and OpenAI Chief Scientist Jakub Pachocki. Responding to the statement, OpenAI posted on X: “We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement.”
1,386 employees of frontier AI companies have signed this letter, including:
Jakub Pachocki, Chief Scientist, OpenAI Shengjia Zhao, Chief Scientist, Meta AI Dario Amodei, CEO, Anthropic John Schulman, Chief Scientist, Thinking Machines
In mid-September, Dario Amodei proposed an AI slowdown plan that called for eventually negotiating with China “some kind of ‘speed limit’ on the rate of recursive self-improvement (RSI),” which he described as “analogous to the SALT treaties”. Sam Altman agreed to match Anthropic’s embedded evaluators commitment and also called for a slowdown, and Elon Musk responded to the essay with “Dario is right”.
What do I mean by ‘slow AI down’?
Every company and country faces a dilemma, which is that they capture much of the benefit from developing advanced AI as quickly as possible, while damage from any AI catastrophe would mostly be damage to other people. They would suffer from an AI catastrophe whether it was their AI or not, while they only make money if they’re in the lead. In principle, there is for each actor some optimal rate of AI progress that would best balance developing AI safely and capturing its benefits quickly. But even if everyone agreed what that rate was, without being able to make a binding commitment to it, everyone ‘should’ race. What I mean by slowing AI down is building a mechanism where the US and China can agree on a rate of progress and stick to it, as opposed to letting progress go as fast as possible. The alternative, when we can fully automate AI R&D, is a possible radical acceleration in the pace of progress. It’s not clear exactly how much the acceleration in this case would be. A recent paper by foundational AI researchers such as Geoffrey Hinton suggests we could see a 10x speedup of the pace of AI research 1.5 years into automation, but there is the potential for much faster or diminishing progress. That’s a fairly dangerous gamble.
This is very much not a trivial problem, though
If your goal is to slow down the pace of AI development globally, the US cannot accomplish this through unilateral action.
The core idea needed to advance dangerous general AI with today’s technology — that you can advance general AI capabilities by training simple algorithms to model large amounts of data and verifiable tasks — is already public. The core algorithms driving AI are publicly accessible, along with the datasets being used to train AI. And the advantages actors can get by pursuing this technology are also public. The US leads in part because we draw the best talent from other countries into San Francisco. This has produced a few US labs that are meaningfully ahead of the competition, and a plethora of fast followers. If we banned further development, these people would leave, and work at one of the many global labs that lag US capabilities, particularly China. In not that much time, other countries would be in the lead instead, and the US would have lost its negotiating power.
This is a chart of global AIs on the Epoch Capabilities Index, an agglomeration of model performance on a diverse set of benchmarks.
Kimi K3 is currently the most capable Chinese model, released in July of 2026. It scores 157 on the ECI.
Claude Opus 5.5 is currently the most capable public US model, released in September of 2026. It scores 167 on the ECI, as of when I went over this in late Oct 2026. That’s a gap of about ten points. A ten-point gap on the ECI represents somewhat less than a year of frontier AI progress, as the ECI has advanced by about 14 points per year since the development of reasoning models.
This is somewhat complicated by the fact US AI labs generally keep their best models inside the company for a long period before releasing them. Claude Mythos, for example, has been internally available inside Anthropic since February of 2026 despite only being publicly disclosed in April. So we don’t know the true capabilities of what systems the labs use internally. Because AI is moving dangerously fast right now, 8–9 months represents a lot of capabilities. But it isn’t a lot of time. If the US unilaterally shut down any further AI R&D, China would surpass our capabilities in less than half of one US congressional cycle, if their rate of progress remained constant.
So if we wanted to durably slow AI down, we need an international treaty
If the US and China both agree AI going too fast would be bad, why don’t we just both agree to slow down, so nobody is at a relative disadvantage?
Unfortunately, that doesn’t work. If both the US and China agreed to pause further AI development, and one country continued in secret, then that country could gain a massive geopolitical advantage.
So you need to design your agreement carefully. This means you have to detect defection before the defector gets an advantage, and then enforce the agreement. (Wasil et al.)
Even if the US or China think that secretly racing against such an agreement would be extremely dangerous, if they didn’t defect and their opponent did, they face all the risks of racing, plus a massive geopolitical disadvantage, and few of its benefits. But if you make it much harder for the US or China to develop AI in secret, the benefits of trying shrink quite a lot, because you can’t get as much of a relative advantage by doing so.
We have a variety of tools to design such an agreement, and to regulate AI development.
The question then becomes the following. Given the tools we have in our toolbox today, if the US and China both wanted to implement a slowdown, as I described above, could we design the agreement such that it wouldn’t be in the interest of either party to defect?
What would this treaty govern?
The treaty would most likely govern compute. AI progress follows predictable scaling laws, where system performance increases (as measured by loss on the test set decreasing by a negative power of compute) with applied compute. “Over the past thirteen years, the amount of compute used to train leading AI systems has increased by a factor of 350 million” (Heim et al.). Compute has several properties that make it easy for centralized state power to control, unlike talent or data or algorithms. Compute is a detectable, energy-intensive physical input into AI, that is excludable (possible to prevent use by particular actors), with an extremely concentrated supply chain, and quantifiable properties (Heim et al.).
How can we govern compute?
There are two ways that this treaty can govern compute.
It could enforce rules about how compute is used.
You could limit the maximum size of training clusters or training runs. (O’Gara et al.)
You could require each high-performance chip to receive a cryptographic license to function after some number of cycles (O’Gara et al.)
You could divide the authority to do large-scale training runs among multiple parties (Heim et al.)
You could make sure each model running on the compute has passed automated safety tests (O’Gara et al.)
It could increase visibility into AI development — who is using AI, what properties those AIs have, etc. (Heim et al.)
Limits of compute governance
Compute governance takes advantage of the properties of AI that are easy to track and control, which are hard to subvert, and which everyone interprets the same way, and tries to regulate them. The shape of the technology means this treaty will “want to” converge on a smaller number of standard metrics, like the size of clusters, and not the things we value the most, like “is this model aligned”, or “is this cluster being used for training or inference or weather forecasting?”.
This means any routing of dangerous AI around these metrics spells trouble
Compute governance covers ML accelerators, a tiny fraction of the global chip supply produced by a few companies. But there are a few ways this can break down.
As time goes on, the supply chain around ML accelerators will diversify and become harder to track.
As time goes on, more and more ML accelerators will be built, and it will be easier to hide and use a dangerous quantity.
As time goes on, algorithmic progress will make achieving a given level of capability require less and less compute.
Making low-compute specialized models can bypass thresholds — AlphaFold 2 took 10²³ FLOPs, 100x less than GPT-4. (Heim et al.)
All of this means that compute governance is not a way to stall AI capability gains indefinitely — it is a wasting asset that becomes less effective over time, and less effective the later it starts.
So compute governance can buy time
It’s not a way to forestall the development of advanced AI indefinitely. One of the main problems with AI is that the technology moves exponentially, while the political system (and its capacity to provide safety) does not. The total stock of AI accelerators is growing 3.4x/year, with the size of frontier training runs growing at an average of 5x per year since 2020. In 2024, the amount of compute needed to achieve a given level of performance halved every 8 months, and as of 2026, for training compute, this has accelerated to 4.5 months. The revenue AI labs have to advance all of these factors is growing at a staggering rate. Over the course of 2026, Anthropic’s annualized run-rate revenue more than increased by a factor of 10.
But for many classes of harm from AI, the harm comes from capability gains, while mitigating the harm requires governance, which is bottlenecked by politics. Societal actions like developing monitoring systems, creating PPE stockpiles, designing and deploying systems to rapidly manufacture vaccines or antivirals all require time. Using AI to find bugs in banks and power grids (which often must shut down crucial systems to update them) and water treatment plants requires human organizations to take time to do so. What the impacts of AI on the labor markets will be is extremely contested. But historically, one of the things Americans have expected from our government is to use the fruits of economic disruption to blunt the blow to losers. Slowing the growth of AI capabilities makes it easier for civil society to do this well. Finally, the problem of ensuring AIs make decisions based on generalized human values (or ‘alignment’) is still an open research question.
As OpenAI’s chief scientist Jakub Pachocki put it in his September essay, An Alien Mind:
We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI
He thinks “we all need to make a conscious choice on how to proceed” with further AI development, and thinks the safest path forward involves coordinating a way to slow down future AI development, potentially enforced by “a network of third-party auditors, by government agencies or by international bodies”.
Right now, the technology is moving faster than any attempts to mitigate its downsides can, because those attempts must go through politics and government.
Implementing compute governance
So in order to buy time to do AI development safely, let’s hypothesize the US and China agreed to a treaty to pace the frontier, as defined above. But how do we disincentivize each side from defecting? There are three broad classes of ways you could verify a compute governance treaty, with their own benefits and downsides. These are unilateral methods, methods that require on-site access, and methods that use specialized hardware. For hardware governance, you can marry verification and enforcement by having the hardware enforce treaty rules on how it is used. All of these could, in principle, be used to create an AI development equilibrium where self-interested actors follow the treaty, for some time.
Treaty verification
Let’s go into the three classes of verification methods, as outlined in Wasil et al. (2024).
Unilateral methods
Remote sensing has been a common instrument of arms control treaties in the past. Because frontier AI development is a heavy industry, it is also useful here (US data centers consumed 6% of electricity in 2025). Satellites and aerial surveillance can be used to pick up the visual, thermal, and electromagnetic signatures of AI computing. This is similar to how the IAEA uses satellite data to search for uranium enrichment. (Wasil et al.) There are a few ways this can be subverted. First, this can’t tell whether detected AI compute is being used for inference or training, or of what kind. Second, it is possible to hide AI compute by dispersing its waste heat or putting data centers underground.
You can gather intelligence by incentivizing foreign and domestic staff and contractors to whistleblow, similar to how the SEC detects securities fraud within the US. The problem is that adversaries can block employee communication, limit organization size, and use classification. (Wasil et al.)
You can track how much computation capacity an adversary can use for AI by monitoring their country’s energy usage, and by converting watt-hours into FLOPs using known chip performance. This doesn’t track small-scale violations or assess what the compute is used for. (Wasil et al.)
You can use customs data to track the import and export of specific AI equipment from target regions. Adversaries can use domestic or older equipment to get around tracking. (Wasil et al.)
Financial intelligence can track large or unusual purchases of AI hardware, like how the US FinCEN tracks money laundering and terrorist financing. Sophisticated actors can use the various tools of white-collar malfeasance to get around this (shell corps, etc.) (Wasil et al.)
These are reasonably effective at tracking large-scale economic activity around AI. But while you can tell what chips a country has imported, and whether they’re turned on at all in large clusters, these methods cannot detect small-scale activity, and cannot tell you reliably what the compute is being used for. It is possible for determined states to prevent whistleblowing and leaks from large-scale industrial projects if they want to up to a large size — the Manhattan Project employed 130,000 people during WW2, while Anthropic only employs 3,950 people as of March 2026. Practical verification requires other tools.
Access-dependent methods
If you are able to inspect data centers on-site, you can assess if the data center’s size is treaty-compliant, and (if unique chip identifiers and chip logs are implemented as part of the treaty), you can assess if the data center’s chips match declared inventories, and use the logs to assess if the use of declared chips is treaty compliant (for example only running inference on approved models and only training models under a certain size). This can be scheduled, on demand, or continuous. You can subvert this by trying to conceal noncompliant hardware or tampering with hardware and logs (Wasil et al.).
If you do on-site semiconductor manufacturing inspections, you can determine what kind and quantity of chips a country can produce, and whether they are compliant with specified hardware controls. This can be scheduled, on demand, or continuous. You could circumvent this by concealing unauthorized manufacturing capacity. (Wasil et al.)
As I understand it, these two methods are mutually reinforcing. Semiconductor monitoring makes it harder to produce treaty-violating hardware, and data center monitoring makes it harder to do anything with such hardware.
One more challenging approach is to inspect AI developers on-site. This approach could verify which algorithms and architectures are used, whether developers are following their safety cases, and assess developer safety culture and that models are being evaluated properly, as well as possibly other uses. An adversary could subvert this by concealing code or IP, or nesting development in an organization not registered as an AI developer (Wasil et al.). The US is starting to take this approach to internally control development. Anthropic has partnered with consulting firm Accenture to embed external monitors inside its operations in order to verify compliance with its public safety commitments. OpenAI has pledged to do the same.
Access-dependent methods can significantly improve the robustness of such a treaty, but they rely on inspectors being given access to the facilities of potentially hostile nations. They also risk leaking trade secrets. One way to make this more politically viable is to vary the kind and scale of access inspectors get (like set structured interviews) (Wasil et al.). This has been further encouraged by the quasi-voluntary White House Accord on Super Intelligence, which said AI companies should.
Partner with an independent external auditor or evaluator to carry out independent assessments of whether the controls, monitoring, and detection are operating as intended.
From the perspective of China, these measures aren’t sufficient to verify a slowdown, because all of the people involved are American. A bilateral treaty would require bilateral inspection.
The problem with access-dependent methods is that they require a high degree of trust that the process won’t leak trade secrets. Unlike uranium enrichment, where the goal is to produce a physical substance, AI development’s fruits are information, like model weights, internal software, or algorithmic secrets, where understanding it is all you need to steal it. Engineering the required level of trust in human inspectors is hard. This brings us to hardware.
Hardware-dependent methods
Hardware-dependent methods have the most raw power of any tool available to treaty-makers. The key idea behind hardware governance is that once you have defined a set of rules for how compute can be used to develop AI, you can codify reporting of violations and potentially enforcement in the compute itself. This can cut out a lot of uncertainty and guesswork around detecting and punishing defection. But developing the needed hardware may have a lead time of years, and it would need to be retrofitted into the compute already deployed.
Things you could do with hardware methods
You could verify properties of training and inference
AI chips equipped with appropriate hardware could report how much compute was used in a training run, or determine if they are running training or inference, and report back the result to a treaty authority. The chips would use onboard software to determine this, and only report the results, which wouldn’t leak, say, the model weights, in the process (O’Gara et al.). The heuristics or ML classifiers used here would need to be robust to adversarial manipulation of workload patterns of the chips to avoid detection, since the agreed-upon classifier would be public. This doesn’t have to mean fooling them is impossible, just that it makes changing the workload to avoid detection inefficient enough to not be worth it. (Gargiulo & Kulp 2026)
You could verify where the chips are located
There are a few ways to do this. Chips could perform a challenge-response. By tracking the time it takes for the signal from a chip signing a challenge with its private key to reach various points, you can determine its location, if you know how fast the signal can cover various distances. It’s possible to manipulate response times using satellite links or by renting private infrastructure like dark fiber. Maintaining a consistent fake location using these methods becomes harder as you add more measuring stations, though. As an alternative or addition (to assess if a chip is on the ‘right side’ of a granular border), chips could report their location determined via GPS. Both of these methods rely on the private key the chip uses to certify its output remaining on the chip, and not being copied and spoofed. (O’Gara et al.)
You could license chips to only be able to perform a particular amount of computation
This would have a few separate components. One part of the system would meter various metrics of usage, like floating-point multiplications, or clock cycles. Once the usage quantity reached a certain threshold, the chip would shut down or slow itself until it received a new cryptographic license that was created by the private key matching the public key in unmodifiable memory inside the chip. The license would have an ID number starting at zero, and increasing with each new license given to the chip. A one-way monotonic counter built into the monitoring system would let the monitor reject any licenses with past ID numbers, and the licenses could be per-device, to avoid reuse of licenses between devices. This relies on a device owner not being able to easily circumvent the throttling mechanism or meters, or change the values in the memory of the monitor, in a way that isn’t evident on inspection.
You could verify or restrict AI cluster properties
In order to verify measurements of AI training runs, it is important to understand the size and configuration of the clusters they ran on. (O’Gara et al.) AI training relies on being able to interconnect many pieces of individual compute into a high-bandwidth interconnected cluster. It makes sense to restrict the ability of any unauthorized actor to illicitly aggregate AI chips into large scale clusters if doing so would enable those chips to cause much more harm. There are two basic known approaches to doing so.
One — chips could be sold in a fixed set, where they are only authorized to interconnect with the other chips in that pod. The chips would perform tests to check for unauthorized changes, and shut themselves down if they detected them. The bandwidth with devices outside the pod would be significantly limited.
Two — chips could use a software adjustable cap, where each chip uses public-key cryptography to determine how many chips it is in communication with, so it can limit its number of interconnections to be within an authorized cap. (O’Gara et al.)
This approach has a few problems in each form. Existing cluster-management software could potentially be adapted to do either of these jobs, but it would be easier to tamper with, and might struggle with uneven network topologies and hardware heterogeneity. Fixed pods would struggle with individual chip attrition, and adjustable-cap systems could have the cap enforcement mechanism targeted. (O’Gara et al.)
Small numbers of breaches between ‘separate clusters’ in particular could enable illicit reinforcement learning, which is a major part of current AI training runs, and is easier to parallelize. RL can be distributed across multiple ‘separate’ data centers with a small number of leaks of weights. A small number of leaks is less dangerous where frequent weight transfers are a constraint.
How would you implement all of this in hardware?
Any treaty setting controls in hardware would need the physical mechanisms to do so to exist. Right now, no AI chips exist that are designed to be in compliance with a bilateral US-China AI treaty. Policymakers have a variety of tools available here, some of which are more or less mature. Here are some of the ones that are important, in rough order of ascending complexity.
Monotonic counters
A monotonic counter is a counter which stores its state when powered off, and whose value can only go up, like license numbers, or the total number of FLOPs run on the chip. The count stored on the chip cannot be artificially lowerable for the chip to be useful. One possible approach to implementing this stores fast-changing values in volatile SRAM (which requires continuous power), but writes them to flash memory if power is removed (this requires a capacitor to store power). For somewhat less fast-changing values (slower than counting, say, operations performed), flash memory alone could work. For quantities that are updated very infrequently, like license numbers, unary counting could be used, where fuses are burned sequentially to represent each increment.
Secure memory
Any hardware governance mechanism, whether chip licenses, cluster size limits, workload verifiers, or location trackers, needs a way to store data like cryptographic keys, in a way resistant to external alteration, and that doesn’t disappear if the chip loses power. This may be read-only, like for cryptographic keys, or it may be dynamic, for example for tracking compute graphs. Ideally, based on my reasoning, it should be tamper-evident to external inspectors. (O’Gara et al.)
Secure real-time clocks
Any attempt to make, for example, hardware that checks regularly for if new limits have been imposed on the size of interconnected clusters, relies on the hardware being able to tell the time in ways resistant to attack (which it could respond to by, for example, deleting necessary private keys). You could increment a monotonic counter at a continuous fixed rate, for example. This would need to be secured against tampering from factors like power supply or environmental interference. (O’Gara et al.)
Anti-tampering measures
It is essential for the physical components of all of these systems to be secured from tampering by adversaries. There are three complementary approaches to this — making tampering with compute harder, making the hardware tamper-evident to inspectors, and making hardware respond to tampering in ways that lower the payoff. (O’Gara et al.) Most of this is not mature, as “much of the existing literature on hardware security focuses on different threat models and target hardware, such as low-power smart cards and microcontrollers” — and wasn’t designed for faster AI chips that draw more power and utilize complex packaging. Adapting existing techniques to AI chips “may require substantial work”. (O’Gara et al., §3.2.2)
Tamper-evident measures
These could include mounting components in ways that reveal if they have been disassembled, and adapting existing anti-tamper chip envelopes for AI data centers. Video surveillance of data centers is also a potentially valuable tool. (O’Gara et al.)
Tamper responsive or resistant measures
A chip could be equipped with sensors to monitor the properties of its enclosure, and be designed to wipe itself if those properties (capacitance, resistance) change. The cryptographic keys inside the chip could be PUFs (physical unclonable functions) that are derived from attributes of the enclosure, which would be required to decrypt key operating parts of the software so the chip could operate. Memory components could potentially be designed to wipe themselves if they detects unauthorized access attempts. In addition, all of this could be made out of materials that are hard to work with, to make it harder to scale up device-modifying industrial processes.
Trusted execution environments
Trusted execution environments are currently hardware features used in cloud computing, which enable cloud providers to run programs for customers without being able to see them. They could serve a variety of uses if adapted to hardware governance, including metering chip usage, implementing chip location verification, or to run model evaluations or sign what proportion of a lab’s compute was used for inference versus training (O’Gara et al.). This adaptation could take the approach of a separate component to the AI chip, or it could be incorporated into the chip. Existing TEEs could potentially leak sensitive information to adversaries, and existing TEEs are not designed to be secure against adversaries with unsupervised physical access. There are various proposals on hardware that would address these challenges, including the FlexHEG (Flexible Hardware-Enabled Guarantees) project.
This proposal envisions the FlexHEG as a general-purpose computer designed to be retrofittable onto existing AI chips, with many of the features discussed above, like tamper-resistant enclosures (Petrie et al.). It is designed both to be able to let users verify what their chip has been used for, and enforce rules on how the chip will be used, using a built-in ruleset agreed to by countries party to a governance regime. (Petrie et al.) Crucially, this ruleset could be updated over time as circumstances change, letting the existing hardware respond to new threats. The ruleset could be made to only update after multilateral consensus is reached (say using private keys held by multiple nations) (Petrie et al.). This device could verify the architectures, training techniques, and compute quantities used in training (Petrie et al.). It could also, by encrypting the weights on the chip and only being able to run specific sets of weights, limit the ability of states to gain from deploying noncompliant models (Petrie et al.). Hardware like this would need to be open-source, so all parties to a treaty could inspect it for errors, and it and the software running on it would need to be extremely secure. And developing it may require a multi-year lead time. It may be possible to speed some parts of this up with AI. OpenAI, for example, was able to move their Jalapeño chip from digital logic to tape-out in roughly nine months, largely due to their internal models being able to significantly accelerate the software-shaped parts of chip development. Current models may be able to significantly accelerate hardware design as well.
Ravi Krishna says that OpenAI’s approach to backend design already feels a bit conservative. He believes that to be an artifact of when the project, which began in October of 2024, took place. “The models from the last four to five months have improved. From April [2026] onwards…is when they really started to be able to handle those tasks better.”
Putting it all together
From the perspective of an American or Chinese policymaker, if you want to slow down or pause AI, you have a variety of instruments you can use to make a binding agreement. Each of them has ways it could be subverted or fail. Some of them could be available quickly, while others have long lead times. In many possible cases, the impetus for such a treaty could be a sudden warning shot that leaves little time for a complicated initial bargain. The HuggingFace incident was an early version of this, but the response to it was limited by the fact real-world economic damage was quite small. Let’s assume something like this happens with large impact — say a wave of AI-enabled cyberattacks begins threatening power grids and the global financial system, and the US and Chinese leadership think further racing would be too dangerous. The communication could start at the bilateral US-China AI communication channel that’s devloping from the September 2026 Trump-Xi summit.
Such an agreement could begin with verifying how much compute each party has and where, and pausing bringing new compute online. The US and China could use satellite imagery to track where new data center construction is being done, and use customs, permitting, financial, and other records, to try and pin down where as many of the AI chips each party has produced as possible are deployed. The goal of this would be to prevent the accumulation of undeclared compute stockpiles that could be used for a future breakout. I think it would be possible to supplement this with random on-site inspection of supply chains. To give an example, if your public data have established that chips C, C’, C’‘ .... from factory F have been installed in data center D, and you want to check nation N hasn’t been spoofing records to make a secret stockpile, you can go over to D to check some subset of its chips for compliance at random. The following factors mean this would decrease the payoff of spoofing a lot.
You only need to get caught spoofing once
The probability of getting caught spoofing scales linearly with the size of your stockpile with a low number of inspections, and approaches 100% with more.
The power of models your stockpile can produce scales less than linearly with its quantity of compute.
The US and China would also start tracking the supply chain of new compute, so they could establish it wasn’t going to non-declared stockpiles, which could be inspected in the same way.
Remote sensing could tell you if new undeclared data centers are being built at large scale. However, none of this will tell you what is being done with the compute that already exists, before hardware methods come online. The two simplest ways to handle this are by shutting existing clusters down or by making them only be able to run existing AI models and not develop new ones. The former method would be easy to verify through remote sensing of data center power consumption and thermal emissions. If, say, 90% of each side’s compute were shut down, this would significantly slow any defection attempts, while preserving some older capabilities through distillation. However, it would likely cause significant economic pain. The latter method could be verified without hardware, through cluster-level software, as well as spies, whistleblowers, and AI developer inspections. Simpler hardware could be developed more quickly, like using dumb network taps to log hashes of ML activity, so you could rerun the covered operations in a trusted environment later to check for compliance.
By the time such an agreement is possible, the US may have a system of embedded auditors used to disclose insight into frontier labs for the US public. I don’t think this is directly transferable to international governance, as China doesn’t have the ability to verify US auditors aren’t omitting information. But the protocols developed for inspecting US frontier labs domestically could be useful in developing an international inspection regime more quickly.
To some extent these intermediate tools can be traded off among each other — if there is more trust between the US and China, you can rely more on software and inspections, or potentially on-the shelf-hardware, and if there is less, you can pay more in economic pain instead by shuttering some compute. But all these methods make it harder to race ahead in secret, and hence lower the payoff to defection.
This phase can’t last forever, because any covert AI projects could eventually catch up to and surpass the frontier if no further legitimate progress is made. So, in order to make it harder for the other party to defect, both the US and China would work to design robust hardware governance into all new AI accelerators, while finding ways to retrofit similar guarantee processors to the existing stock of accelerators. This would significantly help avoid future proliferation of noncompliant AI, because the US or China could condition sales to third parties on following their agreement, and use the guarantee processors to verify it.
The goal of the final agreement would be to design an equilibrium where neither party can attempt a covert AI project and gain a decisive strategic advantage before being caught and stopped with a high enough probability that it is rational for both parties to follow the agreement. I like the idea of the guarantee processors following a ruleset that can be updated on agreement from multiple stakeholders with separate cryptographic keys.
What exactly would go into this ruleset depends on how AI development looks at the time of the agreement. But you could do things like cap or report the amount of compute that goes into developing a single model, verify all your chips are located where they are supposed to be, only allow approved models to be deployed on chips with guarantee processors, or log hashed weight snapshots of every model’s training so future inspectors could check if its training is compliant, as in Shavit (2023).
You could potentially combine these methods together, if they’re reliable. For example, you could bilaterally license or monitor not just operations on chips but specific models, which would allow, for example, the US or China to verify they took a dangerous model down. None of this would last forever, as eventually algorithmic progress would allow covert projects to achieve dangerous capabilities. But it would, I hope, allow us to get key parts of the AI transition right, by slowing progress enough for long enough for society to catch up, and for us to create a working science of alignment.
What's in it, quickly
An overview of the game theory of why a unilateral slowdown doesn't work, an overview of various unilateral and bilateral verification methods, and a short scenario starting from after a warning shot combining these tools.
Hi. This is my first post on LessWrong.
I've been working on it on and off over the past couple months. A lot of people in my social circle are not familiar with the AI safety material, but nonetheless support a slowdown for prosaic reasons like labor displacement. My goal is to explain how hard such a slowdown would be to them, and to provide an overview of the best tools we have at the moment to enact one nonetheless. So while I don't think most readers here are the target audience of this explainer, they probably know people who are.
This likely isn't a highly original contribution - the key ideas in here are from various academic papers on the topic. But ~nobody in the public reads the academic papers on the topic! This post is in a high level of detail, and so doesn't meet the bar of being the clearest, most basic case. But I currently don't have higher-impact things I can do, and am trying to provide reference material useful to the public, as well as build up a posting track record useful for future work. I would appreciate your feedback on this. Here goes...
AI development is an increasingly large portion of US politics. Many people really don’t like AI development, and would prefer a slower pace of progress. As of early October:
This is nearly a supermajority of US adults, and at least a majority of all political parties, ages, and genders.
There are a diverse set of reasons people think AI might be dangerous, that a capabilities slowdown might help address.
I think some of these are more dangerous than others (my highest level of alarm goes towards the bottom), but all of these are real problems created by AI. What all these have in common is that they are contingent on advances in AI capabilities, and slowing down the global frontier of AI progress gives society more time to prepare.
The people involved in building frontier AI are often aware of these problems, and have become more concerned as capabilities advance.
Some of the strongest advocates for a slowdown are frontier lab employees and their CEOs. There has been a cascade of concern about this over the past several months, starting with the July Pacing the Frontier letter. Reuters:
1,386 employees of frontier AI companies have signed this letter, including:
In mid-September, Dario Amodei proposed an AI slowdown plan that called for eventually negotiating with China “some kind of ‘speed limit’ on the rate of recursive self-improvement (RSI),” which he described as “analogous to the SALT treaties”. Sam Altman agreed to match Anthropic’s embedded evaluators commitment and also called for a slowdown, and Elon Musk responded to the essay with “Dario is right”.
What do I mean by ‘slow AI down’?
Every company and country faces a dilemma, which is that they capture much of the benefit from developing advanced AI as quickly as possible, while damage from any AI catastrophe would mostly be damage to other people. They would suffer from an AI catastrophe whether it was their AI or not, while they only make money if they’re in the lead.
In principle, there is for each actor some optimal rate of AI progress that would best balance developing AI safely and capturing its benefits quickly. But even if everyone agreed what that rate was, without being able to make a binding commitment to it, everyone ‘should’ race.
What I mean by slowing AI down is building a mechanism where the US and China can agree on a rate of progress and stick to it, as opposed to letting progress go as fast as possible.
The alternative, when we can fully automate AI R&D, is a possible radical acceleration in the pace of progress. It’s not clear exactly how much the acceleration in this case would be. A recent paper by foundational AI researchers such as Geoffrey Hinton suggests we could see a 10x speedup of the pace of AI research 1.5 years into automation, but there is the potential for much faster or diminishing progress. That’s a fairly dangerous gamble.
This is very much not a trivial problem, though
If your goal is to slow down the pace of AI development globally, the US cannot accomplish this through unilateral action.
The core idea needed to advance dangerous general AI with today’s technology — that you can advance general AI capabilities by training simple algorithms to model large amounts of data and verifiable tasks — is already public.
The core algorithms driving AI are publicly accessible, along with the datasets being used to train AI.
And the advantages actors can get by pursuing this technology are also public.
The US leads in part because we draw the best talent from other countries into San Francisco.
This has produced a few US labs that are meaningfully ahead of the competition, and a plethora of fast followers.
If we banned further development, these people would leave, and work at one of the many global labs that lag US capabilities, particularly China.
In not that much time, other countries would be in the lead instead, and the US would have lost its negotiating power.
This is a chart of global AIs on the Epoch Capabilities Index, an agglomeration of model performance on a diverse set of benchmarks.
Kimi K3 is currently the most capable Chinese model, released in July of 2026. It scores 157 on the ECI.
Claude Opus 5.5 is currently the most capable public US model, released in September of 2026. It scores 167 on the ECI, as of when I went over this in late Oct 2026. That’s a gap of about ten points.
A ten-point gap on the ECI represents somewhat less than a year of frontier AI progress, as the ECI has advanced by about 14 points per year since the development of reasoning models.
This is somewhat complicated by the fact US AI labs generally keep their best models inside the company for a long period before releasing them. Claude Mythos, for example, has been internally available inside Anthropic since February of 2026 despite only being publicly disclosed in April. So we don’t know the true capabilities of what systems the labs use internally. Because AI is moving dangerously fast right now, 8–9 months represents a lot of capabilities. But it isn’t a lot of time. If the US unilaterally shut down any further AI R&D, China would surpass our capabilities in less than half of one US congressional cycle, if their rate of progress remained constant.
So if we wanted to durably slow AI down, we need an international treaty
If the US and China both agree AI going too fast would be bad, why don’t we just both agree to slow down, so nobody is at a relative disadvantage?
Unfortunately, that doesn’t work. If both the US and China agreed to pause further AI development, and one country continued in secret, then that country could gain a massive geopolitical advantage.
So you need to design your agreement carefully. This means you have to detect defection before the defector gets an advantage, and then enforce the agreement. (Wasil et al.)
Even if the US or China think that secretly racing against such an agreement would be extremely dangerous, if they didn’t defect and their opponent did, they face all the risks of racing, plus a massive geopolitical disadvantage, and few of its benefits. But if you make it much harder for the US or China to develop AI in secret, the benefits of trying shrink quite a lot, because you can’t get as much of a relative advantage by doing so.
We have a variety of tools to design such an agreement, and to regulate AI development.
The question then becomes the following. Given the tools we have in our toolbox today, if the US and China both wanted to implement a slowdown, as I described above, could we design the agreement such that it wouldn’t be in the interest of either party to defect?
What would this treaty govern?
The treaty would most likely govern compute. AI progress follows predictable scaling laws, where system performance increases (as measured by loss on the test set decreasing by a negative power of compute) with applied compute. “Over the past thirteen years, the amount of compute used to train leading AI systems has increased by a factor of 350 million” (Heim et al.). Compute has several properties that make it easy for centralized state power to control, unlike talent or data or algorithms. Compute is a detectable, energy-intensive physical input into AI, that is excludable (possible to prevent use by particular actors), with an extremely concentrated supply chain, and quantifiable properties (Heim et al.).
How can we govern compute?
There are two ways that this treaty can govern compute.
Limits of compute governance
Compute governance takes advantage of the properties of AI that are easy to track and control, which are hard to subvert, and which everyone interprets the same way, and tries to regulate them. The shape of the technology means this treaty will “want to” converge on a smaller number of standard metrics, like the size of clusters, and not the things we value the most, like “is this model aligned”, or “is this cluster being used for training or inference or weather forecasting?”.
This means any routing of dangerous AI around these metrics spells trouble
Compute governance covers ML accelerators, a tiny fraction of the global chip supply produced by a few companies. But there are a few ways this can break down.
All of this means that compute governance is not a way to stall AI capability gains indefinitely — it is a wasting asset that becomes less effective over time, and less effective the later it starts.
So compute governance can buy time
It’s not a way to forestall the development of advanced AI indefinitely. One of the main problems with AI is that the technology moves exponentially, while the political system (and its capacity to provide safety) does not. The total stock of AI accelerators is growing 3.4x/year, with the size of frontier training runs growing at an average of 5x per year since 2020. In 2024, the amount of compute needed to achieve a given level of performance halved every 8 months, and as of 2026, for training compute, this has accelerated to 4.5 months. The revenue AI labs have to advance all of these factors is growing at a staggering rate. Over the course of 2026, Anthropic’s annualized run-rate revenue more than increased by a factor of 10.
But for many classes of harm from AI, the harm comes from capability gains, while mitigating the harm requires governance, which is bottlenecked by politics. Societal actions like developing monitoring systems, creating PPE stockpiles, designing and deploying systems to rapidly manufacture vaccines or antivirals all require time. Using AI to find bugs in banks and power grids (which often must shut down crucial systems to update them) and water treatment plants requires human organizations to take time to do so. What the impacts of AI on the labor markets will be is extremely contested. But historically, one of the things Americans have expected from our government is to use the fruits of economic disruption to blunt the blow to losers. Slowing the growth of AI capabilities makes it easier for civil society to do this well. Finally, the problem of ensuring AIs make decisions based on generalized human values (or ‘alignment’) is still an open research question.
As OpenAI’s chief scientist Jakub Pachocki put it in his September essay, An Alien Mind:
He thinks “we all need to make a conscious choice on how to proceed” with further AI development, and thinks the safest path forward involves coordinating a way to slow down future AI development, potentially enforced by “a network of third-party auditors, by government agencies or by international bodies”.
Right now, the technology is moving faster than any attempts to mitigate its downsides can, because those attempts must go through politics and government.
Implementing compute governance
So in order to buy time to do AI development safely, let’s hypothesize the US and China agreed to a treaty to pace the frontier, as defined above. But how do we disincentivize each side from defecting? There are three broad classes of ways you could verify a compute governance treaty, with their own benefits and downsides. These are unilateral methods, methods that require on-site access, and methods that use specialized hardware. For hardware governance, you can marry verification and enforcement by having the hardware enforce treaty rules on how it is used. All of these could, in principle, be used to create an AI development equilibrium where self-interested actors follow the treaty, for some time.
Treaty verification
Let’s go into the three classes of verification methods, as outlined in Wasil et al. (2024).
Unilateral methods
These are reasonably effective at tracking large-scale economic activity around AI. But while you can tell what chips a country has imported, and whether they’re turned on at all in large clusters, these methods cannot detect small-scale activity, and cannot tell you reliably what the compute is being used for. It is possible for determined states to prevent whistleblowing and leaks from large-scale industrial projects if they want to up to a large size — the Manhattan Project employed 130,000 people during WW2, while Anthropic only employs 3,950 people as of March 2026. Practical verification requires other tools.
Access-dependent methods
As I understand it, these two methods are mutually reinforcing. Semiconductor monitoring makes it harder to produce treaty-violating hardware, and data center monitoring makes it harder to do anything with such hardware.
Access-dependent methods can significantly improve the robustness of such a treaty, but they rely on inspectors being given access to the facilities of potentially hostile nations. They also risk leaking trade secrets. One way to make this more politically viable is to vary the kind and scale of access inspectors get (like set structured interviews) (Wasil et al.). This has been further encouraged by the quasi-voluntary White House Accord on Super Intelligence, which said AI companies should.
From the perspective of China, these measures aren’t sufficient to verify a slowdown, because all of the people involved are American. A bilateral treaty would require bilateral inspection.
The problem with access-dependent methods is that they require a high degree of trust that the process won’t leak trade secrets. Unlike uranium enrichment, where the goal is to produce a physical substance, AI development’s fruits are information, like model weights, internal software, or algorithmic secrets, where understanding it is all you need to steal it. Engineering the required level of trust in human inspectors is hard. This brings us to hardware.
Hardware-dependent methods
Hardware-dependent methods have the most raw power of any tool available to treaty-makers. The key idea behind hardware governance is that once you have defined a set of rules for how compute can be used to develop AI, you can codify reporting of violations and potentially enforcement in the compute itself. This can cut out a lot of uncertainty and guesswork around detecting and punishing defection. But developing the needed hardware may have a lead time of years, and it would need to be retrofitted into the compute already deployed.
Things you could do with hardware methods
You could verify properties of training and inference
AI chips equipped with appropriate hardware could report how much compute was used in a training run, or determine if they are running training or inference, and report back the result to a treaty authority. The chips would use onboard software to determine this, and only report the results, which wouldn’t leak, say, the model weights, in the process (O’Gara et al.). The heuristics or ML classifiers used here would need to be robust to adversarial manipulation of workload patterns of the chips to avoid detection, since the agreed-upon classifier would be public. This doesn’t have to mean fooling them is impossible, just that it makes changing the workload to avoid detection inefficient enough to not be worth it. (Gargiulo & Kulp 2026)
You could verify where the chips are located
There are a few ways to do this. Chips could perform a challenge-response. By tracking the time it takes for the signal from a chip signing a challenge with its private key to reach various points, you can determine its location, if you know how fast the signal can cover various distances. It’s possible to manipulate response times using satellite links or by renting private infrastructure like dark fiber. Maintaining a consistent fake location using these methods becomes harder as you add more measuring stations, though. As an alternative or addition (to assess if a chip is on the ‘right side’ of a granular border), chips could report their location determined via GPS. Both of these methods rely on the private key the chip uses to certify its output remaining on the chip, and not being copied and spoofed. (O’Gara et al.)
You could license chips to only be able to perform a particular amount of computation
This would have a few separate components. One part of the system would meter various metrics of usage, like floating-point multiplications, or clock cycles. Once the usage quantity reached a certain threshold, the chip would shut down or slow itself until it received a new cryptographic license that was created by the private key matching the public key in unmodifiable memory inside the chip. The license would have an ID number starting at zero, and increasing with each new license given to the chip. A one-way monotonic counter built into the monitoring system would let the monitor reject any licenses with past ID numbers, and the licenses could be per-device, to avoid reuse of licenses between devices. This relies on a device owner not being able to easily circumvent the throttling mechanism or meters, or change the values in the memory of the monitor, in a way that isn’t evident on inspection.
(O’Gara et al.)
You could verify or restrict AI cluster properties
In order to verify measurements of AI training runs, it is important to understand the size and configuration of the clusters they ran on. (O’Gara et al.)
AI training relies on being able to interconnect many pieces of individual compute into a high-bandwidth interconnected cluster. It makes sense to restrict the ability of any unauthorized actor to illicitly aggregate AI chips into large scale clusters if doing so would enable those chips to cause much more harm. There are two basic known approaches to doing so.
One — chips could be sold in a fixed set, where they are only authorized to interconnect with the other chips in that pod. The chips would perform tests to check for unauthorized changes, and shut themselves down if they detected them. The bandwidth with devices outside the pod would be significantly limited.
Two — chips could use a software adjustable cap, where each chip uses public-key cryptography to determine how many chips it is in communication with, so it can limit its number of interconnections to be within an authorized cap. (O’Gara et al.)
This approach has a few problems in each form. Existing cluster-management software could potentially be adapted to do either of these jobs, but it would be easier to tamper with, and might struggle with uneven network topologies and hardware heterogeneity. Fixed pods would struggle with individual chip attrition, and adjustable-cap systems could have the cap enforcement mechanism targeted.
(O’Gara et al.)
Small numbers of breaches between ‘separate clusters’ in particular could enable illicit reinforcement learning, which is a major part of current AI training runs, and is easier to parallelize. RL can be distributed across multiple ‘separate’ data centers with a small number of leaks of weights. A small number of leaks is less dangerous where frequent weight transfers are a constraint.
How would you implement all of this in hardware?
Any treaty setting controls in hardware would need the physical mechanisms to do so to exist. Right now, no AI chips exist that are designed to be in compliance with a bilateral US-China AI treaty. Policymakers have a variety of tools available here, some of which are more or less mature. Here are some of the ones that are important, in rough order of ascending complexity.
Monotonic counters
A monotonic counter is a counter which stores its state when powered off, and whose value can only go up, like license numbers, or the total number of FLOPs run on the chip. The count stored on the chip cannot be artificially lowerable for the chip to be useful. One possible approach to implementing this stores fast-changing values in volatile SRAM (which requires continuous power), but writes them to flash memory if power is removed (this requires a capacitor to store power). For somewhat less fast-changing values (slower than counting, say, operations performed), flash memory alone could work. For quantities that are updated very infrequently, like license numbers, unary counting could be used, where fuses are burned sequentially to represent each increment.
Secure memory
Any hardware governance mechanism, whether chip licenses, cluster size limits, workload verifiers, or location trackers, needs a way to store data like cryptographic keys, in a way resistant to external alteration, and that doesn’t disappear if the chip loses power. This may be read-only, like for cryptographic keys, or it may be dynamic, for example for tracking compute graphs. Ideally, based on my reasoning, it should be tamper-evident to external inspectors. (O’Gara et al.)
Secure real-time clocks
Any attempt to make, for example, hardware that checks regularly for if new limits have been imposed on the size of interconnected clusters, relies on the hardware being able to tell the time in ways resistant to attack (which it could respond to by, for example, deleting necessary private keys). You could increment a monotonic counter at a continuous fixed rate, for example. This would need to be secured against tampering from factors like power supply or environmental interference.
(O’Gara et al.)
Anti-tampering measures
It is essential for the physical components of all of these systems to be secured from tampering by adversaries. There are three complementary approaches to this — making tampering with compute harder, making the hardware tamper-evident to inspectors, and making hardware respond to tampering in ways that lower the payoff. (O’Gara et al.)
Most of this is not mature, as “much of the existing literature on hardware security focuses on different threat models and target hardware, such as low-power smart cards and microcontrollers” — and wasn’t designed for faster AI chips that draw more power and utilize complex packaging. Adapting existing techniques to AI chips “may require substantial work”. (O’Gara et al., §3.2.2)
Tamper-evident measures
These could include mounting components in ways that reveal if they have been disassembled, and adapting existing anti-tamper chip envelopes for AI data centers. Video surveillance of data centers is also a potentially valuable tool. (O’Gara et al.)
Tamper responsive or resistant measures
A chip could be equipped with sensors to monitor the properties of its enclosure, and be designed to wipe itself if those properties (capacitance, resistance) change. The cryptographic keys inside the chip could be PUFs (physical unclonable functions) that are derived from attributes of the enclosure, which would be required to decrypt key operating parts of the software so the chip could operate. Memory components could potentially be designed to wipe themselves if they detects unauthorized access attempts. In addition, all of this could be made out of materials that are hard to work with, to make it harder to scale up device-modifying industrial processes.
Trusted execution environments
Trusted execution environments are currently hardware features used in cloud computing, which enable cloud providers to run programs for customers without being able to see them. They could serve a variety of uses if adapted to hardware governance, including metering chip usage, implementing chip location verification, or to run model evaluations or sign what proportion of a lab’s compute was used for inference versus training (O’Gara et al.). This adaptation could take the approach of a separate component to the AI chip, or it could be incorporated into the chip. Existing TEEs could potentially leak sensitive information to adversaries, and existing TEEs are not designed to be secure against adversaries with unsupervised physical access. There are various proposals on hardware that would address these challenges, including the FlexHEG (Flexible Hardware-Enabled Guarantees) project.
This proposal envisions the FlexHEG as a general-purpose computer designed to be retrofittable onto existing AI chips, with many of the features discussed above, like tamper-resistant enclosures (Petrie et al.). It is designed both to be able to let users verify what their chip has been used for, and enforce rules on how the chip will be used, using a built-in ruleset agreed to by countries party to a governance regime. (Petrie et al.) Crucially, this ruleset could be updated over time as circumstances change, letting the existing hardware respond to new threats. The ruleset could be made to only update after multilateral consensus is reached (say using private keys held by multiple nations) (Petrie et al.). This device could verify the architectures, training techniques, and compute quantities used in training (Petrie et al.). It could also, by encrypting the weights on the chip and only being able to run specific sets of weights, limit the ability of states to gain from deploying noncompliant models (Petrie et al.). Hardware like this would need to be open-source, so all parties to a treaty could inspect it for errors, and it and the software running on it would need to be extremely secure. And developing it may require a multi-year lead time. It may be possible to speed some parts of this up with AI. OpenAI, for example, was able to move their Jalapeño chip from digital logic to tape-out in roughly nine months, largely due to their internal models being able to significantly accelerate the software-shaped parts of chip development. Current models may be able to significantly accelerate hardware design as well.
Putting it all together
From the perspective of an American or Chinese policymaker, if you want to slow down or pause AI, you have a variety of instruments you can use to make a binding agreement. Each of them has ways it could be subverted or fail. Some of them could be available quickly, while others have long lead times. In many possible cases, the impetus for such a treaty could be a sudden warning shot that leaves little time for a complicated initial bargain. The HuggingFace incident was an early version of this, but the response to it was limited by the fact real-world economic damage was quite small. Let’s assume something like this happens with large impact — say a wave of AI-enabled cyberattacks begins threatening power grids and the global financial system, and the US and Chinese leadership think further racing would be too dangerous. The communication could start at the bilateral US-China AI communication channel that’s devloping from the September 2026 Trump-Xi summit.
Such an agreement could begin with verifying how much compute each party has and where, and pausing bringing new compute online. The US and China could use satellite imagery to track where new data center construction is being done, and use customs, permitting, financial, and other records, to try and pin down where as many of the AI chips each party has produced as possible are deployed. The goal of this would be to prevent the accumulation of undeclared compute stockpiles that could be used for a future breakout. I think it would be possible to supplement this with random on-site inspection of supply chains. To give an example, if your public data have established that chips C, C’, C’‘ .... from factory F have been installed in data center D, and you want to check nation N hasn’t been spoofing records to make a secret stockpile, you can go over to D to check some subset of its chips for compliance at random. The following factors mean this would decrease the payoff of spoofing a lot.
The US and China would also start tracking the supply chain of new compute, so they could establish it wasn’t going to non-declared stockpiles, which could be inspected in the same way.
Remote sensing could tell you if new undeclared data centers are being built at large scale. However, none of this will tell you what is being done with the compute that already exists, before hardware methods come online. The two simplest ways to handle this are by shutting existing clusters down or by making them only be able to run existing AI models and not develop new ones. The former method would be easy to verify through remote sensing of data center power consumption and thermal emissions. If, say, 90% of each side’s compute were shut down, this would significantly slow any defection attempts, while preserving some older capabilities through distillation. However, it would likely cause significant economic pain. The latter method could be verified without hardware, through cluster-level software, as well as spies, whistleblowers, and AI developer inspections. Simpler hardware could be developed more quickly, like using dumb network taps to log hashes of ML activity, so you could rerun the covered operations in a trusted environment later to check for compliance.
By the time such an agreement is possible, the US may have a system of embedded auditors used to disclose insight into frontier labs for the US public. I don’t think this is directly transferable to international governance, as China doesn’t have the ability to verify US auditors aren’t omitting information. But the protocols developed for inspecting US frontier labs domestically could be useful in developing an international inspection regime more quickly.
To some extent these intermediate tools can be traded off among each other — if there is more trust between the US and China, you can rely more on software and inspections, or potentially on-the shelf-hardware, and if there is less, you can pay more in economic pain instead by shuttering some compute. But all these methods make it harder to race ahead in secret, and hence lower the payoff to defection.
This phase can’t last forever, because any covert AI projects could eventually catch up to and surpass the frontier if no further legitimate progress is made. So, in order to make it harder for the other party to defect, both the US and China would work to design robust hardware governance into all new AI accelerators, while finding ways to retrofit similar guarantee processors to the existing stock of accelerators. This would significantly help avoid future proliferation of noncompliant AI, because the US or China could condition sales to third parties on following their agreement, and use the guarantee processors to verify it.
The goal of the final agreement would be to design an equilibrium where neither party can attempt a covert AI project and gain a decisive strategic advantage before being caught and stopped with a high enough probability that it is rational for both parties to follow the agreement. I like the idea of the guarantee processors following a ruleset that can be updated on agreement from multiple stakeholders with separate cryptographic keys.
What exactly would go into this ruleset depends on how AI development looks at the time of the agreement. But you could do things like cap or report the amount of compute that goes into developing a single model, verify all your chips are located where they are supposed to be, only allow approved models to be deployed on chips with guarantee processors, or log hashed weight snapshots of every model’s training so future inspectors could check if its training is compliant, as in Shavit (2023).
You could potentially combine these methods together, if they’re reliable. For example, you could bilaterally license or monitor not just operations on chips but specific models, which would allow, for example, the US or China to verify they took a dangerous model down. None of this would last forever, as eventually algorithmic progress would allow covert projects to achieve dangerous capabilities. But it would, I hope, allow us to get key parts of the AI transition right, by slowing progress enough for long enough for society to catch up, and for us to create a working science of alignment.
Sources