Really appreciate your laying out your thoughts here. Scattered late-night replies:
#0: I think this is a duplicate of #3.
#1: In one sense, all the great powers have an interest in not building superintelligence---they want to live! Of course, some great powers may disagree; they would like to subvert the treaty. But they still would prefer lesser powers not develop AGI! Indeed, everyone in this setup would prefer that everyone else abide by the terms of the treaty, which ain't nothing, and in fact is about as much as you'd hope for in the nuclear setting.
#2: I don't think the economic value here is a huge incentive by itself, though I agree it matters. If it's illegal to develop AGI, then any use of AGI for your economy would have to be secret. Of course, if developing AGI gives you a strategic advantage you can use to overwhelm the other nations of the world, then that's an incentive in itself! There's also the possibility that you develop safe AGI, so nobody bothers to enforce the treaty or punish you for breaking it. And this AGI is contained and obedient, and you prevent other countries from having it, so you crush the nations of the world? Or you share it globally and everyone is rich---in this story, your country wants what's best for the world but also believes that it knows what is best for the world. Plausible. (See: Iraq war.) All kinds of reasonable stories you can tell here, but the incentive isn't clear-cut. It would be a harder sell.
#3: Yeah, this is a real difficulty.
#4: Yep, even if we get a pause, the clock is (stochastically) on. Which means we must use the pause wisely and fast. But as for the procedural "difficulty in drawing a red line," I'm not too worried about that. We can pick arbitrary thresholds at which to set the legal compute cap. Sure, it's hard to know what's "correct," but I'm not too worried about that.
#5: Possibly. Parts of it are harder, parts of it are easier. Centrifuges are easier to hide than Stargate, but a single server rack is easier still. An international third party of verifiers could eliminate a lot of the mutual mistrust issues. Of course, that's not incorruptible, but hey, nothing's perfect. Probably someone should be working on "cryptographic methods for verifying that no frontier AI training is going on without disclosing much else about what's happening on these chips."
I agree with you that on net, this is a harder treaty to enforce than the global nuclear weapons treaties. On the other hand, I don't think we've tried that hard on the nonproliferation, relative to all the other things our civilization tries hard at. The obstacles you describe seem surmountable, at least for a while. And all we need to do is buy time. It ain't intractable. And I have no better ideas.
overnOne of the basic suggestions for dealing with the threat of AI is a global treaty banning training sufficiently large models. Summarizing the forthcoming If Anyone Builds it Everyone Dies, Scott Alexander describes this plan as "more-or-less lifted from the playbook for dealing with nuclear weapons."[1]
I am deeply skeptical that anything along the lines of the nuclear non-proliferation regime will work for AI. My goal here is to sketch out some of the basic reasons why, beginning with a brief summary of the non-proliferation regime that you can safely skip if you're familiar with it.
Nuclear weapons are a 1940s-era technology. In this sense, it quite remarkable that we have managed to limit their diffusion. In 1963, John F. Kennedy observed: "I am haunted by the feeling that by 1970, unless we are successful [at non-proliferation], there may be 10 nuclear powers instead of four, and by 1975, 15 or 20." Judged by this standard, the non-proliferation regime has been a success. As of 2025, there are only nine nuclear powers, and only ten states have ever developed nuclear weapons.[2] Another dozen states or so have seriously pursued nuclear weapons before eventually abandoning those efforts under international pressure.
Legally speaking, the center piece of the non-proliferation regime is the Treaty on the Non-Proliferation of Nuclear Weapons (NPT), originally signed in 1968 and now ratified by all but five countries.[3] The NPT set up a two-tier system. The five countries with acknowledged nuclear weapons as of 1968 joined the treaty as nuclear weapons states; all others joined as non-nuclear weapons states.[4]
The non-nuclear weapons NPT members agreed not to develop nuclear weapons. In return, they obtained two basic promises from the nuclear states: first, the nuclear weapons states agreed to provide assistance in developing peaceful uses of nuclear energy (a promise that they have largely honored). Second, the nuclear weapons states pledged to pursue "a treaty on general and complete disarmament under strict and effective international control," which would eventually end the disparate treatment (a promise that has not been honored).
Three basic mechanisms backstop the NPT. First, the International Atomic Energy (IAEA) safeguards system exists to verify compliance with non-proliferation obligations; for example, verifying that states are not diverting materials from their declared peaceful programs for military use. This allows for the fulfillment of the NPT bargain as nuclear weapons states can safely share nuclear information with non nuclear weapons states with a degree of assurance that it will not be abused. Second, the Nuclear Suppliers Group limits the exports of nuclear technology. Third, and most importantly, the great powers (most notably the United States) have used military, political, and economic power to prevent proliferation. It is dubious that the other measures would do much good without this.
The best argument in favor of the NPT regime is that it has technically never failed. No NPT member has ever built nuclear weapons. All but one of the subsequent proliferators declined to join the NPT, and so they never violated it. North Korea joined the NPT, announced its withdrawal consistent with the provisions of the treaty, and then built nuclear weapons after withdrawing.[5]
The best argument against the NPT regime is that the preceding paragraph hinges on an irrelevant technicality, and that the number of nuclear states has gradually grown over time. At least compared to expectations in the 1960s, the NPT regime has clearly blunted proliferation, but it has not stopped it.
The most important (and least interesting) argument against a model based on the non-proliferation regime is that its level of success is not nearly good enough. There is a reason that no one is writing books entitled "If Eleven Countries Build AI, Everybody Dies." So, at baseline, any kind of international arrangement has to work considerably better than the NPT.
The simplest cynical explanation of the non-proliferation regime is as a cartel. Having developed nuclear weapons, the nuclear weapons states (who, not coincidentally, are also the five permanent members of the UN Security Council) wanted to stop other states from joining the club. Whatever their differences with one another, the major powers had a meaningful shared interest in preventing the emergence of any additional nuclear rivals. The NPT, then, only worked because it did not ask the great powers to give up anything important.
The failure of NPT Article VI (requiring the nuclear weapons states to "pursue negotiations in good faith... on a treaty on general and complete disarmament under strict and effective international control") further confirms the inability of the NPT to constrain the great powers.
A second important aspect of the NPT regime is the ability to separate civilian nuclear power (which is economically valuable, though it has failed to live up to early expectations) from military use. While there have been some crazy schemes to use nuclear explosives peacefully, nuclear weapons are really only useful as weapons. There is no other compelling reason to build them.
Because nuclear weapons have little economic value, states either build them deliberately or not at all. Only militaries and perhaps terrorist groups have any reason to conduct the kind of work needed to build one.
AGI, of course, has extreme economic potential. Private companies with no apparent interest in using it to destroy humanity are building it right now. There will always be a different, and greater, interest in building AI than in building nuclear weapons.
In this sense, AGI is more like drugs than weapons. There is an inexorable, irresistible economic logic to it that cannot be stopped. In the 1990s, there was a great deal of concern about black market sales of nuclear material -- centered especially on the idea of some disaffected Russian selling a "suitcase nuke" to a rogue state or group. To this day, there is still no effective international safeguard against something like this. More than anything, the lack of such transactions reflects a lack of demand.
AGI is not like that. If everyone doesn't die, developing AGI assures unbelievable wealth. When unbelievable wealth is available, capitalism finds a way, as most clearly demonstrated by the fact that extreme levels of coercion and otherwise intolerable intrusions on liberty are not good enough to stop the smuggling of drugs even into prisons. So long as the incentive is there, someone will follow it. If you squelch one pathway, another will open.
The economic value also means that any control project must worry not only about nation states but about many, many non-state actors. Anyone with a few hundred decent engineers and a data center is a potential threat; this is very different than worrying about perhaps two dozen states around the world. Policing such a regime is much harder; states that are willing to do their own policing domestically will have to intrude heavily into civil liberties. In areas where domestic policing is unavailable (though lack of willingness or lack of capability), policing from the outside is borderline impossible. Perhaps the saving grace is that most of the people capable of building AGI likely don't want to move to a lab under the control of a Mexican cartel or in Somalia or something, but when vast amounts of money are at stake, anything is possible.
Finally, the domestic politics are very significant when large amounts of money are on the table. No one gets rich building nuclear weapons, so the domestic push to do so is always limited. If the great powers could negotiate a very convincing treaty to ban nuclear weapons, there wouldn't be many powerful actors in the US that any reason to object. On the other hand, many people expect to get very, very rich building AGI. The threat against any ban, then, comes from the domestic level just as much as the international level.[6]
As noted, the goal of an international AI treaty would be to completely stop anyone from developing sufficiently powerful AI and not merely to blunt the number of AI powers. This is a qualitatively different goal than the one of the NPT regime.
Unfortunately, the rewards to cheating on an arms control treaty grow larger as the treaty becomes more effective/imposes stricter limits. This is why nuclear arms control treaties between the US and Russia have always set relatively high allowances. If we agree on 1,000 warheads each, and one side builds 1,800 then the strategic impact is muted. If we have agreed on zero, and someone builds 800, then they have gained a decisive edge. Cheating from 1,000 to 1,800 often won't be worth much; cheating from 0 to 800 means you may well be able to dominate the world.
Given that the major powers already have nuclear weapons, the upsides to building a nuclear weapon are limited and heavily blunted by the existence of those other nuclear powers. Building nuclear weapons has given North Korea an excellent assurance that no outside power will invade its territory, but little more than that. In contrast, in a world of zero, being the only nuclear power might well allow global domination or at least far more ambitious goals than mere security -- surely, if North Korea had nuclear weapons and the United States did not, they would be able to rapidly "reunify" the peninsula.
The existence of other nuclear powers (and deterrence) also means that some level of failure in the NPT regime is tolerable. The same is not possible when the goal is zero. This is why the second goal of the NPT (eventual nuclear disarmament) has been a failure. Going to zero introduces risks far greater than going to some low-but-not-zero number.
Holding AGI to zero means the rewards to defection are potentially enormous, and thus the incentives to cheat are orders of magnitude larger.
While criticality accidents sometimes occur, one does not end up with a usable nuclear weapon by tinkering with some other system in an attempt to produce something else. The Manhattan Project did not assemble to play around with fission and then stumble into weapons. There is a very clear, bright-line difference between a nuclear weapon and anything else.
Half the problem with AI is that we have no idea if any such bright line exists or where it is. Someone tinkering around with something could plausibly create AGI or at least a major step towards it. This is harder to control and produces more conflict, given the economic incentive to push things as far as one safely can.
Even worse, the threshold will fall over time. Right now, training a frontier AI model takes a lot of effort and a decidedly abnormal level of compute. But, this will change over time and "ordinary" compute will gradually converge towards what is needed to train a large model. Assume that the following basic propositions are true: 1) Moore's law or something like it will continue to hold and 2) even if we train no further large models, algorithms relevant to AI in one way or another will continue to improve.
In this case, the day will eventually come when a handful of engineers with access to consumer hardware can build AGI by themselves. When this is your threat vector, it cannot be controlled.
Nuclear weapons are not like this. There has been no diffusion of nuclear technology for civilian use because there is simply no economic reason for it. If it turned out that, over time, we all carried around very small but increasing quantities of enriched uranium in our pockets, nuclear terrorism would eventually be a certainty. Thankfully, we do not do this. But, unless we do something to stop technological progress writ large, we will all gradually carry around greater and greater compute in our pockets.
Perhaps the timeline here is long enough to save us (in that someone will figure something out before three guys in a suburban home can train AGI), but that's a risky bet if you think that short timelines are currently plausible. The number of engineers at the frontier labs is not very large; and while compute budgets are currently very impressive, Moore's law is very fast. To stop this, you somehow have to freeze not just AI research but nearly everything computational in place.
Monitoring compliance is the central challenge for any arms control agreement. This is hard enough for nuclear weapons. It would be well-nigh impossible for AI.
Monitoring has to be able to navigate a difficult trade-off: intensive monitoring is necessary to ensure compliance but intensive monitoring also threatens other national secrets. In the nuclear case, that means (at a minimum), that inspectors need access to reactors, enrichment facilities, and reprocessing facilities. The good news about these is that they're not too sensitive from a national security perspective unless you're doing something illicit there. Allowing the IAEA in to visit your reactors does not generally risk spillage of, say, military secrets.
But AI monitoring is not like that. Imagine what China would need to do in order to be confident that the US was not secretly training models. At a minimum, they would need something along the lines of access to every data center in the US. So, what happens when the Chinese show up to inspect the NSA's Utah Data Center? Do you think they'll just walk away reassured after the inspectors are told "don't worry, we're not training models in there; we're just using this data center to spy on your military"? And, on the flip side, do you really think the US government can be induced to let the inspectors in?
It's not just government data at risk. China is engaged in a very active campaign to steal as many American corporate secrets as possible, and those secrets largely live in data centers. Letting Chinese inspectors have access is like giving them the keys to the castle (and, of course, precisely the same logic applies in reverse).
Even if you somehow solve that problem, giving access to declared data centers is not good enough. Nothing is stopping the US from building an off-grid data center with a dedicated power supply in a bunker on a military base. What if the Chinese inspectors want to go poke around the Pentagon and make sure there are no hidden servers there? And so on.
The difficult of doing this on the nuclear side is already severe if a state is suspected of clandestine activities. The IAEA failed to find Iraq's nuclear program in the 1980s because its inspections were too deferential. UNSCOM and the United States made precisely the opposite mistake in the 1990s, making demands that were so intrusive that Saddam refused to comply even though he did not have an active nuclear program.
The problem is much worse on the AI side because dual use technology is so much more common, and data centers can be concealed more easily than nuclear sites. While this problem could perhaps eventually be overcome, the great powers also have spent decades refining their intelligence capabilities for detecting and monitoring nuclear sites. They have no comparable expertise in detecting AI training.
There's a risk in responding to a claim from a book that isn't publicly out yet based on a review, but this all pretty much aligns with existing statements and I trust Scott as a summarist.
South Africa developed a small nuclear arsenal, which it dismantled in 1989.
Those being India, Israel, North Korea, and Pakistan (all of which have nuclear weapons that the treaty would prohibit) and South Sudan (which will presumably get around to ratification at some point)
Israel had already developed nuclear weapons by this point but, even today, has not officially acknowledged this fact.
There is some debate about the precise technicalities here. The NPT allows states to withdraw after giving three months notice. North Korea announced its withdrawal in March 1993, then agreed to "suspend" its withdrawal in negotiations with the US and the UN. In January 2003, North Korea announced an immediate withdrawal from the treaty, arguing that three 3 month notice requirement had been met in 1993.
Along these lines, note that resistance to climate treaties in the US typically has very little to do with foreign policy and a great deal to do with economic interests.