Nuclear arms treaties happened AFTER nukes had been demonstrated. AI pause would need to happen BEFORE ASI comes into existence.
If I had to explain the issue in just 2 sentences, those are the sentences I would say. Now let's elaborate:
1) Explaining the danger of nukes is really easy. "A giant explosion will evaporate an entire city". That's it, congratulations, you have successfully explained the danger of nukes. Now try to explain the danger of superintelligent AI in a way that will be convincing to the average person AND to policymakers, and won't require a 2-hour long lecture with a PowerPoint presentation.
ASI risk is abstract and cannot be easily demonstrated (until it's too late), and is easy to caricature as science fiction.
2) During the Cold War companies weren't selling nukes for profit. An anti-nuclear treaty constrains a weapon. An anti-AI treaty constrains a technology that can be used for coding, drug discovery, material science, physics and math, robotics, etc. Dual-use makes enforcement of a pause economically painful, and it will only get worse as diffusion of AI progresses.
3) There is no incentive to keep making nukes more and more powerful beyond a certain point. If you can destroy your enemy's cities and military bases, it's good enough. Nobody wants a nuke powerful enough to make the entire Earth uninhabitable (well, except for that one guy). But when it comes to making AI more and more intelligent, there is no "ok, intelligent enough" ceiling.
Note that I am implicitly assuming that ASI will be far more intelligent than the greatest humans (no ceiling at the human level), which brings me to point 4.
4) As of 2026, most people do not believe that a superintelligent AI will be created any time soon, if ever. Even among so-called "Godfathers of AI" there is no unanimous agreement. Bengio and Hinton have high p(doom), LeCun disagrees. Bengio and Hinton agree that ASI is (relatively) close, LeCun disagrees on that as well.
5) A pause where defection is not detectable is meaningless. It is nearly impossible to conduct a nuclear explosion test completely undetected today, even an underground test. That's not the case for frontier AI development, especially for algorithmic improvements because they do not leave physical traces. A frontier training run may look like ordinary data-center/cloud activity unless inspectors have "standing behind your shoulder" level of access.
5.5) A slightly different issue is setting thresholds for what counts as a defection. Thresholds based on training FLOPs/parameter count/benchmark numbers are arbitrary and can be gamed to a large degree. Plus, due to the "jagged frontier", a model can exceed a danger threshold on one test while remaining relatively safe on others, further complicating things.
6) With nukes, humanity had a clear "warning shot": Hiroshima and Nagasaki bombing. It made it clear to the whole world that nukes are dangerous. With ASI, there may not be a clear warning shot until it's too late. You can call it "Wake Me Up When It's Too Late" problem.
7) The pace of progress in AI is already outpacing the pace of changes in the political landscape, and this will only get worse.
Nuclear arms treaties happened AFTER nukes had been demonstrated. AI pause would need to happen BEFORE ASI comes into existence.
If I had to explain the issue in just 2 sentences, those are the sentences I would say. Now let's elaborate:
1) Explaining the danger of nukes is really easy. "A giant explosion will evaporate an entire city". That's it, congratulations, you have successfully explained the danger of nukes. Now try to explain the danger of superintelligent AI in a way that will be convincing to the average person AND to policymakers, and won't require a 2-hour long lecture with a PowerPoint presentation.
ASI risk is abstract and cannot be easily demonstrated (until it's too late), and is easy to caricature as science fiction.
2) During the Cold War companies weren't selling nukes for profit. An anti-nuclear treaty constrains a weapon. An anti-AI treaty constrains a technology that can be used for coding, drug discovery, material science, physics and math, robotics, etc. Dual-use makes enforcement of a pause economically painful, and it will only get worse as diffusion of AI progresses.
3) There is no incentive to keep making nukes more and more powerful beyond a certain point. If you can destroy your enemy's cities and military bases, it's good enough. Nobody wants a nuke powerful enough to make the entire Earth uninhabitable (well, except for that one guy). But when it comes to making AI more and more intelligent, there is no "ok, intelligent enough" ceiling.
Note that I am implicitly assuming that ASI will be far more intelligent than the greatest humans (no ceiling at the human level), which brings me to point 4.
4) As of 2026, most people do not believe that a superintelligent AI will be created any time soon, if ever. Even among so-called "Godfathers of AI" there is no unanimous agreement. Bengio and Hinton have high p(doom), LeCun disagrees. Bengio and Hinton agree that ASI is (relatively) close, LeCun disagrees on that as well.
5) A pause where defection is not detectable is meaningless. It is nearly impossible to conduct a nuclear explosion test completely undetected today, even an underground test. That's not the case for frontier AI development, especially for algorithmic improvements because they do not leave physical traces. A frontier training run may look like ordinary data-center/cloud activity unless inspectors have "standing behind your shoulder" level of access.
5.5) A slightly different issue is setting thresholds for what counts as a defection. Thresholds based on training FLOPs/parameter count/benchmark numbers are arbitrary and can be gamed to a large degree. Plus, due to the "jagged frontier", a model can exceed a danger threshold on one test while remaining relatively safe on others, further complicating things.
6) With nukes, humanity had a clear "warning shot": Hiroshima and Nagasaki bombing. It made it clear to the whole world that nukes are dangerous. With ASI, there may not be a clear warning shot until it's too late. You can call it "Wake Me Up When It's Too Late" problem.
7) The pace of progress in AI is already outpacing the pace of changes in the political landscape, and this will only get worse.