TL;DR: There seems to be a general lack of awareness of how tractable/feasible verification mechanisms for international AI governance agreements are, and correcting this would probably make pacing proposals more politically feasible.
The Skeptic’s Case for Verification
The most common objection I see to proposals for pacing (or halting) frontier AI development (in the interest of mitigating the catastrophic and existential risk posed by not doing so) is that it’s pointless; China will continue due to the economic and military advantages of doing so, surpass us in the “AI race”, and either gain geopolitical dominance or kill us all anyway. As Ted Cruz put it, “If there are going to be killer robots, I would rather they be American killer robots, rather than Chinese killer robots.” The answer is to seek an international agreement with China (and everyone else) for us to all pace frontier development.
The most common objection I see to this proposal is that it’s pointless; they will covertly defect, surpass us in the “AI race”, and either gain geopolitical dominance or kill us all anyway. The answer is to develop the means to verify compliance. Just as DARPA’s Project Vela was established to develop means to verify compliance with the 1963 Limited Nuclear Test Ban Treaty, and the International Monitoring System was developed to verify compliance with the 1996 Comprehensive Nuclear Test-Ban Treaty, so too may we be able to develop means to verify compliance with the global pacing of AI progress.
The most common objection I see to this proposal is that it’s pointless; given how many advanced AI chips are already scattered throughout the world, verifying that AI research isn’t continuing anywhere is either totally infeasible or would require such intrusive/costly surveillance and restrictions that no one would ever agree to it. Some secret lab would continue on, surpass us in the “AI race”, and either gain geopolitical dominance or kill us all anyway.
The answer is that it’s not so cut and dry; frontier AI progress largely hinges on accumulating and using together massive amounts of specialized chips whose production is concentrated to only a handful of companies. Data centers are huge, hard to hide, and there almost certainly hasn’t been enough incentive yet for any lab or country to go through the cost of proactively hiding an AI development project with enough compute to feasibly advance frontier capabilities. Strengthening the security and tracking of AI chips (among other strategies) would likely be enough to catch all such projects.
“And then what?” the skeptic may object, “Shut it all down? Subject everything to such heavy bureaucracy that it effectively does, or not enough to make much of a difference in the first place?” Not necessarily! There are several proposed or already underway projects for developing lower-cost means to verify that only certain workloads are being run on the chips, such as that they’re only being used to serve existing models and not for any unapproved large training-runs of new models.
“But how do we know any of this will pan out?” the skeptic may object. We don’t! But given what we know so far, it seems feasible and tractable enough to be worth trying. That is, explore our options for verification and potentially fund programs for accelerating the development of verification technologies. IFP offers specific policy recommendations for doing this here.
“Alright, but why would we do any of that? All this talk of AI doom is just hysteria, after all,” the skeptic may object. This has been far more discussed, and I don’t expect to change your mind on it if nothing else has yet[1]. Fortunately, I don’t have to! All this requires you to accept is that there’s enough risk posed by unbounded AI development that it’s worth trying to develop the option to internationally pace it. Given the rapid rate of AI progress, and insufficient investment into safety as evident by alltherogueagentincidents (presumably largely as the result of current competitive pressure), this is a much harder claim to deny. Keep in mind, over 1,300 frontier lab employees have already expressed support of this. So has famous “GenAI skeptic” Gary Marcus, exactly on the grounds that AI doom is overblown but current evidence of risk is enough to warrant this regardless.
“But what about open-source, and stifling innovation?” the skeptic may still object, “they may say it will only be for the biggest training runs, but in practice this is going to be used as a tool of control, and effectively insulate the incumbents from most competition! And what if particularly compute-efficient techniques for training frontier models are discovered? You’ll either be screwed or have to implement unacceptably authoritarian intervention anyway!” These are valid concerns, and although it currently seems like a resolvable tradeoff[2], it may not pan out! But again, I’m only proposing we try to prepare better options for pacing frontier development before we need them in the case that we do. If AI incidents do continue to a clearly unacceptable level (which I would argue they already have, but this is an appeal to the skeptics), it’s much more likely to either really be too late at that point or only be solvable with far harsher interventions. Nuclear non-proliferation has been far from perfect, but it was still clearly worth trying; there are far fewer states with nuclear weapons now (and thus a far lower chance of nuclear war) than there otherwise would have been if we had started later. I don’t think the handful of additional countries that have acquired nuclear weapons since we started would warrant saying “see, all the nuclear non-proliferation junk was a total waste of time!” And for reference, most AI regulation proposals use models trained with at least 10^25 or 10^26 FLOPs as the threshold for which they apply, which Jack Clark estimated in 2024[3] would cost roughly $10M or $100M respectively, and that both thresholds but especially the latter would only hit the top handful of labs. And yes, we can’t perfectly control the spread of unfiltered open-weights models, but it still makes sense to explore our options for limiting their risk!
(If you remain unconvinced, here’s a much more technically-detailed case. And feel free to make the opposing case in the comments of this post.)
The Advocacy Gap
There will be those who still oppose this, largely from a heuristic of hard skepticism to any government intervention, or motivated reasoning from financial incentive; the Trump Administration is still letting Nvidia sell chips to China, despite blocking this being the most obvious action to take if the concern really is beating China as they claim! It’s because of this that I’m not optimistic that the Administration’s position is going to flip before the next election, short of some extraordinarily legible warning shot or other unprecedented jump in salience.[4] But my vision here is to raise the political salience of verification proposals. AI governance researchers have been developing the ideas for a while now, and yet I almost never see them addressed to the White House and Congress where it matters most. There’s a sort of “advocacy gap” here. Mostly, the only response given to “but China” by pacing advocates that I’ve seen has been “it’s in China’s interest to pace AI development too”. This is true! But as a result, international governance proposals implicitly hinge on whether the person they’re being pitched to wants to accept a much higher (perceived) risk of “losing” to China due to covert defection when they hardly have to, and I suspect many of the politicians who do support chip export controls to China but not an AI treaty could have their minds swayed if made aware of this. I also suspect that even when verification is proposed, there’s an implicit assumption that it’s too intractable to take seriously that goes unaddressed, likely from not being aware of how centralized AI chip production and large-scale training runs are or the progress made thus far. For those who remain opposed, raising salience of this limits the reasonable responses from “but China!” or “but a deal with China is infeasible!” to “risk is so low that it’s not worth exploring our options for trying”. That’s a much harder sell now, and I expect it will become increasingly so as worse incidents happen. And let me remind you that China is already showing interest in this, and that there’s momentum among the rest of the UN!
So, insofar as you can, correct this![5] Bring it up to White House officials and members of Congress, or get more of those who can to do so. Get it addressed at international AI governance dialogues partaken in by the US (such as the G20 summit in December), and by journalists, and in media appearances, and by members of Congress and the White House, and by advocates/advisors they trust, etc.
As in, only restricting the most expensive training runs is probably enough to mitigate most of the problem, and there’s more to be done in making compliance cheap.
I would have liked to make this point using more up-to-date estimation methodology and checking whether the thresholds would have applied to MiMo-v2.6, the top open-weights model on Artificial Analysis at the time of writing, but much of the relevant detail is fuzzy or opaque. I’m inclined to believe 10^26 wouldn’t have applied and 10^25 maybe would have given efficiency improvements that have happened since, but please let me know if you can make a proper estimate.
Broadly speaking, it seems like they’ve been more willing to regulate so long as it only hits the biggest players and not the rest of the ecosystem. Their security framework explicitly only applies to frontier closed-weights models, and they’ve placed sudden export controls on Anthropic’s Fable in the past; maybe including stipulations for this will make an agreement more politically feasible?
Let's focus in on the technical question of how feasible these verification proposals are. What do you think the main proposals are? What are the main technical arguments for feasibility? You linked a couple posts but what do you think the best arguments/proposals are (given one of the posts is now deprecated)? What do you think of ZK proofs having a large slowdown? What about how trusted computing security is hard (given the "attacker" has access to the device) e.g. gpu-unlock? How are you estimating trusted computing security difficulty relative to web server difficulty?
I ask because, if you are asking people to advocate for something, it helps to determine whether the thing is correct first (to tell if advocating for it is a good idea), and to help argue for the correct conclusions about the idea, whatever those conclusions may be.
TL;DR: There seems to be a general lack of awareness of how tractable/feasible verification mechanisms for international AI governance agreements are, and correcting this would probably make pacing proposals more politically feasible.
The Skeptic’s Case for Verification
The most common objection I see to proposals for pacing (or halting) frontier AI development (in the interest of mitigating the catastrophic and existential risk posed by not doing so) is that it’s pointless; China will continue due to the economic and military advantages of doing so, surpass us in the “AI race”, and either gain geopolitical dominance or kill us all anyway. As Ted Cruz put it, “If there are going to be killer robots, I would rather they be American killer robots, rather than Chinese killer robots.” The answer is to seek an international agreement with China (and everyone else) for us to all pace frontier development.
The most common objection I see to this proposal is that it’s pointless; they will covertly defect, surpass us in the “AI race”, and either gain geopolitical dominance or kill us all anyway. The answer is to develop the means to verify compliance. Just as DARPA’s Project Vela was established to develop means to verify compliance with the 1963 Limited Nuclear Test Ban Treaty, and the International Monitoring System was developed to verify compliance with the 1996 Comprehensive Nuclear Test-Ban Treaty, so too may we be able to develop means to verify compliance with the global pacing of AI progress.
The most common objection I see to this proposal is that it’s pointless; given how many advanced AI chips are already scattered throughout the world, verifying that AI research isn’t continuing anywhere is either totally infeasible or would require such intrusive/costly surveillance and restrictions that no one would ever agree to it. Some secret lab would continue on, surpass us in the “AI race”, and either gain geopolitical dominance or kill us all anyway.
The answer is that it’s not so cut and dry; frontier AI progress largely hinges on accumulating and using together massive amounts of specialized chips whose production is concentrated to only a handful of companies. Data centers are huge, hard to hide, and there almost certainly hasn’t been enough incentive yet for any lab or country to go through the cost of proactively hiding an AI development project with enough compute to feasibly advance frontier capabilities. Strengthening the security and tracking of AI chips (among other strategies) would likely be enough to catch all such projects.
“And then what?” the skeptic may object, “Shut it all down? Subject everything to such heavy bureaucracy that it effectively does, or not enough to make much of a difference in the first place?” Not necessarily! There are several proposed or already underway projects for developing lower-cost means to verify that only certain workloads are being run on the chips, such as that they’re only being used to serve existing models and not for any unapproved large training-runs of new models.
“But how do we know any of this will pan out?” the skeptic may object. We don’t! But given what we know so far, it seems feasible and tractable enough to be worth trying. That is, explore our options for verification and potentially fund programs for accelerating the development of verification technologies. IFP offers specific policy recommendations for doing this here.
“Alright, but why would we do any of that? All this talk of AI doom is just hysteria, after all,” the skeptic may object. This has been far more discussed, and I don’t expect to change your mind on it if nothing else has yet[1]. Fortunately, I don’t have to! All this requires you to accept is that there’s enough risk posed by unbounded AI development that it’s worth trying to develop the option to internationally pace it. Given the rapid rate of AI progress, and insufficient investment into safety as evident by all the rogue agent incidents (presumably largely as the result of current competitive pressure), this is a much harder claim to deny. Keep in mind, over 1,300 frontier lab employees have already expressed support of this. So has famous “GenAI skeptic” Gary Marcus, exactly on the grounds that AI doom is overblown but current evidence of risk is enough to warrant this regardless.
“But what about open-source, and stifling innovation?” the skeptic may still object, “they may say it will only be for the biggest training runs, but in practice this is going to be used as a tool of control, and effectively insulate the incumbents from most competition! And what if particularly compute-efficient techniques for training frontier models are discovered? You’ll either be screwed or have to implement unacceptably authoritarian intervention anyway!” These are valid concerns, and although it currently seems like a resolvable tradeoff[2], it may not pan out! But again, I’m only proposing we try to prepare better options for pacing frontier development before we need them in the case that we do. If AI incidents do continue to a clearly unacceptable level (which I would argue they already have, but this is an appeal to the skeptics), it’s much more likely to either really be too late at that point or only be solvable with far harsher interventions. Nuclear non-proliferation has been far from perfect, but it was still clearly worth trying; there are far fewer states with nuclear weapons now (and thus a far lower chance of nuclear war) than there otherwise would have been if we had started later. I don’t think the handful of additional countries that have acquired nuclear weapons since we started would warrant saying “see, all the nuclear non-proliferation junk was a total waste of time!” And for reference, most AI regulation proposals use models trained with at least 10^25 or 10^26 FLOPs as the threshold for which they apply, which Jack Clark estimated in 2024[3] would cost roughly $10M or $100M respectively, and that both thresholds but especially the latter would only hit the top handful of labs. And yes, we can’t perfectly control the spread of unfiltered open-weights models, but it still makes sense to explore our options for limiting their risk!
(If you remain unconvinced, here’s a much more technically-detailed case. And feel free to make the opposing case in the comments of this post.)
The Advocacy Gap
There will be those who still oppose this, largely from a heuristic of hard skepticism to any government intervention, or motivated reasoning from financial incentive; the Trump Administration is still letting Nvidia sell chips to China, despite blocking this being the most obvious action to take if the concern really is beating China as they claim! It’s because of this that I’m not optimistic that the Administration’s position is going to flip before the next election, short of some extraordinarily legible warning shot or other unprecedented jump in salience.[4] But my vision here is to raise the political salience of verification proposals. AI governance researchers have been developing the ideas for a while now, and yet I almost never see them addressed to the White House and Congress where it matters most. There’s a sort of “advocacy gap” here. Mostly, the only response given to “but China” by pacing advocates that I’ve seen has been “it’s in China’s interest to pace AI development too”. This is true! But as a result, international governance proposals implicitly hinge on whether the person they’re being pitched to wants to accept a much higher (perceived) risk of “losing” to China due to covert defection when they hardly have to, and I suspect many of the politicians who do support chip export controls to China but not an AI treaty could have their minds swayed if made aware of this. I also suspect that even when verification is proposed, there’s an implicit assumption that it’s too intractable to take seriously that goes unaddressed, likely from not being aware of how centralized AI chip production and large-scale training runs are or the progress made thus far. For those who remain opposed, raising salience of this limits the reasonable responses from “but China!” or “but a deal with China is infeasible!” to “risk is so low that it’s not worth exploring our options for trying”. That’s a much harder sell now, and I expect it will become increasingly so as worse incidents happen. And let me remind you that China is already showing interest in this, and that there’s momentum among the rest of the UN!
So, insofar as you can, correct this![5] Bring it up to White House officials and members of Congress, or get more of those who can to do so. Get it addressed at international AI governance dialogues partaken in by the US (such as the G20 summit in December), and by journalists, and in media appearances, and by members of Congress and the White House, and by advocates/advisors they trust, etc.
For what it’s worth, I recommend this: https://80000hours.org/problem-profiles/loss-of-control.
As in, only restricting the most expensive training runs is probably enough to mitigate most of the problem, and there’s more to be done in making compliance cheap.
I would have liked to make this point using more up-to-date estimation methodology and checking whether the thresholds would have applied to MiMo-v2.6, the top open-weights model on Artificial Analysis at the time of writing, but much of the relevant detail is fuzzy or opaque. I’m inclined to believe 10^26 wouldn’t have applied and 10^25 maybe would have given efficiency improvements that have happened since, but please let me know if you can make a proper estimate.
Broadly speaking, it seems like they’ve been more willing to regulate so long as it only hits the biggest players and not the rest of the ecosystem. Their security framework explicitly only applies to frontier closed-weights models, and they’ve placed sudden export controls on Anthropic’s Fable in the past; maybe including stipulations for this will make an agreement more politically feasible?
I’ll have more concrete/practical output for this soon. Well, hopefully.