I don't know, I'm skeptical that this global standards body, regulating both China and the US, will have enough power
Agreed, and I'd add that the AI Office is unusually competent and thoughtful, which is part of why I wouldn't put all my eggs in a brand-new international body.
I also agree on the Code being a great baseline in principle. But so far the Office hasn't mounted a response proportionate to the threat. I personally believe that we are already far above acceptable risk levels, yet those companies have not been fined.
Incidents keep happening, offensive cyber capabilities are growing fast, we still can't tell whether models are aligned, and no company was planning to slow down until this week. Not all of that is the AI Office's job. But the gap between the threat picture and the institutional response is quite wide. Von der Leyen has not spoken about AI risks in a long time, and this is not a good sign.
Also, we already know that companies are not applying existing best practices; multiple trackers show this (Guidelight, FLI, SaferAI, and our unreleased tracker at CeSIA).
Which is why I land where David does: given how unbelievably messy evaluating all of this is, and how dire the situation is (let's not forget that Ajeya said: "I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.", and that there is a massive amount of inertia in the system), a simple rule like a compute-based pause looks far easier to enforce than a body adjudicating whether companies are compliant or not.
Related: AI pause: the case for ASAP.
The standard reason why anything would probably add a lot on EU Code of Practice enforcement is that IIRC, most judgements by the EU take many years to play out, often decades. IIRC, the European court is currently adjudicating many GDPR enforcement cases opened around 2020.
It's plausible AI enforcement will be different, but my current (weakly held) prior is that the European courts would take so long to deliver any judgement or do anything as to be approximately irrelevant here.
That's not at all how it works though!
The part of the AI Act we are talking about is enforced by the Commission. The Commission can directly request measures or hand out fines. This is similar to the DMA and DSA laws, which were passed a bit before the AI Act. Under the DSA and DMA laws, Apple, AliExpress, Google, Meta, Temu and X have already been fined in the last one and a half years for sums ranging between 120 million and 890 million. TikTok, AliExpress and X (and maybe others, I'm mostly going by a quick Claude summary here) have also agreed to various changes in response to the Commission's DMA and DSA requests.
Companies can and often do attack fining decisions in court, and the adjudication can indeed take a while (though I think usually not six years). But the fine needs to be paid or at least locked down by the company at the time of fining. Later, if the Court decides on the company's side, the Commission needs to pay back the money with a relatively high interest rate. But this means that even if court cases take a long time, companies still have an incentive to comply with the law, since the fine is paid immediately.
I would appreciate if you passed on this correction to the people among whom this is "the standard reason" the AI Act can't work.
(Note that I'm not a lawyer, I'm not working directly on this question, I'm definitely not speaking for my employer, and I'm just writing things based on public information I and Claude found online.)
Under 5% of AI compute is hosted in the EU (as opposed to ~90% in US+China), only ~one of the frontier companies are headquartered in the EU, and the act is designed in a way that the United States (at the time the Biden administration) found very dissatisfactory, as it was seen as promoting EU interests over American interests. China was essentially not involved.
The act doesn't even require the reporting of the Hugging Face incident as it's currently designed, since it exempts pre-market research and testing. It also doesn't require reporting for models used for military purposes. They are essentially limited to imposing fines, which historically have maxed out at a few billion dollars, which is not a compelling enough number to greatly alter how the AI companies behave.
If the goal is to have a regulatory framework that the frontier AI companies actually comply with, it can't be top-down imposed by an unelected group of people in a political organization that has no jurisdiction over where the companies are based, pay taxes, and train their models.
Even if the US Code of Practice was word-for-word the exact same as the EU version, if it was willingly entered into by the US AI companies, and was under US jurisdiction, it would have an order of magnitude more credibility and enforcement power.
"The act doesn't even require the reporting of the Hugging Face incident as it's currently designed"
=> This sounds overconfident. FYI: from the GPAI Guidelines, paragraph 100: "The AI Office considers that this obligation [to report relevant information about ‘serious incidents’ under Article 55(1), point (c), AI Act] covers serious cybersecurity breaches related to the model or its physical infrastructure, including the (self-)exfiltration of model parameters and cyberattacks"
"If the goal is to have a regulatory framework that the frontier AI companies actually comply with, it can't be top-down imposed by an unelected group of people in a political organization that has no jurisdiction over where the companies are based, pay taxes, and train their models. "
=> Frontier AI labs signed the Code voluntarily though.
If the goal is to have a regulatory framework that the frontier AI companies actually comply with, it can't be top-down imposed by an unelected group of people in a political organization that has no jurisdiction over where the companies are based, pay taxes, and train their models.
Why is the "top-down imposed by an unelected group of people" part relevant? Isn't most technical regulation imposed top-down by unelected experts and bureaucrats?
In the US, essentially all detailed technical regulation comes from agencies staffed by career civil servants and political appointees: EPA emissions standards, FDA drug approvals, FCC spectrum rules, NHTSA vehicle safety standards, OSHA workplace rules, FAA airworthiness certification. Congress passes enabling statutes with broad mandates and the agencies write the actual binding rules through notice-and-comment rulemaking. Same in every developed country.
I don't think I should go into details (and I'm not a lawyer), but I'm not sure people should put too much weight into solhando's interpretation of the AI Act wrt things like the Hugging Face event.
And yes, having a global body with more explicit support from the governments of US and China would be great, though my guess is that it would be less than an order of magnitude more powerful, because of the reasons I will describe below.
it can't be top-down imposed by an unelected group of people
The AI Act was passed by the European Parliament, which is a directly elected legislative body. The AI Act contains Article 55, requiring providers of the most advanced models to conduct state-of-the-art model evaluations, assess and mitigate systemic risks, keep track of and report information about serious incidents, and ensure adequate cybersecurity of their models and their physical instrastucture.
The AI Act also foresees that a Code of Practice will be written (Art 56). The authors of the Code are indeed not directly elected, but have been appointed by the Commission, which is accountable to the Parliament.
As Kaj Sotala also says, I think this is a very normal and legitimate way for a technical regulation to work.
But yes, I agree that if you want to take much more drastic actions (e.g. if it turns out that at some level of capabilities everyone needs to pause for many years to get actually solid safety cases), then you probably need more legitimacy and more direct buy-in from world leaders. But this is my whole point. The global standards body Dario and Demis envision is also explicitly framed as a technical body, looking at evals and safety arguments and so on. The rulebooks and the enforcement of that global standards body won't be any more of a "direct democracy" than what the EU Commission is doing - technical regulations just don't work like that anywhere. If you want more drastic actions with direct and full-on buy-in from world leaders, you have a better chance with a pause than with a technical regulation.
Re last paragraph: But would it be substantially more compelling for Chinese companies?
Ie, I totally believe that the US can build much stronger regulations for its own companies. But David’s post was about international governance regimes, not domestic ones.
I don't know, I'm skeptical that this global standards body, regulating both China and the US, will have enough power to fully stop development when a company's evaluations or internal cybersecurity is deemed to be not quite up to their standards.
What about when they’re egregiously not up to the standards, or equivalently, not up to pretty loose standards?
Ie., a country’s scientists comes back to their home government and says: This agreement is being clearly violated. I think we’d have a scientific consensus saying so in our own country if only we had enough time to establish that. The other country is putting us in serious danger. Please ask them to stop or to put in more effort into showing we’re wrong or else.
Do you still think a response greater than 3% of the company’s revenue is unlikely? Or that it’s just not very valuable to be able to act on this kind of thing because for every regime you can realistically set up, you’ll be able to do _whatever you want_ with plausible deniability because there’s so many places you can fudge numbers and pick favorable evals? (Such that there’s no value add from being able to rule out clear stuff.)
Also, I think a plausible reaction to even mild violations could be withdrawing from the agreement, which can be quite bad if both countries benefitted from the agreement being in place.
I think there is some value. I think it's great that the Code and the AI Office exists. Similar thing with more buy-in from the US and China could have some additional value. But mounting a really big response to things quickly is not an easy business. I would be more positive if people's ambitious governance proposals included a line saying that "our proposal is different from the Code because the the regulatory body/bodies in question will have a call with the US President and the Chinese Premier every Monday". AI 2040 has that, which makes it more credible that their proposal makes a situation an order of magnitude better compared to what we already have. Still, even then, I'm worried that things will move too fast by default, and there will be too much distraction and noise that will drown out the most important signals. I explained that worry in response to AI 2040 in this comment. https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=j4QbYtfDYLcNnTroN
(As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.)
In a recent essay, Dario Amodei advocates global coordination to pace the frontier of AI development. Level 2, which he already considers to be ambitious, reads as:
Elsewhere, Demis Hassabis envisions a FINRA-style self-regulatory standards body, which "would provide a strong starting point for creating shared international standards on Frontier AI":
The good news is that all of this already exists. The European AI Act has been in effect since August 2025, and the AI Office got its enforcement powers this August. The AI Act has many parts, but the most relevant to frontier AI safety is Article 55, requiring providers of the most advanced AI models to perform state-of-the-art model evaluations, assess and mitigate systemic risks, keep track of and report serious incidents and ensure adequate cybersecurity of their models.
The rules apply to all companies placing models on the European market - i.e. all of them, whether they are American, Chinese, or from other nations.
The Code of Practice details out an implementation of these rules. It specifies CBRN, Cyber offense, Loss of Control and Harmful Manipulation as systemic risks that the companies always need to address. It requires companies having a Safety and Security Framework; writing detailed Model Reports explaining the evaluations they have done and the conclusions they have drawn from them; having good security of their models and the physical infrastructure of the models, including guarding against "(self-)exfiltration or sabotage carried out by models"; and a number of other things. All of this is described in great detail in the Code.
I recommend reading Miles Kodama's blog post on the AI Futures Project blog: The world's first frontier AI regulation is surprisingly thoughtful.
Or you could read the Code of Practice itself, it's truly a sight to behold. I will weep tears of joy if the international or US standards envisioned by Dario and Demis are as strong but well-targeted as the Code.
(By the way, the AI Office is hiring. If you are an EU citizen, consider applying - I think the AI Office is a great place for steering things in a better direction. You can also DM me if you have questions about the AIO.)
Still, one might be tempted to think that despite the existence of this great global regulatory framework, the AI safety situation is not 100% solved.
Therefore, I think it's important for people proposing global (or even national) regulatory frameworks to answer the question of what their proposal adds over the Code of Practice and the AI Office already existing.
The European Union can already fine companies up to 3% of their global revenue for violating the AI Act, and in some cases, can restrict deployment on the EU market (about 25% of the global market). Perhaps people imagine the new standards bodies to have more power than that, being able to fully block development and deployment for a long time if companies don't comply? I don't know, I'm skeptical that this global standards body, regulating both China and the US, will have enough power to fully stop development when a company's evaluations or internal cybersecurity is deemed to be not quite up to their standards.
Overall, it's hard for me to see how this global standards body will accomplish more than the already existing voluntary efforts from the companies plus the global enforcement power of the AI Office. Evaluations, cybersecurity, alignment and safety cases are all such technical and finicky topics. I don't see how the very technical standards body assessing these things will have the power and legitimacy to take much more drastic actions than the equivalent of fining companies for 3% of their revenue.
Maybe you can accomplish a bit more if these standards bodies have direct lines to the heads of states, who consider global AI regulation one of their highest priorities, as envisioned in AI 2040: Plan A. Still, I have some severe skepticism of how well that would work in practice, as I explain in this comment.
Dario Amodei lists a Level 4 of global coordination:
He is very skeptical that this is possible, though to his credit, he thinks we should still try.
I agree that a full-on international pause is probably harder to achieve than setting up any kind of global standards body. But I don't think it's harder than setting up a global standards body that actually has enough teeth to add significant additional value over the already existing Code of Practice. Pausing is a relatively simple concept, and violations are relatively easy to notice, especially if the pause is largely based on limiting the amount of compute. It feels so much easier to call up the full support of world leaders to take serious steps against a pause violation than against things like "the global standards body says that a company's evaluations are not well-elicited enough".
So I urge everyone that if we get a rare shot at getting international cooperation on AI, we aim higher than setting up something that already exists through the Code of Practice.