Summary

This is a blogpost based on a red-teaming of a Control.AI’s treaty proposal to build AI safely, A Narrow Path
The treaty likely to run into trouble in China as currently proposed, because of lack of compatibility with China’s political system
Some potential risks in China include: regulatory capture, interference or non-compliance by local governments, treaty evasion by labs, and potential national defection. These make treaty enforcement fragile without a major redesign
We need to develop treaty incentives and mechanisms that are compatible with China's governance:
- Control.AI and other treaty designs should replace the assumption of independent regulators with an emphasis on verifiability and competence, similar to China’s civil nuclear sector
- We need to build treaty structures that anticipate evasion, e.g. using strong national and/or international supervisory mechanisms
- National defection risk needs to be seen as an immediate problem. We have to consider what incentives can be used to bring nation states on-side from the very beginning
- Ultimately, we need to make sure that we get genuine buy-in from China and the CCP. We can’t assume that a powerful actor will play along just for the sake of adhering to treaty norms

Mini-Disclaimer: The ideas in this post are about political dynamics in China and the incentives that shape them. There's always a risk that coverage of these issues might be read as an adversarial or accusatory narrative against China, turning shared concerns into an us-versus-them dynamic that makes cooperation on AI governance harder. This dynamic can also spill over into negative sentiment toward Chinese people or others of East Asian descent.

I hope this post manages to avoid this, and my readers appreciate these concerns!

Introduction

Our future might well depend on whether we can get global leaders to agree to meaningful treaty commitments before we build super-intelligent systems.

This means that the success of any global governance regime will hinge on getting a treaty deal with China.

To see how plausible such a deal might be, as part of an Apart Research red-teaming policy sprint I examined Control.AI’s governance proposal A Narrow Path from a Chinese political economy angle. Sadly, I found that the treaty as currently proposed rests on unrealistic assumptions about how one-party political systems like China’s actually function.

A Narrow Path

For those who haven’t heard of it, A Narrow Path presents an ambitious vision for a treaty that aims to: a) cover the technical pathways by which misaligned superintelligence could emerge; and b) establish institutions and safeguards that create the conditions for stable global implementation.

As treaties go, it’s definitely on the bold side. “Phase 0” of this plan (subtitled Safety) proposes a 20-year moratorium on the most powerful models, designed to give humanity breathing space to develop aligned and safe AI systems. Phases 1 and 2 then outline how humanity could reach a stable and safe future state, having achieved international coordination and Safe Transformative AI.

Phase 0 of the treaty proposal, which I focus on here, aims to ensure a state of safety by covering all the bases needed to prevent the emergence of super-intelligent systems:

Banning AIs that improve other AIs
Prohibiting systems that can escape their environments
Ensuring models are bounded and predictable
Requiring strict licenses for models over a certain size

These would be enforced through national-level independent agencies, corporate oversight, and an international treaty agreement.

The logic of A Narrow Path is clear, but it mostly abstracts away from the messier issues of political enforcement and incentives. Noticing this, I decided to red-team A Narrow Path from the perspective of China’s political economy (mainly the study of incentives in politics) to test how the treaty might succeed or fail in that context.

What’s been done before

This is a relatively neglected angle. Governance researchers are recognising that incentives are important in any future AI deal, and there are some good papers on incentives for sharing the benefits of AI and prestige motivations. But I think there’s too much focus in this literature on signatories as abstract geopolitical variables, rather than focusing on specific actors and their distinctive political structures.

There are a few exceptions, such as Safer.AI's work on promising topics for US-China coordination and Governance.ai’s analysis of Beijing’s strategic priorities and analogies to institutions such as the International Atomic Energy Agency, which engage more seriously with issues like China's internal governance.

There’s also a bunch of work that engages deeply with China’s domestic AI governance, by researchers like Matt Sheehan, Jeff Ding, Concordia AI, and the ChinaTalk ecosystem, but this rarely intersects with concrete treaty design or enforcement analysis. I also haven’t seen examples of China-focused AI governance work explaining how and where things could go wrong after signing a treaty.

On the other hand, China’s political economy literature gives us many clear reasons to anticipate treaty mechanisms not working, which I look into in this post. I’ll draw on specific case studies from Chinese governance in sectors like tech regulation, financial oversight, and energy policy, to show how enforcement mechanisms can fail, and to argue that any viable treaty needs to take domestic political and institutional dynamics into account.

To make this concrete, I’ve split the argument into four key assumptions and their corresponding realities, shown in the figure below. Each assumption is discussed in its own section.

Figure 1: Risky assumptions about AI Governance in China

1. Regulatory independence is vulnerable to capture

A pillar of the Narrow Path proposal is the idea that national AI regulators should have “adequate independence from political decision-making”. These regulators would play a major role in:

Enforcing broader norms and restrictions around superintelligence (Section 1)
Containing potentially dangerous models (Section 2)
Monitoring and limiting recursive self-improvement (Section 3)
Making sure that deployment only occurs if a valid safety case has been made (Section 4)
Upholding the licensing regime (Sections 5 and 6)

The proposal also outlines an impressive set of enforcement powers for these independent national regulators. This includes:

Shutting down training runs or entire projects
Permanently revoking licences
Firing teams and prosecuting individuals
In extreme cases, fully dissolving companies and auctioning their assets.

On paper, this makes sense: independence would allow regulators to do all this stuff free from political interference. A robust independent institution with real “teeth” (enforcement powers and meaningful penalties) is clearly the ideal way of monitoring and controlling frontier AI development. But, sadly, in China’s political system, regulatory independence faces a lot of challenges.

Why wouldn’t this work in China?

Basically, Political Economy 101.^[1]

In democratic systems with separations of power, regulators can enforce a set of rules even when business or government would rather they didn’t. In systems with centralised authority, not so much.

Here, powerful state actors and the businesses they support just find it terribly inconvenient to be criticised, monitored or restricted by independent regulators. Instead, regulators can be susceptible to pressure from all sides: central government, local government and the private sector.

Figure 2: Diagram of potential regulatory challenges

How it could go wrong (a potential scenario):

China signs the treaty. Regulators begin by following treaty norms. But under pressure from various sides, they’re gradually nudged to adjust or sidestep these requirements.

What begins as a small concession on the part of the regulators: a bit of underreporting on a training run here, a sandbagged model test there, starts to accumulate.

Before long, a frontier lab is steering towards super-intelligent capabilities, with the regulator either unaware or complicit.

This isn’t hypothetical. In China, regulators can face pressures that lead them to prioritise political or industry interests over strict rule enforcement. Examples range from attempts to independently regulate the power sector in the early 2000s, to high-profile incidents like the 2020 Ant Group IPO (initial public offering). These cases suggest that nominally autonomous agencies may not always operate with full independence, especially when national priorities or shifting political conditions are at stake.

The current state of AI regulation reinforces this point. The State of AI Safety in China (2025) report describes how enforcement from the two leading bodies, the Cyberspace Administration of China (CAC) and the Ministry of Science and Technology (MOST), uses “campaign-style enforcement” which remains “selective and uneven”, following state or Party mandates.

These regulators can lack the independence, power or incentives to consistently uphold national law, let alone enforce an international treaty.

What we can do about it

This doesn’t have to be a fatal flaw. True regulatory independence is unrealistic in this context, but competence, transparency and oversight are plausible goals. Regulation in the Chinese civil nuclear sector (a success story, by most accounts) has focused on a combination of the three. The International Atomic Energy Agency conducts treaty-based inspections in the country, working alongside the China Atomic Energy Authority and the National Nuclear Safety Administration. These Chinese bodies both operate within Party-led ministries, but their arrangement has provided a system of technical verification and compliance that could offer a useful precedent for AI governance (Andrews-Speed, 2020).

I'm confident that closer attention to how this model works in China could make A Narrow Path more viable.

2. Local governance can complicate national enforcement

Even if a treaty could bring competent, aligned regulation to AI governance in China, that would only address part of the problem. A lot of the day-to-day implementation would still fall to local governments, creating a principal–agent problem, where the “agent” (local officials, in this case) is given a task, but has different incentives to the “principal” (central government), so pursues individual or local priorities instead.

In China, this dynamic often plays out in sensitive or strategic areas. The central government appoints capable regulators, but enforcement defaults to local authorities whose incentives are not always aligned. Local leaders are rewarded for GDP growth, investment, and prestige, and often benefit from informal deals with business, which makes them more willing to bypass national laws or mandates.

How it could go wrong (a potential scenario):

A frontier lab becomes a “regional champion,” closely tied to local officials who see it as a source of growth and prestige.

In return for this success, the government works with the lab to shape safety assessments, filter information to regulators, and make sure that inspections are carefully managed or delayed.

This process follows a common pattern, as people (like myself) who experienced the early years of China’s Air Pollution Prevention Plan (2013–2017) will remember. To protect their industrial sectors, local governments in smog-afflicted Hebei and Shandong under-reported emissions, disabled air quality monitors, and delayed enforcement. This only shifted when the central government introduced its own central inspection teams, which bypassed local authorities entirely.

In the AI sector, we can see how these trends might begin. Local officials see AI labs, data centres, and related infrastructure as valuable assets. Despite the current division of labour between the hubs of Beijing, Shenzhen and Shanghai, with each focusing on different sectors of the AI economy, it’s easy to imagine competition between these hubs heating up. We also see strong backing for “regional champions” in less developed areas, such as iFlytek in Anhui and cloud computing centres in Guizhou, which could be a source of central and local government incentives becoming misaligned.

This isn’t just about frontier labs and model training. Section 5 of A Narrow Path calls for cloud providers to implement shutdown mechanisms and KYC (Know-Your-Customer) reporting (citing Egan & Heim, 2023). The same section also mentions Hardware-Enabled Mechanisms (HEMs) that depend on transparency and cooperation throughout the chip supply chain. You might be able to guarantee central control of frontier labs, but the tangle of cloud infrastructure, chip manufacturing, and hardware logistics means that local governance incentives, and the associated vulnerabilities, crop up all along the supply chain.

What we can do about it

Local governance should be treated as a likely point of failure. This is something to address diplomatically and carefully. You can’t just write into the treaty text, “we don’t trust your local officials”, especially if it singles out a specific signatory. But there are ways to address this. Alongside increased international verification of treaty mechanisms at various stages, treaty design can still build in measures to limit the ability of local actors to ignore or subvert requirements.

On the international level, borrowing from IAEA protocols, a treaty can use capacity-building as a polite euphemism for observing local actors. Providing equipment, training, and standardised protocols can be framed as technical support, keeping local governments on-side, while also allowing the international body to monitor what's going on.

A treaty can also insist on centralised governance and demonstrated competence. In China, things tend to work better when the central government and Party get involved. China successfully uses centralised control to manage regional instability in sectors like the military and financial system. There is some hard data behind this claim: greater involvement by Party members has been associated with improved regulatory transparency in the scandal-beset food safety system, (Gao et al. 2023), and more credibility in corporate governance (Xie et. al, 2022).

Can we borrow from this? I'm not sure. Increased centralised control doesn’t fit well with the principles of independent regulation (e.g. OECD, 2016), and poses additional risks (see section 4).

That said, China’s governance system is highly centralised, and this is likely to remain the case. As such, involving the CCP more directly in domestic AI governance could be one pathway to improve competence and transparency.

3. Labs might exploit technical and institutional loopholes

Let's say China signs a treaty like A Narrow Path, regulators are empowered to do their job, and the local government is either aligned with safety goals or effectively monitored. Does this make the system secure?

Perhaps not.

Even sincere enforcement by aligned national regulators might be meaningless if the labs are able to find ways around the regulations; and this process could follow different rules in China that the treaty does not anticipate.

A Narrow Path proposes limiting frontier AI systems through a licensing regime based largely on compute thresholds (the amount of computing power used to train a model).

To enforce this, it gives examples of how regulation has effectively worked in the past in the West. In fields like pharmaceuticals and nuclear safety, Western companies are expected to self-monitor and set up internal systems to meet liability and regulatory demands. These might involve oversight committees, review boards and compliance training. But in China (to the extent to which these mechanisms actually exist) firms often find ways to avoid or bypass this kind of internal oversight.

*Figure 3: Technical strategies and legal loopholes to bypass regulations in Chinese labs*

How it could go wrong (a potential scenario):

A lab trains multiple sub-threshold model “shards” ^[2] on local, and grey-market GPUs. Oversight committees and review boards certify that operations comply with licensing rules. After passing a basic audit, the lab merges them into a far more capable system and deploys at scale. Regulators, relying on internal paperwork, miss the leap in capabilities.

Although this scenario could be a risk anywhere, some elements of this scenario would be disproportionately likely in China. In China’s fiercely competitive, low-margin tech industry, regulatory enforcement has historically struggled more due to weaker liability and less legal predictability (see Lee, 2018).

One recent example was the retail giant Pinduoduo, who created malware-ridden apps, which exploited Android system vulnerabilities to collect excessive user data and interfere with rival apps. One factors behind this was the company's limited internal corporate oversight bodies, which lacked the independence and technical capacity to challenge powerful firms. This suggests that, if a lab wanted to bypass regulations, there would be very little resistance from within the company.

From the angle of how a lab might avoid the technical compute governance limits included in the Narrow Path proposal, Shavit (2023) outlines various ways compute controls might be circumvented. In China, this might look like labs accessing chips that are more difficult to monitor through grey-market channels, under-monitored supply chains, and firmware tampering. Current trends in AI model architectures suggest that it will get easier to build capable systems that don’t get caught by the policy, for a couple of reasons:

Compute-based limits are becoming less reliable for judging how powerful a given model is. In China, Kimi K2 and Deepseek V3 are catching up in capabilities to much larger Western models, despite using far less training compute.
Large-scale use of inference compute, as seen in ChatGPT’s o3 and GPT-5 Thinking, can significantly enhance performance while not reaching FLOP limits during training runs (Ord, 2025).

In short, labs are strongly economically incentivised to bypass treaty rules, oversight mechanisms are insufficient, and trends suggest that it will get way easier to build smaller but highly capable models. This could make the rules laid out in the treaty increasingly easy to bypass.

What we can do about it

Some technical fixes could improve A Narrow Path’s approach to monitoring compute: Shavit (2023) suggests technical measures such as weight snapshot logging^[3], transcript verification,^[4] and chip-level security^[5] to close loopholes. But all these require more than just tweaking physical hardware. They also rely on Chinese data centre operators to enforce usage limits, local chipmakers like SMIC or Huawei to implement logging at the hardware level, and government officials to monitor compliance. This brings us back to political economy.

To stop companies across the supply chain from evading treaty regulation, effective implementation probably means that we need an unusual degree of political willingness to scrutinise domestic firms. Within China, one underexplored mechanism may be the Party committee (党委 dangwei) system—party officials currently embedded in research institutions and technology firms, who, if aligned with AI safety, could improve top-down compliance and enforcement.

4. National incentives can shift over time

If all the other issues are resolved and local governments, regulators, and frontier labs are aligned or effectively managed, one higher-level risk remains: what if the national government is no longer incentivised to follow the treaty?

When developing a mechanism for global AI cooperation, we need to recognise that parties might only follow the rules as long as it remains in their interest to do so. A Narrow Path does address the risk of state-level defection in Phase 1, but this concern arrives only after the treaty system is in place.

As China becomes more self-sufficient in hardware and foundational models (see MERICS, 2025), the incentive to “defect” (using game theory logic) from the treaty increases. China today struggles to produce its own cutting-edge hardware at scale, and is therefore more likely to comply if it wants to preserve access to international markets and maintain its reputation. On the other hand, a more self-sufficient actor (such as the US today, or China in the near future) doesn't see as much cost to defection, especially if enforcement is weak or slow.

This basic dilemma can be illustrated with a simple payoff grid:^[6]

*Table 1 – Qualitative Payoff Matrix in chip-dependent and self-sufficient countries*

How it could go wrong (a potential scenario):

As Chinese fabs develop scalable production of sub-5nm^[7] chips, and Chinese models inch towards the frontier, both the US and China start considering the possibilities that the strategic value of reaching advanced AI before their competitor outweighs the benefits of continuing to comply with the treaty.

One of the two countries begins openly supporting a civil-military fusion-style lab that skirts around treaty rules, which prompts the other country to respond in kind. The treaty collapses and both sides slide back into arms race dynamics.

To followers of geopolitics, this follows a familiar pattern. States often comply with treaties while it suits their interests, then drop out once the balance shifts:

North Korea remained in the Nuclear Non-Proliferation Treaty until it was able to develop nuclear weapons.
The U.S. left the Anti-Ballistic Missile Treaty when missile defence became a higher priority.
Russia withdrew from the Treaty on Conventional Armed Forces in Europe when it conflicted with its regional aims.

As China gains greater AI capability and hardware sovereignty, its long-term incentives to remain bound by a treaty might collapse.

I’d predict that this geopolitical impasse (or the belief in such a geopolitical impasse) will be the biggest challenge for a sufficiently ambitious AI treaty. Just as the US, as the current leader, might see little reason to sign and therefore allow China to “catch up” by hobbling its own cutting-edge abilities, China's incentives to comply with the treaty might transform when they reach self-sufficiency.

What we can do about it

A Narrow Path needs to think earlier about how to maintain incentives against treaty non-compliance. To sustainably bring China and the US on board, a treaty needs a mix of sticks and carrots. Sticks could come through strict sanctions or trade restrictions that raise the cost of staying out or defecting. Carrots could include prestige and influence, access to markets and advanced chips (with Hardware-Enabled Mechanisms to ensure safety), high-status roles in global projects like the global alignment project (GUARD) mentioned in Phase 1 of A Narrow Path, and exclusive use of safety-tested state-of-the-art models.

Early commitments to benefit sharing in the event of economically transformative AGI could also strengthen buy-in. This might look like a system where, if a nation-state or coalition develops a safe and highly profitable AGI system, all parties would commit to sharing at least some of the benefits, leaving all treaty participants far better off than before. It might sound trite, but the idea that safe AI could allow all of humanity to “win” might be the kind of shared belief needed to sustain such a treaty.

Final Takeaways

Bottom line: a global governance framework for AI needs to be coherent with political economy incentives. Understanding this can help us to:

Think more clearly about what is and isn’t realistically possible
Avoid wasted effort on proposals that can’t be enforced, and
Design solutions to foreseeable problems like those discussed in this paper.

Guaranteeing enforcement of a treaty in China won’t be easy. But the worst takeaway from this red-teaming effort would be that establishing effective international governance for AI involving China is impossible or unrealistic. First, it’s likely wrong. The US and leading AI labs may currently be a more significant roadblock to progress on AI governance. But also, pragmatically, suggesting that Chinese actors are unwilling or unable to comply could signal bad faith before negotiations begin, and risks turning into a self-fulfilling prophecy.

Without being naïve, I think we can be more pro-active and optimistic about China’s role in a global AI governance regime. Although A Narrow Path overlooks several political challenges to treaty enforcement in China, some (such as the assumption of independent oversight) could be addressed with modest revisions. And, from a purely logistical perspective, there could even be advantages to China’s centralised system, where, given national buy-in, strong national mandates and Party integration could embed AI safety commitments within frontier labs. Having a very clear veto force might make it easier to avoid a situation like OpenAI’s boardroom dramas.

But this argument brings us back to the importance of securing genuine buy-in from Chinese institutions. This might be the central high-level challenge at the heart of any serious attempt at governing AI at a global scale.

Future Work

This is an exercise in how attempts to develop an ambitious and comprehensive treaty for AI might fail because of China-specific factors. Over the next few weeks and months, I hope to build on this by looking at the following questions from a China-centric perspective:

How a range of other treaty proposals hold up when viewed through the lens of Chinese incentives and institutional structures. (e.g. A Narrow Path’s Phases 1 and 2, the AI governance strategies outlined by Convergence Analysis: Cooperative Development, Strategic Advantage, and Global Moratorium)
How China’s proposed “global AI cooperation organisation,” announced by Premier Li Qiang at the 2025 World AI Conference, might influence treaty design, and what a credible agreement could look like with China in a leading or co-leading role, drawing on its recent Global AI Governance Action Plan

I plan to look at these questions in a short series of blog posts that should to help bridge the gap between international AI treaty design and China’s approach to AI governance.

I’d love to collaborate with anyone interested, so please contact me at jack.stennett.new@gmail.com or my Linked In.

I’d like to thank Apart Research and Control.AI for hosting the hackathon that initiated this project. Apart Labs assisted in funding and supporting the research, without which this work would not have been possible. Jacob Haimes, Tolga Bilge and Dave Kasten provided insightful feedback on the sprint document and initial draft.

^{^}
The idea of regulatory capture was first theorised about in Stigler’s seminal paper (1971) and this paper does a good job of reviewing more recent evidence
^{^}
Model sharding involves dividing a large model into smaller, more manageable pieces or “shards” and distributing these shards across multiple devices or pieces of hardware, which can pose a challenge for compute governance
^{^}
Weight snapshot logging involves periodically saving model parameters during training to create a verifiable history of capability development
^{^}
Transcript verification refers to cryptographically checking detailed training logs to confirm the declared compute use matches reality
^{^}
Chip-level security means here embedding hardware features that monitor, limit, and attest to AI training activity at the processor level
^{^}
I'll avoid a formal game theory model here, as I did note earlier in the piece that this field tends to focus too much on nations as abstract geopolitical variables!
^{^}
“Sub-5 nm” is a marketing term for highly miniaturised, precise semiconductor manufacturing.