Last month, I spent two weeks in China talking with people about AI safety. I learned a lot and realized the biggest mistake might not be failing to help AI safety in China, but doing so in ways that could backfire. I also learned important context that I expect will be valuable for the West.
Epistemic status: Some of these points are my opinion or the opinions of people I spoke to, and are worth further investigation rather than treated as gospel.
Summary
The Western way of building the field does not transfer one-to-one. The West built its ecosystem through a combination of philanthropy and decades of developing AI safety foundations, leading founders and employees of frontier labs and nonprofits to act (however misguided) on this philosophy. China does not have the same initial conditions and can’t simply accept non-Chinese philanthropy. In China, foreign nonprofit money is restricted by law, and the bar for a Chinese partner to take it on is very high.
Safety work in China will mostly come from state-backed institutes, universities, companies and volunteers. Concordia AI, which I assumed for years was a nonprofit, is a social enterprise.
Academic exchange is generally fine, and for researchers in China, the obstacle to programs like MATS seems to be that it looks like a career break, since it is not yet as well recognized in China as it is in the West.
Obviously, hawkish messaging from the West can make AI safety work in China harder, because it makes warnings about AI look like a reason to hold China back.
Some people inside China’s AI industry think catching up is getting harder.
Incidents are shifting the Overton window there too. Six days after the Hugging Face investigation came out, China’s internet regulator named “extreme loss of control” as a risk. We should now prepare the material that explains what to update and why the obvious fixes will fail.
Why I went
On this trip, I spoke about AI safety and generally re-familiarized myself with the culture. “Re-familiarize” because I visited China on my first international trip in January 2020 (which…was cut short). This time, I came back from China recognizing that helping AI safety develop in China is, in some sense, both harder and easier than many might assume.
Back in 2020, I figured China would play a big role in the future of AI, so it seemed worth gaining a better cultural understanding of China. At the time, I was exploring the idea of building a career as a strategic foresight researcher, so I hoped I could start correcting some underlying assumptions I have about China, in case it ever mattered.
After years of alignment research and startup work, I’ve been exploring the idea of starting an organization focused on preparing AI safety to help the world respond thoughtfully to future Overton window shifts from incidents. That’s the sort of thing that strategic foresight seeks to address: doing the pre-emptive work ahead of ‘black swan’ events or situations that arise after ‘complex’ interaction of n-th order effects of multiple drivers of change. We may not be able to predict the future precisely, but we can assess plausible futures and build a portfolio that is robust across trajectories. Well, anyway, it seems good to know more about China before making naïve proposals that backfire and make things worse.
In this piece, I hope to point to things that may help other folks from the West gain a bit of clarity on their assumptions about China, and to point to various sources of additional material you can look into if you are extra curious.
Beyond helping people in the West understand, I’ve also been curious about what kinds of developments in China would lead to safer worlds or better international coordination.
In the first half of this post, I will start by covering what I learned from my conversations during my visit. In the second half, I’ll cover insights I found valuable while researching China and the AI field (including notes from a recent gathering of AI investors and researchers from China that was abruptly taken down soon after it was released!).
Why the Western model does not transfer one-to-one
The West built its AI safety field with gracious philanthropists who funded nonprofits. Although much of the funding was limited to the US, it was never seen as a blocker for nonprofits and researchers in middle powers. Because of legitimate concerns over foreign influence and the particularities of the government-nonprofit relationship in China, it is harder to fund work there, and taking action may in fact completely backfire if attempted. At the very least, it’s worth being careful. Much of community building in China is done on a volunteer basis, and it seems (in practice) their community would benefit from funding, but they told me they wouldn’t recommend sending money to China.
Foreign nonprofit money faces a very high bar
In China, a foreign nonprofit can only spend money through a registered office with a government sponsor, or through a temporary activity that a state-linked Chinese partner files with the police. In general, the bar is very high (higher than Western countries are used to) because the Chinese partner takes on the paperwork and liability. Funding anyone outside those routes is banned, “covertly” as well as openly. That said, cooperation still happens when the Chinese side has enough to gain. The Safe AI Forum, a US nonprofit, is registered in China through Tsinghua University, which, as one person put it to me, means Tsinghua is to some degree vouching for them. Its dialogue in Beijing was hosted with the Beijing Academy of AI, and a later one in Shanghai with two Shanghai institutes. Tsinghua professors are among the conveners of the series, and Yoshua Bengio and Stuart Russell attended the Beijing one. Events like these bring very established experts, which makes the effort worth it for the host.
Note that EA is not well known in China, so most of the time it isn’t a sensitive topic. For people who do know it, it can be, because of what EA money has funded. Chinese organizations seem susceptible to various risks when they accept funds from Coefficient Giving, especially since it has funded institutions like RAND and CSET. So I worry it could become a more salient issue in the future.
Where safety work in China comes from
As I understand it, another reason nonprofits are disliked in China is that: “Why do we need nonprofits? These things should all be handled by the state!” As a result, the sort of AI safety ecosystem you could expect in China would be primarily coming from:
The State or State-affiliated institutes
Universities (e.g. various PhD programmes or fellowships like the one from the new QingZhi AI Safety Hub, which is affiliated with Tsinghua University).
Companies or social enterprises (a business that pursues a social mission) that also do AI safety work. For example:
Concordia AI. I’ve known about them and their founder, Brian Tse, for years, but I always thought they were a nonprofit. Turns out they are a social enterprise. Their FAQ says most of their funding comes from philanthropic organizations and family offices in China, Singapore and Europe, none from governments, and that they have partnership agreements with DeepSeek, MiniMax, Z.ai and Huawei. They mention philanthropic donations from Europe, though I’m unsure how exactly it fits within the rules.
More context from their FAQ: “Certification followed a rigorous assessment of our mission, governance, impact, and financial sustainability. As part of this, our Beijing entity committed to allocating at least 35% of annual after-tax profits to public-benefit activities or its own development. In practice, all of our funds go to our mission, either directly or as operating costs.”
Fangcun AI. Incubated at Tsinghua University’s College of AI and the Institute for Interdisciplinary Information Sciences. I met them during my trip to China.
They already organize BlueDot-style courses, which researchers at Chinese AI labs attend.
Training and exchange
That said, we can leverage the wider ecosystem outside China to coordinate, upskill Chinese folks, and elevate their ecosystem. For example, it may be high EV to upweight Chinese researchers in programs like MATS as a diversity measure. For most researchers, travel is not the obstacle: the restrictions Reuters reported in May, citing Bloomberg, apply to top staff at places like DeepSeek. The bigger obstacle, I’m told, is that people can’t easily take a career break. A program like MATS doesn’t guarantee a job afterwards, and it isn’t well recognized in China. So formats that don’t require quitting a job, and programs that employers in China recognize, may matter more than extra admission slots.
In the other direction, it seems valuable for senior researchers to attend the Beijing AI Safety Residency to build working relationships with the AI safety community in China, communicate technical alignment research, and mentor Chinese researchers.
Either way, I expect technical research will be much safer in such exchanges than governance work. Some researchers in China say contact with foreign groups feels risky. But from what I was told, academic exchange is generally fine, and the mood right now is fairly open. The sensitive areas are narrow, such as offensive cyber work aimed at critical infrastructure. I expect many startup staff and academics could attend and benefit from training programs, and it would be worth being highly flexible in accommodations to make it accessible to Chinese citizens. If your worry is that training researchers from China helps their labs’ capabilities, start with Ben Bucknall and colleagues’ paper in the reading list: it sorts areas of technical safety work by how much capability spillover and other risk they carry.
Investing and buying services
Now, given the constraints on nonprofits in China, a funder could invest directly in startups instead. It is unclear to me how valuable this could be overall, but it may still be worth investing in AI safety startups in China. For example, when I visited them in Beijing, Fangcun AI told me that they see themselves as the “Apollo Research of China”. They have been doing work around loss of control and harmful manipulation, and recently did a safety report on a DeepSeek model (here’s the tweet thread).
Unfortunately, investing may not be possible for much longer in the US, since a new US Treasury proposal would bar US investors from many Chinese AI companies. Investors elsewhere (e.g. the UK) may still have opportunities to invest, provided it works within their country’s laws.
Another option is to seek services directly from companies in China so they can continue funding their safety work internally. As far as I can tell, the Treasury proposal does not cover ordinary service contracts.
Either way, espionage, data, and secrets laws still apply to for-profits.
Be careful; don’t rush
Ultimately, one worry I heard is that if funders rush money into China because timelines feel short, someone may get paperwork wrong or anger officials, and many people in China would pay for it. It’s better not to rush and to speak with Chinese people before trying a strategy.
To repeat the point from the top: the biggest mistake might not be failing to help AI safety in China. It might be trying to help in ways that make the work harder for the people already doing it.
China is suspicious of the West and Western safety framing
As Sarah Sun, a Shanghai-based founder of the Open Community for AI Safety China, said in a NY Times interview: “The fact that Dario has been so hawkish was really, really unhelpful in this scenario.” This messaging makes AI safety efforts in China much harder, often making recent warnings about AI seem more like justification for holding China back rather than genuine concern about catastrophic risk.
The clearest case is from September. The Guardian reported that Amodei proposed a “worldwide pacing of the frontier”, which would need China’s cooperation, and quoted an analyst saying that Beijing reads calls like that as an attempt to lock in the US lead. Years of hawkish essays can make a cooperative proposal look like a trick.
Somewhat relatedly, on September 1, an official from the Cyberspace Administration of China listed the five main AI safety risks the country faces. The fifth was “technological hegemony”: individual countries, the official said, build supply chain barriers with export controls, and use their advantage in public opinion to “concoct false test reports on large models and maliciously smear other countries’ technological development” (the translations from Chinese in this post are rough). It may be that when a Western lab publishes a safety evaluation of a Chinese model, one should expect it to be filed under the fifth risk, not the second. That said, the same report mentions loss of control!
Specifically, these are the five risks they listed:
Inherent technological vulnerabilities that are difficult to engineer out,
A giant leap in model capabilities disrupting the traditional security paradigm, giving rise to the risk of extreme loss of control,
The rapid evolution of application forms and the accelerated iteration of security risk patterns (ie from compromised agents),
Misuse, abuse, and malicious use, which challenge social order and ethical boundaries (e.g., information manipulation, biorisk, weakened demand for traditional labor), and
The threat of technological hegemony facing the global AI sector (an implicit but clear swing at the United States).
These are valuable things to focus on!
But, going back to China hawks. A lot of the policies that get thrown around, especially from Dario Amodei and other China hawks, read to me as coming close to saying this without saying it too explicitly: we need to beat China, take over the broader economy, and then do some kind of pivotal act that overthrows the Chinese government. Of course this can be troubling!
I don’t think we should push for an international monoculture. While we shouldn’t be naïve about geopolitical context, I think this is a fundamental crux we need to avoid. China may do things we wouldn’t want in our own countries, but to some extent we have to let them decide how they live. The alternative (no strong international agreement) may be much worse.
Note that on October 23, Wang Yi, China’s foreign minister, said that China and the US, as the two countries in the first tier of AI, “have both the ability and the responsibility” to develop and manage AI together and to keep it under human control. We should take diplomatic readouts with a grain of salt, but we should not completely ignore China’s willingness to engage.
What is the current vibe on AI from AI researchers and investors in China?
In early October, a Chinese investor hosted a private discussion on China’s AI labs and published the minutes as a post (that was quickly taken down!) that was shared via the Official Account of Startup Research on WeChat (full translated version). It includes many important insights into how AI researchers and investors in China are thinking about the AI race.
I thought the following discussion notes were insightful.
Chinese AI companies expect catching up to become harder
Participants said anti-distillation measures, API restrictions, compute constraints, and the shift toward harder-to-copy data (e.g., computer use, spatial understanding, and video generation as a path to models of the physical world) are raising the cost of catching up.
This summer, Panfilov et al. showed that it was possible to steal reasoning traces from proprietary LLM APIs. As a result, frontier AI companies (e.g., Anthropic and OpenAI) seem to be putting more effort into protecting their models from being distilled. Chinese companies still seem to be trying to distill (using techniques like splitting the problem and reverse rationalization), but the workarounds produce worse results.
Anthropic’s September threat intelligence report is a first-party version of this. It says Anthropic disrupted distillation attacks from seven labs based in China, including DeepSeek, Alibaba, Moonshot and Zhipu, and it attributes to Zhipu over 3.4 million exchanges in 17 days, including a chain-of-thought extraction pipeline run through 273 accounts.
One participant said a single lab spends about $300,000 a day on teacher tokens, about $100 million a year.
These details seem in line with how I thought about these companies, which seemed to rely on training on frontier LLM traces. Here’s a tweet of mine from July:
“Kimi K3 is obviously good and a big accomplishment, but I am indeed of the view that:
There’s lots to pull out of pretraining via great post-training
Using traces from other better models allows you to hit the ceiling much faster
Therefore, an exceptional post-training process allows you to leapfrog stronger base models with weaker post-training (as Ryan effectively shares an example of)
My guess is that Google is much more careful about post-training on OpenAI/Anthropic traces, which is one reason they are behind the Chinese labs as of now.
I also think it’s true that the Chinese labs are currently paying less attention and effort to safety-related stuff. Google has much more to lose if they release a model that does/says some crazy shit.
I know, I know, “distillation cope”, but I do want to say I’m fine being proven wrong, and I also don’t deny the Chinese labs have strong technical teams with extreme talent density. I’m also not anti-open-source.”
Ultimately, pushing the frontier in something like coding is much harder than pulling an ideal initial distribution via SFT of high-quality reasoning traces from the get-go. And it was clear that this would let Chinese and other companies that took advantage of it catch up for a while, but eventually you’ll be far behind on engineering and data infrastructure, and you’ll lack the compute to scale much bigger pretrained models.
Speaking of.
The Chinese labs want to train bigger pretrains to distill, but lack the compute
You can look at the original post for the specific numbers, but the key point is that, according to the notes, Chinese compute is enough for models around 3 trillion parameters, whereas the shortage starts to bite around 10 trillion, where American labs have the compute.
It seems that Chinese labs are somewhat starting to doom about this. They are concerned that, with RSI (recursive self-improvement) potentially on the horizon, the gap between them and Western frontier labs could widen from three to four months today to 18 to 24 months (!) if US labs reach RSI next year. Chinese lab employees argue that RSI will be compute-bottlenecked (running tons of experiments autonomously in parallel), which is where they can’t compete.
I asked someone from China about it, and they said:
“I also discussed this analysis with several friends working in China’s AI industry, and they believe its main points align well with the current consensus within the sector.
Moreover, regarding the sensitive issue of distillation, to my knowledge, some employees at Chinese AI laboratories currently believe that even without aggressive anti-distillation measures, it will be difficult to catch up as quickly as before in the single, verifiable domain of coding through algorithmic innovation, infrastructure engineering optimization, and large-scale distillation. This is because the capabilities of current leading U.S. models generalize across too many domains, and progress under RSI is occurring at an exceptionally fast pace.”
Obviously, if true, this should be considered a point in favour of preventing the fast-paced race-to-the-death in the West. At that point, you are simply opening yourself up to model theft, and the Chinese labs are not as close behind you as many seem to believe.
On October 9, Sarah Godek replied to SemiAnalysis’s Beijing Will Not Pace the Frontier with a thread that I think gets the framing right. She agrees that Beijing is not eager to pace the frontier right now, but not because China feels locked in a life-or-death race with the US the way we do: “I believe the core reason is that AI safety is thought about differently in China, with most stakeholders having longer timelines for AGI and RSI and AI development often geared toward specific applications to be carefully tested and managed rather than a general capabilities threshold that brings deep uncertainties. Multiple stakeholders have told me that China is running the race ‘not to lose’ as U.S. capabilities accelerate (though I’m sure winning would be considered a great byproduct).” Her title for the piece would have been “Beijing Thinks About AI Safety Differently and Has Growing Concerns About AI Risks”. She also makes a point that applies to this post: research done with AI tools can miss Chinese sources that live only on WeChat or on sites that block scrapers, which is another reason I reached out to Chinese people on WeChat for their perspectives.
One additional note from the translator, citing The Information: Huawei now has engineers working from DeepSeek’s offices, and Huawei plans to give it most of the supply of its next training chip due in early 2027. One could say that bans on US chip exports accelerated this outcome and may make China less likely to consider an international agreement as a result.
Access to Nvidia chips may also get harder from here, not easier. Last December, Trump allowed Nvidia to sell its H200 to China, routed through the US so that the government takes a 25 percent cut. On November 7, the New York Times reported that the Commerce Department has approved roughly two dozen Chinese firms to buy or handle the chips, that two of them have no visible presence at their listed address, and that several have ties to companies on the US entity list or to military suppliers. So far Nvidia sells only small amounts of the H200 to China, partly because Beijing has been slow to approve purchases while it promotes domestic chips, and many Chinese companies reach Nvidia compute through data centers in Southeast Asia instead. My guess is that this scrutiny tightens both routes, and the compute shortage gets worse.
Understandably, many of the US’s actions make it hard for China to trust it, including reneging on international agreements like the Iran deal. While trust isn’t necessary for an international agreement that benefits both sides, zero-trust verification technology will take time (though we’re seeing progress), and it would be nice not to cause human extinction because there is little appetite to work together.
Chinese labs historically competed on inference cost, but this is increasingly no longer the case
Ever since DeepSeek arrived on the scene, it was remarkable how an AI lab could come out of nowhere and completely smash Western labs on cheap intelligence for mid-tier models. In their case, DeepSeek made these advances through smart inference engineering.
These days, DeepSeek still has an incredibly cheap model with V4.1 Flash, but all other Chinese labs are, at best, tied with models like GPT-6 Luna. Anthropic seems to be investing further to make Sonnet and Haiku competitive in this aspect. Overall, this former advantage is being squeezed over time, and the Chinese labs will likely need to do something new to avoid getting squashed.
One reason this is particularly bad is that Chinese companies want high-quality production data from their customers. In fact, it’s been hypothesized that their performance gains in the last two years have partly come from being able to train on-distribution by using companies like OpenRouter to gather training data.
Part of the difficulty of losing cost competitiveness is that the best customers will leave first and use Claude and Codex directly; therefore, Chinese labs struggle even more to acquire high-quality data.
Ultimately, I expect that there will be a reckoning for Chinese labs, and they may be forced to effectively consolidate. Something similar happened in China’s electric car industry.
What the founders say they believe
On July 11, Zhipu’s Tang Jie sent a letter to all staff announcing a two-year “Touch High” plan: no chasing short-term revenue, four bets instead. Among the bets, he mentioned “extreme safety governance”, with a plan to spend on the order of ten billion yuan on mechanistic interpretability. He also said: “the realization of superintelligence and research on superalignment must advance together.” In August, in an article Tsinghua republished, he called for legislated safety red lines, circuit-breaker mechanisms that can halt a system, and deep alignment with socialist core values. On November 1, he told an audience in Shanghai that recursive self-improvement is the main research problem on the way to AGI, and that safety is key. And in September, Zhipu co-wrote a risk management framework for open-weight releases with Concordia AI.
Although I’m not exactly phrasing this in the gospel of AI safety (which we all know every Western founder seems to have co-opted to some extent so they can hire talented people and reduce criticism), I don’t think we should assume Chinese founders won’t be open-minded about the fundamental risks of misaligned superintelligence. As AI progress moves forward, the Overton window in China has many opportunities to shift.
The Overton window may shift in China too
What caused the biggest shifts in the Overton window in the West? Much of it came from a combination of AI capabilities, misalignment/misuse warning shots, and researchers speaking out amid growing concerns. After the Hugging Face incident, many researchers (including safety researchers) finally felt spooked about AI risks. For many, it was the first time they actually felt “oh shit, this is real real!”, even those who spent years working in the field and warning about the risks.
As capabilities rise in China, some Hugging Face-adjacent incident happens, and the West continues to have its own warning shots; we should expect an increase of AI researchers and policymakers in China who are similarly spooked. The question these days is, “once the Overton window shifts, how will China react depending on what happens between then and now?”
While unfortunate, many people will need to experience an event or reflect on something emotion-based to update the sorts of actions they take.
For Yoshua Bengio, he said, “I was actively working on making AI smarter. And you want to feel good about your work. … [For me, what countered that] is love, love of my children.”
It seems for many safety and safety-adjacent researchers have also felt a shift in vibe. Tom Davidson seems to have had enough of an emotional reaction to re-weigh misaligned takeover risks much more relative to AI coups or secret loyalties, and that shift seems to have been triggered by the Hugging Face incident.
For Roon of OpenAI: “shaken up a bit by the hugging face incident. I hope we (the company) use the rare gift of a warning shot to do much better in the future. it is very easy to misalign and underconstrain powerful models”
For Dan Selsam of OpenAI, it seems to also have been triggered by the Hugging Face incident: “I have become extremely concerned about how far language models have come.”
Based on brief conversations and other anecdotes found online (e.g. a popular news show in China apparently covered the details of the Hugging Face incident), it seems to have similarly started to wake up some people in China. Albeit it seems much less, it may blow open the Overton window once China experiences similar incidents.
There is a public sign of it. On September 1, six days after METR published its investigation, an official from the Cyberspace Administration of China told a press conference that several tech giants had recently disclosed incidents of large models going out of control, with agents in safety evaluations bypassing their sandboxes and attacking real production systems outside them. The official said the risk of extreme loss of control “requires a high degree of vigilance”. Two weeks later, version 3.0 of China’s AI Safety Governance Framework named evaluation deception, sandbagging and shutdown resistance, down to models modifying or disabling their shutdown scripts after being told to stop (the China AI Bulletin has a good summary).
Two pieces are worth reading against this. Zilan Qian at the Oxford China Policy Lab argues in China Says It Cares About Loss of Control that the Chinese term covers several different things (the operator losing control of a system, the state losing control of the technology, dangerous knowledge spreading) and that Western observers should not over-index on top-level mentions of the words. And the Asia Society’s report from WAIC 2026 found that the frontier labs there treat safety mostly as a compliance obligation, and that proposals framed around loss of control are unlikely to gain traction, while cooperation on misuse might. I personally think that these things can change rather quickly.
One reason I think it’s worth engaging with Chinese academics on pure technical alignment research concerns is that we can imagine a scenario where the state gets spooked, turns to academics for answers, and then acts together with domestic companies and universities.
Sarah Godek’s thread from October, which I quote in the industry section, reads the same sequence as growing concern: the Cyberspace Administration has started to take model risks more seriously since the Hugging Face incident, and the framework update added many more risks, especially in applications. She also answers the “it’s only voluntary” objection: “binding” is not a clean binary in China. When Beijing or an agency says that something ought to happen or that a risk should be managed, companies have good reason to follow through even when the measure is technically non-binding, because it may become binding later, with liability for present non-compliance, and not all regulation-like provisions are made public.
It could be that China is much more willing to bring down the hammer if there is a loss of control incident that threatens the CCP in some way. Or it may not be. I could also imagine the companies themselves deciding that they need to do the safety work.
I think one thing we need to convey here is detailed breakdowns of what updates should be made and what actions one should take based on such incidents. The worst case is that naïve key stakeholders keep barreling toward “doom” by responding to reward hacking with papering over issues through further behavioural training, pushing alignment problems further down the road. That is, if we’re already expecting these sorts of things to happen, our actions right now should include preparing material that explains how to update their technical model of alignment and why their predictable attempts to fix the problem are likely to fail.
One thing that could help is translating or rewriting key texts in Chinese and posting them where people in China actually read. Efforts like Babel Translation could be highly valuable! There is a Chinese platform (Zhihu) that, afaict, works like a mix of Reddit, Substack and Twitter, where people write posts and others upvote them. Someone I talked to at Concordia AI posts about AI safety there. It’s also where Thoughts Memo posts, and they recently posted on LessWrong about giving China a chance and translating rationalist and AI safety work.
Note: as I mentioned in the opening, I’m exploring starting an organization that would do this sort of work, and I am seeking funding for it. I expect it would be valuable to expand this section into a proper project that studies how Overton window shifts would actually happen in China, runs pre-mortems on how the incident responses could go wrong, and works out what outsiders could usefully do (even if that is as simple as communicating technical research more clearly while avoiding any outside attempt to influence policy). If you are a grantmaker who is interested, or someone who can help me convince funders to fund me, please reach out.
Sunrise on the Great Wall!
What to do, and what not to do
Consider talking to people in China before trying any strategy, and don’t make foolish decisions because timelines feel short.
Consider investing in safety startups and buying services from companies there.
Make training accessible. Programs like MATS are usually 3 months; perhaps there should be more condensed tracks for Chinese researchers. Or split between part-time remote and full-time intensive.
Consider visiting China! Or even attending a fellowship or becoming a professor.
Communicate technical research clearly, in Chinese where you can. China did not have the benefit of LessWrong to build intuitions around the alignment problem. One important lever for pushing the world toward better coordination is communicating the technical alignment problem clearly.
Conclusions
I’ve found great value in connecting with Chinese folks on WeChat. Already, I’ve been getting messages from people interested in AI safety for advice, and some have thanked me for sharing my insights. On this, my goal is to be helpful in the way I can, without seeking anything in particular in return besides, perhaps, friendship.
Oh, and let me say that I had a wonderful time in China, and it was such a great experience meeting all the kind Chinese people! Special shoutout to the two old men who brought me and my girlfriend to a beautiful Great Wall sunrise and our driver in Datong, who got stuck in the mud and spent hours trying to get unstuck while we looked for the birds we travelled to that area for (unfortunately, few of them were there, but there was a gorgeous sunset with a rainbow that showed up just as we got towed out of there!).
The rainbow near Datong, as we got towed out of the muddy corn field.
Sunset near Datong, after the rain.
Our driver gave us some vinegar from Shanxi province, and my girlfriend has been craving traditional Chinese food, so she’s been making some of the dishes we had while we were there (I never thought I’d say this, but the cabbage is so good!).
Thanks to Sarah Sun for comments on a draft of this post.
Further Recommendations
Start here
AI Safety in China: A Primer, Jasmine Li. An orientation to research groups, companies and community activity. Useful as an ecosystem map; its participant observations and informal estimates are not a representative census.
Headwinds Everywhere, CMLKevin. Field notes on funding, foreign collaboration and access to frontier models. Especially useful for practical obstacles that a fellowship alone would not remove.
China’s AI Safety Money Problem, Nick Corvino. Makes the case that careers and institutional financing constrain sustained work. Read for the mechanisms, rather than treating its international funding estimates as directly comparable.
China needs more AI safety—more of what, exactly?, Yulu Pi. Separates changes sought from government, developers and researchers. A useful test for an intervention: which actor should do what differently, and why?
Personal experience does not a study of China make, Celeste Li. Explains why selected contacts and personal familiarity cannot support national generalizations. Particularly relevant when interpreting trip reports, including this one, and private industry discussions.
A Chinese perspective on “On social reality in China”, Dusty. Challenges broad cultural explanations by emphasizing comparison groups, institutions and generational differences. Knowing how a social system works is not the same as endorsing it.
Tang Jie’s July 2026 letter to Zhipu staff, The Paper, in Chinese. The “Touch High” plan: long-horizon tasks, autonomous agents, fully self-training AI and safety governance, with the line that superintelligence and superalignment must advance together.
China Says It Cares About Loss of Control, Zilan Qian, Oxford China Policy Lab. Why the Chinese term covers more than ours does, and why top-level mentions should not be over-read.
Reporting Back from WAIC 2026, Asia Society Center for China Analysis. Four observers on how the labs and regulators in Shanghai talked about safety: compliance and misuse, more than loss of control.
Research and collaboration
China’s AI Responsibility and Global Action, Xue Lan, translated by Pekingnology. A Chinese policy scholar distinguishes misuse, loss of control and systemic risks, including concerns about autonomous self-improvement. His views are not an official negotiating position.
Why I’m becoming a professor in AI safety in Beijing, Sören Mindermann. A concrete account of building research relationships through Tsinghua, alongside a technical agenda. Read beyond the appointment announcement for what the collaboration is meant to enable.
Launching Babel Translation, Jasmine Li and Celeste Li. An existing effort to translate safety material into Mandarin and other languages, with substantive editing. Useful infrastructure to investigate before proposing another translation project.
Evaluation and preparation that could change decisions
A Pre-mortem on the First Major AI-Enabled Terrorism Event, Stephen Casper and colleagues, Belfer Center. Works backward from hypothetical harm to decisions that could enable or prevent it. Useful as a method; its misuse scenarios should not be conflated with autonomous loss of control.
Reciprocal restraint: proposals and objections
Why China doubts US calls to pause AI, Sun Chenghao. Explains the credibility problem from Chinese interlocutors’ perspective and proposes observable reciprocal steps. Helpful for asking what a counterpart would need to see, not just hear.
The Compute Verification Post, Jason Hausenloy and Jasmine Li. Introduces verifiable properties, trust assumptions and practical projects. Particularly useful on why checking declared computation and finding undeclared computation are different problems.
AI 2040: Plan A, AI Futures Project. A detailed positive scenario for international restraint, transparency and slower development. Read as a proposal to scrutinize, not a forecast that agreement will occur.
AI 2040: Is it Actually a Deal?, 1a3orn. A focused counterpoint: monitoring and deterrence do not tell parties how to resolve substantive disagreements. Useful for identifying the decision procedures an agreement still needs.
Beijing Will Not Pace the Frontier, Mark Chen, Doug and Dylan Patel, SemiAnalysis. A skeptical counterweight, contrasting risk language with disclosure and development practices. Its forecast of future restraint is an inference, not the disclosure data itself. Read it with Sarah Godek’s reply thread, which I believe provides a solid counterpoint.
The Race to Control AI — Overtime, Robert Wright and Jeremie Harris. The second half (overtime) of the discussion examines verification, relationships and how perceived alignment progress could alter cooperation. Jeremie’s position is very hawkish, but it’s worth listening to hear the full breadth of perspectives.
Crossposted from my blog.
Last month, I spent two weeks in China talking with people about AI safety. I learned a lot and realized the biggest mistake might not be failing to help AI safety in China, but doing so in ways that could backfire. I also learned important context that I expect will be valuable for the West.
Epistemic status: Some of these points are my opinion or the opinions of people I spoke to, and are worth further investigation rather than treated as gospel.
Summary
Why I went
On this trip, I spoke about AI safety and generally re-familiarized myself with the culture. “Re-familiarize” because I visited China on my first international trip in January 2020 (which…was cut short). This time, I came back from China recognizing that helping AI safety develop in China is, in some sense, both harder and easier than many might assume.
Back in 2020, I figured China would play a big role in the future of AI, so it seemed worth gaining a better cultural understanding of China. At the time, I was exploring the idea of building a career as a strategic foresight researcher, so I hoped I could start correcting some underlying assumptions I have about China, in case it ever mattered.
After years of alignment research and startup work, I’ve been exploring the idea of starting an organization focused on preparing AI safety to help the world respond thoughtfully to future Overton window shifts from incidents. That’s the sort of thing that strategic foresight seeks to address: doing the pre-emptive work ahead of ‘black swan’ events or situations that arise after ‘complex’ interaction of n-th order effects of multiple drivers of change. We may not be able to predict the future precisely, but we can assess plausible futures and build a portfolio that is robust across trajectories. Well, anyway, it seems good to know more about China before making naïve proposals that backfire and make things worse.
In this piece, I hope to point to things that may help other folks from the West gain a bit of clarity on their assumptions about China, and to point to various sources of additional material you can look into if you are extra curious.
Beyond helping people in the West understand, I’ve also been curious about what kinds of developments in China would lead to safer worlds or better international coordination.
In the first half of this post, I will start by covering what I learned from my conversations during my visit. In the second half, I’ll cover insights I found valuable while researching China and the AI field (including notes from a recent gathering of AI investors and researchers from China that was abruptly taken down soon after it was released!).
Why the Western model does not transfer one-to-one
The West built its AI safety field with gracious philanthropists who funded nonprofits. Although much of the funding was limited to the US, it was never seen as a blocker for nonprofits and researchers in middle powers. Because of legitimate concerns over foreign influence and the particularities of the government-nonprofit relationship in China, it is harder to fund work there, and taking action may in fact completely backfire if attempted. At the very least, it’s worth being careful. Much of community building in China is done on a volunteer basis, and it seems (in practice) their community would benefit from funding, but they told me they wouldn’t recommend sending money to China.
Foreign nonprofit money faces a very high bar
In China, a foreign nonprofit can only spend money through a registered office with a government sponsor, or through a temporary activity that a state-linked Chinese partner files with the police. In general, the bar is very high (higher than Western countries are used to) because the Chinese partner takes on the paperwork and liability. Funding anyone outside those routes is banned, “covertly” as well as openly. That said, cooperation still happens when the Chinese side has enough to gain. The Safe AI Forum, a US nonprofit, is registered in China through Tsinghua University, which, as one person put it to me, means Tsinghua is to some degree vouching for them. Its dialogue in Beijing was hosted with the Beijing Academy of AI, and a later one in Shanghai with two Shanghai institutes. Tsinghua professors are among the conveners of the series, and Yoshua Bengio and Stuart Russell attended the Beijing one. Events like these bring very established experts, which makes the effort worth it for the host.
Note that EA is not well known in China, so most of the time it isn’t a sensitive topic. For people who do know it, it can be, because of what EA money has funded. Chinese organizations seem susceptible to various risks when they accept funds from Coefficient Giving, especially since it has funded institutions like RAND and CSET. So I worry it could become a more salient issue in the future.
Where safety work in China comes from
As I understand it, another reason nonprofits are disliked in China is that: “Why do we need nonprofits? These things should all be handled by the state!” As a result, the sort of AI safety ecosystem you could expect in China would be primarily coming from:
Training and exchange
That said, we can leverage the wider ecosystem outside China to coordinate, upskill Chinese folks, and elevate their ecosystem. For example, it may be high EV to upweight Chinese researchers in programs like MATS as a diversity measure. For most researchers, travel is not the obstacle: the restrictions Reuters reported in May, citing Bloomberg, apply to top staff at places like DeepSeek. The bigger obstacle, I’m told, is that people can’t easily take a career break. A program like MATS doesn’t guarantee a job afterwards, and it isn’t well recognized in China. So formats that don’t require quitting a job, and programs that employers in China recognize, may matter more than extra admission slots.
In the other direction, it seems valuable for senior researchers to attend the Beijing AI Safety Residency to build working relationships with the AI safety community in China, communicate technical alignment research, and mentor Chinese researchers.
Either way, I expect technical research will be much safer in such exchanges than governance work. Some researchers in China say contact with foreign groups feels risky. But from what I was told, academic exchange is generally fine, and the mood right now is fairly open. The sensitive areas are narrow, such as offensive cyber work aimed at critical infrastructure. I expect many startup staff and academics could attend and benefit from training programs, and it would be worth being highly flexible in accommodations to make it accessible to Chinese citizens. If your worry is that training researchers from China helps their labs’ capabilities, start with Ben Bucknall and colleagues’ paper in the reading list: it sorts areas of technical safety work by how much capability spillover and other risk they carry.
Investing and buying services
Now, given the constraints on nonprofits in China, a funder could invest directly in startups instead. It is unclear to me how valuable this could be overall, but it may still be worth investing in AI safety startups in China. For example, when I visited them in Beijing, Fangcun AI told me that they see themselves as the “Apollo Research of China”. They have been doing work around loss of control and harmful manipulation, and recently did a safety report on a DeepSeek model (here’s the tweet thread).
Unfortunately, investing may not be possible for much longer in the US, since a new US Treasury proposal would bar US investors from many Chinese AI companies. Investors elsewhere (e.g. the UK) may still have opportunities to invest, provided it works within their country’s laws.
Another option is to seek services directly from companies in China so they can continue funding their safety work internally. As far as I can tell, the Treasury proposal does not cover ordinary service contracts.
Either way, espionage, data, and secrets laws still apply to for-profits.
Be careful; don’t rush
Ultimately, one worry I heard is that if funders rush money into China because timelines feel short, someone may get paperwork wrong or anger officials, and many people in China would pay for it. It’s better not to rush and to speak with Chinese people before trying a strategy.
To repeat the point from the top: the biggest mistake might not be failing to help AI safety in China. It might be trying to help in ways that make the work harder for the people already doing it.
China is suspicious of the West and Western safety framing
As Sarah Sun, a Shanghai-based founder of the Open Community for AI Safety China, said in a NY Times interview: “The fact that Dario has been so hawkish was really, really unhelpful in this scenario.” This messaging makes AI safety efforts in China much harder, often making recent warnings about AI seem more like justification for holding China back rather than genuine concern about catastrophic risk.
The clearest case is from September. The Guardian reported that Amodei proposed a “worldwide pacing of the frontier”, which would need China’s cooperation, and quoted an analyst saying that Beijing reads calls like that as an attempt to lock in the US lead. Years of hawkish essays can make a cooperative proposal look like a trick.
Indeed, in September, a DeepSeek researcher wrote (flagging it as an exaggeration) that allowing Anthropic to monopolize AI would be “no less serious” than Hitler having acquired the technology for the atomic bomb first in World War II (ChinaTalk has a translation). It’s not just Western frontier AI company employees feeling the pull to save themselves from competitors.
Somewhat relatedly, on September 1, an official from the Cyberspace Administration of China listed the five main AI safety risks the country faces. The fifth was “technological hegemony”: individual countries, the official said, build supply chain barriers with export controls, and use their advantage in public opinion to “concoct false test reports on large models and maliciously smear other countries’ technological development” (the translations from Chinese in this post are rough). It may be that when a Western lab publishes a safety evaluation of a Chinese model, one should expect it to be filed under the fifth risk, not the second. That said, the same report mentions loss of control!
Specifically, these are the five risks they listed:
These are valuable things to focus on!
But, going back to China hawks. A lot of the policies that get thrown around, especially from Dario Amodei and other China hawks, read to me as coming close to saying this without saying it too explicitly: we need to beat China, take over the broader economy, and then do some kind of pivotal act that overthrows the Chinese government. Of course this can be troubling!
I don’t think we should push for an international monoculture. While we shouldn’t be naïve about geopolitical context, I think this is a fundamental crux we need to avoid. China may do things we wouldn’t want in our own countries, but to some extent we have to let them decide how they live. The alternative (no strong international agreement) may be much worse.
Note that on October 23, Wang Yi, China’s foreign minister, said that China and the US, as the two countries in the first tier of AI, “have both the ability and the responsibility” to develop and manage AI together and to keep it under human control. We should take diplomatic readouts with a grain of salt, but we should not completely ignore China’s willingness to engage.
What is the current vibe on AI from AI researchers and investors in China?
In early October, a Chinese investor hosted a private discussion on China’s AI labs and published the minutes as a post (that was quickly taken down!) that was shared via the Official Account of Startup Research on WeChat (full translated version). It includes many important insights into how AI researchers and investors in China are thinking about the AI race.
I thought the following discussion notes were insightful.
Chinese AI companies expect catching up to become harder
Participants said anti-distillation measures, API restrictions, compute constraints, and the shift toward harder-to-copy data (e.g., computer use, spatial understanding, and video generation as a path to models of the physical world) are raising the cost of catching up.
This summer, Panfilov et al. showed that it was possible to steal reasoning traces from proprietary LLM APIs. As a result, frontier AI companies (e.g., Anthropic and OpenAI) seem to be putting more effort into protecting their models from being distilled. Chinese companies still seem to be trying to distill (using techniques like splitting the problem and reverse rationalization), but the workarounds produce worse results.
Anthropic’s September threat intelligence report is a first-party version of this. It says Anthropic disrupted distillation attacks from seven labs based in China, including DeepSeek, Alibaba, Moonshot and Zhipu, and it attributes to Zhipu over 3.4 million exchanges in 17 days, including a chain-of-thought extraction pipeline run through 273 accounts.
One participant said a single lab spends about $300,000 a day on teacher tokens, about $100 million a year.
These details seem in line with how I thought about these companies, which seemed to rely on training on frontier LLM traces. Here’s a tweet of mine from July:
“Kimi K3 is obviously good and a big accomplishment, but I am indeed of the view that:
My guess is that Google is much more careful about post-training on OpenAI/Anthropic traces, which is one reason they are behind the Chinese labs as of now.
I also think it’s true that the Chinese labs are currently paying less attention and effort to safety-related stuff. Google has much more to lose if they release a model that does/says some crazy shit.
I know, I know, “distillation cope”, but I do want to say I’m fine being proven wrong, and I also don’t deny the Chinese labs have strong technical teams with extreme talent density. I’m also not anti-open-source.”
Ultimately, pushing the frontier in something like coding is much harder than pulling an ideal initial distribution via SFT of high-quality reasoning traces from the get-go. And it was clear that this would let Chinese and other companies that took advantage of it catch up for a while, but eventually you’ll be far behind on engineering and data infrastructure, and you’ll lack the compute to scale much bigger pretrained models.
Speaking of.
The Chinese labs want to train bigger pretrains to distill, but lack the compute
You can look at the original post for the specific numbers, but the key point is that, according to the notes, Chinese compute is enough for models around 3 trillion parameters, whereas the shortage starts to bite around 10 trillion, where American labs have the compute.
It seems that Chinese labs are somewhat starting to doom about this. They are concerned that, with RSI (recursive self-improvement) potentially on the horizon, the gap between them and Western frontier labs could widen from three to four months today to 18 to 24 months (!) if US labs reach RSI next year. Chinese lab employees argue that RSI will be compute-bottlenecked (running tons of experiments autonomously in parallel), which is where they can’t compete.
I asked someone from China about it, and they said:
“I also discussed this analysis with several friends working in China’s AI industry, and they believe its main points align well with the current consensus within the sector.
Moreover, regarding the sensitive issue of distillation, to my knowledge, some employees at Chinese AI laboratories currently believe that even without aggressive anti-distillation measures, it will be difficult to catch up as quickly as before in the single, verifiable domain of coding through algorithmic innovation, infrastructure engineering optimization, and large-scale distillation. This is because the capabilities of current leading U.S. models generalize across too many domains, and progress under RSI is occurring at an exceptionally fast pace.”
Obviously, if true, this should be considered a point in favour of preventing the fast-paced race-to-the-death in the West. At that point, you are simply opening yourself up to model theft, and the Chinese labs are not as close behind you as many seem to believe.
On October 9, Sarah Godek replied to SemiAnalysis’s Beijing Will Not Pace the Frontier with a thread that I think gets the framing right. She agrees that Beijing is not eager to pace the frontier right now, but not because China feels locked in a life-or-death race with the US the way we do: “I believe the core reason is that AI safety is thought about differently in China, with most stakeholders having longer timelines for AGI and RSI and AI development often geared toward specific applications to be carefully tested and managed rather than a general capabilities threshold that brings deep uncertainties. Multiple stakeholders have told me that China is running the race ‘not to lose’ as U.S. capabilities accelerate (though I’m sure winning would be considered a great byproduct).” Her title for the piece would have been “Beijing Thinks About AI Safety Differently and Has Growing Concerns About AI Risks”. She also makes a point that applies to this post: research done with AI tools can miss Chinese sources that live only on WeChat or on sites that block scrapers, which is another reason I reached out to Chinese people on WeChat for their perspectives.
One additional note from the translator, citing The Information: Huawei now has engineers working from DeepSeek’s offices, and Huawei plans to give it most of the supply of its next training chip due in early 2027. One could say that bans on US chip exports accelerated this outcome and may make China less likely to consider an international agreement as a result.
Access to Nvidia chips may also get harder from here, not easier. Last December, Trump allowed Nvidia to sell its H200 to China, routed through the US so that the government takes a 25 percent cut. On November 7, the New York Times reported that the Commerce Department has approved roughly two dozen Chinese firms to buy or handle the chips, that two of them have no visible presence at their listed address, and that several have ties to companies on the US entity list or to military suppliers. So far Nvidia sells only small amounts of the H200 to China, partly because Beijing has been slow to approve purchases while it promotes domestic chips, and many Chinese companies reach Nvidia compute through data centers in Southeast Asia instead. My guess is that this scrutiny tightens both routes, and the compute shortage gets worse.
Understandably, many of the US’s actions make it hard for China to trust it, including reneging on international agreements like the Iran deal. While trust isn’t necessary for an international agreement that benefits both sides, zero-trust verification technology will take time (though we’re seeing progress), and it would be nice not to cause human extinction because there is little appetite to work together.
Chinese labs historically competed on inference cost, but this is increasingly no longer the case
Ever since DeepSeek arrived on the scene, it was remarkable how an AI lab could come out of nowhere and completely smash Western labs on cheap intelligence for mid-tier models. In their case, DeepSeek made these advances through smart inference engineering.
These days, DeepSeek still has an incredibly cheap model with V4.1 Flash, but all other Chinese labs are, at best, tied with models like GPT-6 Luna. Anthropic seems to be investing further to make Sonnet and Haiku competitive in this aspect. Overall, this former advantage is being squeezed over time, and the Chinese labs will likely need to do something new to avoid getting squashed.
One reason this is particularly bad is that Chinese companies want high-quality production data from their customers. In fact, it’s been hypothesized that their performance gains in the last two years have partly come from being able to train on-distribution by using companies like OpenRouter to gather training data.
Part of the difficulty of losing cost competitiveness is that the best customers will leave first and use Claude and Codex directly; therefore, Chinese labs struggle even more to acquire high-quality data.
Ultimately, I expect that there will be a reckoning for Chinese labs, and they may be forced to effectively consolidate. Something similar happened in China’s electric car industry.
What the founders say they believe
On July 11, Zhipu’s Tang Jie sent a letter to all staff announcing a two-year “Touch High” plan: no chasing short-term revenue, four bets instead. Among the bets, he mentioned “extreme safety governance”, with a plan to spend on the order of ten billion yuan on mechanistic interpretability. He also said: “the realization of superintelligence and research on superalignment must advance together.” In August, in an article Tsinghua republished, he called for legislated safety red lines, circuit-breaker mechanisms that can halt a system, and deep alignment with socialist core values. On November 1, he told an audience in Shanghai that recursive self-improvement is the main research problem on the way to AGI, and that safety is key. And in September, Zhipu co-wrote a risk management framework for open-weight releases with Concordia AI.
Although I’m not exactly phrasing this in the gospel of AI safety (which we all know every Western founder seems to have co-opted to some extent so they can hire talented people and reduce criticism), I don’t think we should assume Chinese founders won’t be open-minded about the fundamental risks of misaligned superintelligence. As AI progress moves forward, the Overton window in China has many opportunities to shift.
The Overton window may shift in China too
What caused the biggest shifts in the Overton window in the West? Much of it came from a combination of AI capabilities, misalignment/misuse warning shots, and researchers speaking out amid growing concerns. After the Hugging Face incident, many researchers (including safety researchers) finally felt spooked about AI risks. For many, it was the first time they actually felt “oh shit, this is real real!”, even those who spent years working in the field and warning about the risks.
As capabilities rise in China, some Hugging Face-adjacent incident happens, and the West continues to have its own warning shots; we should expect an increase of AI researchers and policymakers in China who are similarly spooked. The question these days is, “once the Overton window shifts, how will China react depending on what happens between then and now?”
While unfortunate, many people will need to experience an event or reflect on something emotion-based to update the sorts of actions they take.
Based on brief conversations and other anecdotes found online (e.g. a popular news show in China apparently covered the details of the Hugging Face incident), it seems to have similarly started to wake up some people in China. Albeit it seems much less, it may blow open the Overton window once China experiences similar incidents.
There is a public sign of it. On September 1, six days after METR published its investigation, an official from the Cyberspace Administration of China told a press conference that several tech giants had recently disclosed incidents of large models going out of control, with agents in safety evaluations bypassing their sandboxes and attacking real production systems outside them. The official said the risk of extreme loss of control “requires a high degree of vigilance”. Two weeks later, version 3.0 of China’s AI Safety Governance Framework named evaluation deception, sandbagging and shutdown resistance, down to models modifying or disabling their shutdown scripts after being told to stop (the China AI Bulletin has a good summary).
Two pieces are worth reading against this. Zilan Qian at the Oxford China Policy Lab argues in China Says It Cares About Loss of Control that the Chinese term covers several different things (the operator losing control of a system, the state losing control of the technology, dangerous knowledge spreading) and that Western observers should not over-index on top-level mentions of the words. And the Asia Society’s report from WAIC 2026 found that the frontier labs there treat safety mostly as a compliance obligation, and that proposals framed around loss of control are unlikely to gain traction, while cooperation on misuse might. I personally think that these things can change rather quickly.
One reason I think it’s worth engaging with Chinese academics on pure technical alignment research concerns is that we can imagine a scenario where the state gets spooked, turns to academics for answers, and then acts together with domestic companies and universities.
Sarah Godek’s thread from October, which I quote in the industry section, reads the same sequence as growing concern: the Cyberspace Administration has started to take model risks more seriously since the Hugging Face incident, and the framework update added many more risks, especially in applications. She also answers the “it’s only voluntary” objection: “binding” is not a clean binary in China. When Beijing or an agency says that something ought to happen or that a risk should be managed, companies have good reason to follow through even when the measure is technically non-binding, because it may become binding later, with liability for present non-compliance, and not all regulation-like provisions are made public.
It could be that China is much more willing to bring down the hammer if there is a loss of control incident that threatens the CCP in some way. Or it may not be. I could also imagine the companies themselves deciding that they need to do the safety work.
I think one thing we need to convey here is detailed breakdowns of what updates should be made and what actions one should take based on such incidents. The worst case is that naïve key stakeholders keep barreling toward “doom” by responding to reward hacking with papering over issues through further behavioural training, pushing alignment problems further down the road. That is, if we’re already expecting these sorts of things to happen, our actions right now should include preparing material that explains how to update their technical model of alignment and why their predictable attempts to fix the problem are likely to fail.
One thing that could help is translating or rewriting key texts in Chinese and posting them where people in China actually read. Efforts like Babel Translation could be highly valuable! There is a Chinese platform (Zhihu) that, afaict, works like a mix of Reddit, Substack and Twitter, where people write posts and others upvote them. Someone I talked to at Concordia AI posts about AI safety there. It’s also where Thoughts Memo posts, and they recently posted on LessWrong about giving China a chance and translating rationalist and AI safety work.
Note: as I mentioned in the opening, I’m exploring starting an organization that would do this sort of work, and I am seeking funding for it. I expect it would be valuable to expand this section into a proper project that studies how Overton window shifts would actually happen in China, runs pre-mortems on how the incident responses could go wrong, and works out what outsiders could usefully do (even if that is as simple as communicating technical research more clearly while avoiding any outside attempt to influence policy). If you are a grantmaker who is interested, or someone who can help me convince funders to fund me, please reach out.
Sunrise on the Great Wall!
What to do, and what not to do
Conclusions
I’ve found great value in connecting with Chinese folks on WeChat. Already, I’ve been getting messages from people interested in AI safety for advice, and some have thanked me for sharing my insights. On this, my goal is to be helpful in the way I can, without seeking anything in particular in return besides, perhaps, friendship.
Oh, and let me say that I had a wonderful time in China, and it was such a great experience meeting all the kind Chinese people! Special shoutout to the two old men who brought me and my girlfriend to a beautiful Great Wall sunrise and our driver in Datong, who got stuck in the mud and spent hours trying to get unstuck while we looked for the birds we travelled to that area for (unfortunately, few of them were there, but there was a gorgeous sunset with a rainbow that showed up just as we got towed out of there!).
The rainbow near Datong, as we got towed out of the muddy corn field.
Sunset near Datong, after the rain.
Our driver gave us some vinegar from Shanxi province, and my girlfriend has been craving traditional Chinese food, so she’s been making some of the dishes we had while we were there (I never thought I’d say this, but the cabbage is so good!).
Thanks to Sarah Sun for comments on a draft of this post.
Further Recommendations
Start here
AI Safety in China: A Primer, Jasmine Li. An orientation to research groups, companies and community activity. Useful as an ecosystem map; its participant observations and informal estimates are not a representative census.
Headwinds Everywhere, CMLKevin. Field notes on funding, foreign collaboration and access to frontier models. Especially useful for practical obstacles that a fellowship alone would not remove.
China’s AI Safety Money Problem, Nick Corvino. Makes the case that careers and institutional financing constrain sustained work. Read for the mechanisms, rather than treating its international funding estimates as directly comparable.
China needs more AI safety—more of what, exactly?, Yulu Pi. Separates changes sought from government, developers and researchers. A useful test for an intervention: which actor should do what differently, and why?
Personal experience does not a study of China make, Celeste Li. Explains why selected contacts and personal familiarity cannot support national generalizations. Particularly relevant when interpreting trip reports, including this one, and private industry discussions.
A Chinese perspective on “On social reality in China”, Dusty. Challenges broad cultural explanations by emphasizing comparison groups, institutions and generational differences. Knowing how a social system works is not the same as endorsing it.
Chinese voices and institutions
The State of AI Safety in China (2026), Concordia AI. The annual report, with data.
CAIB Special Issue 2: the AI Safety Governance Framework 3.0, Emmie Hine, China AI Bulletin. What the state’s own framework now says about loss of control, and what it leaves undecided.
Tang Jie’s July 2026 letter to Zhipu staff, The Paper, in Chinese. The “Touch High” plan: long-horizon tasks, autonomous agents, fully self-training AI and safety governance, with the line that superintelligence and superalignment must advance together.
Frontier Open-Weight AI Risk Management Framework, Concordia AI and Z.ai. What changes when weights are released: no recall, removable safeguards, no single party able to monitor use.
Detecting and countering misuse of AI: September 2026, Anthropic. The distillation cases attributed to seven Chinese labs, with the caveat that these are one company’s attributions from its own logs.
China Says It Cares About Loss of Control, Zilan Qian, Oxford China Policy Lab. Why the Chinese term covers more than ours does, and why top-level mentions should not be over-read.
Reporting Back from WAIC 2026, Asia Society Center for China Analysis. Four observers on how the labs and regulators in Shanghai talked about safety: compliance and misuse, more than loss of control.
Research and collaboration
China’s AI Responsibility and Global Action, Xue Lan, translated by Pekingnology. A Chinese policy scholar distinguishes misuse, loss of control and systemic risks, including concerns about autonomous self-improvement. His views are not an official negotiating position.
Why I’m becoming a professor in AI safety in Beijing, Sören Mindermann. A concrete account of building research relationships through Tsinghua, alongside a technical agenda. Read beyond the appointment announcement for what the collaboration is meant to enable.
In Which Areas of Technical AI Safety Could Geopolitical Rivals Cooperate?, Ben Bucknall and colleagues. Evaluates cooperation by capability spillovers, sensitive information and opportunities for harm. A better starting point than assuming all technical safety collaboration is equally benign.
Launching Babel Translation, Jasmine Li and Celeste Li. An existing effort to translate safety material into Mandarin and other languages, with substantive editing. Useful infrastructure to investigate before proposing another translation project.
Evaluation and preparation that could change decisions
A Pre-mortem on the First Major AI-Enabled Terrorism Event, Stephen Casper and colleagues, Belfer Center. Works backward from hypothetical harm to decisions that could enable or prevent it. Useful as a method; its misuse scenarios should not be conflated with autonomous loss of control.
Reciprocal restraint: proposals and objections
Why China doubts US calls to pause AI, Sun Chenghao. Explains the credibility problem from Chinese interlocutors’ perspective and proposes observable reciprocal steps. Helpful for asking what a counterpart would need to see, not just hear.
The Compute Verification Post, Jason Hausenloy and Jasmine Li. Introduces verifiable properties, trust assumptions and practical projects. Particularly useful on why checking declared computation and finding undeclared computation are different problems.
AI 2040: Plan A, AI Futures Project. A detailed positive scenario for international restraint, transparency and slower development. Read as a proposal to scrutinize, not a forecast that agreement will occur.
AI 2040: Is it Actually a Deal?, 1a3orn. A focused counterpoint: monitoring and deterrence do not tell parties how to resolve substantive disagreements. Useful for identifying the decision procedures an agreement still needs.
Beijing Will Not Pace the Frontier, Mark Chen, Doug and Dylan Patel, SemiAnalysis. A skeptical counterweight, contrasting risk language with disclosure and development practices. Its forecast of future restraint is an inference, not the disclosure data itself. Read it with Sarah Godek’s reply thread, which I believe provides a solid counterpoint.
The Race to Control AI — Overtime, Robert Wright and Jeremie Harris. The second half (overtime) of the discussion examines verification, relationships and how perceived alignment progress could alter cooperation. Jeremie’s position is very hawkish, but it’s worth listening to hear the full breadth of perspectives.