Our government is determined to lose the AI race in the name of winning the AI race.
The least we can do, if prioritizing winning the race, is to try and actually win it.
This is a bizarre pair of claims to make. But I think it illustrates a surprisingly common mistake from the AI safety community, which I call "jumping down the slippery slope". More on this in a forthcoming blog post, but the key idea is that when you look at a situation from a high level of abstraction, it often seems like sliding down a slippery slope towards a bad equilibrium is inevitable. From that perspective, the sort of people who think in terms of high-level abstractions feel almost offended when people don't slide down that slope. On a psychological level, the short-term benefit of "I get to tell them that my analysis is more correct than theirs" outweighs the long-term benefit of "people aren't sliding down the slippery slope".
One situation where I sometimes get this feeling is when a shopkeeper charges less than the market rate, because they want to be kind to their customers. This is typically a redistribution of money from a wealthier person to less wealthy people; and either way it's a virtuous thing to do. But I sometimes actually get annoyed at them, and itch to smugly say "listen, you dumbass, you just don't understand economics". It's like a part of me thinks of reaching the equilibrium as a goal in itself, whether or not we actually like the equilibrium.
This is obviously a much worse thing to do in AI safety. Relevant examples include Situational Awareness and safety-motivated capability evaluations (e.g. "building great capabilities evals is a thing the labs should obviously do, so our work on it isn't harmful"). It feels like Zvi is doing this here too. Why is trying to actually win it the least we can do? Isn't this exactly the opposite of what would promote crucial international cooperation on AI? Is it really so annoying when your opponents are shooting themselves in the foot that it's worth advocating for them to stop doing that?
It kinda feels like the old joke:
On a beautiful Sunday afternoon in the midst of the French Revolution the revolting citizens led a priest, a drunkard and an engineer to the guillotine. They ask the priest if he wants to face up or down when he meets his fate. The priest says he would like to face up so he will be looking towards heaven when he dies. They raise the blade of the guillotine and release it. It comes speeding down and suddenly stops just inches from his neck. The authorities take this as divine intervention and release the priest.
The drunkard comes to the guillotine next. He also decides to die face up, hoping that he will be as fortunate as the priest. They raise the blade of the guillotine and release it. It comes speeding down and suddenly stops just inches from his neck. Again, the authorities take this as a sign of divine intervention, and they release the drunkard as well.
Next is the engineer. He, too, decides to die facing up. As they slowly raise the blade of the guillotine, the engineer suddenly says, "Hey, I see what your problem is ..."
Zvi is arguing "X implies Y" here. Zvi happens to believe Y but disbelieve X; however, he is writing to people who think "X and not-Y", in order to nudge them to support Y.
Here X = it is good for the US to build superintelligence fast, before China does, and Y = we should have some diffusion rules making it harder for China to catch up to the USA.
Zvi believes Z = nobody should be building superintelligence soon, and believes Z implies Y, but it is useful to show that X implies Y as well.
Hot (?) take, the USG shooting itself in the foot as it pertains to AI is good actually and we should not be risking interrupting them.
Like, okay, there are different ways the USG could shoot itself in the foot:
(1) is obviously bad. But (3) and (4) are great[1].
And (2) is also potentially good: because the others would use the resources worse or won't use them for AGI acceleration at all.
Like, take China. Its mindset is famously of a "fast follower", with dicey attempts at innovation being internally unpopular[2]; and the Chinese AI researchers probably know as much about the world-dominantion fantasies motivating American CEOs as they do about the AGI doom (i. e., barely anything, reportedly). So there's neither willingness nor motivation to race to AGI there. ... Unless the US AGI labs succeed at manufacturing that race. Which, come to think of it, they would stop trying to do if they start believing they'd lose it.
So the US AGI labs losing beneficial access to the raw resources that could be converted into AI progress (chips, energy, talent) is good in my books.
There's a potential argument here that the US AGI companies would be better to have ahead because they're more likely to get AGI right due to being more safety-conscious, or offer us more opportunities to reform them into being properly safety-conscious. I don't think much of that argument. I would rather have e. g. 3 more years until AGI than bet on fringe possibilities like those.
There's also an argument that keeping the raw resources in Western hands would make it easier to ban AGI research (by controlling supply chains and/or negotiating an international ban with China) if we do manage to wake the USG up to the omnicide risk. This is a more solid argument... But still something I'd trade away for timelines a few years longer.
As they pertain to slowing down AI progress, I mean. Obviously they can be parts of overall-terrible-for-the-world policies like tariffs or restricting immigration.
DeepSeek is not evidence against this vision, but rather, its confirmation: they did not innovate, only reverse-engineered and optimized.
I disagree on DeekSeek and innovation. Yes R1 is obviously a reaction to o1, but its MoE model is pretty innovative, and it is Llama 4 that obviously copied DeepSeek. But yes I agree innovation is unpopular in China. But from interviews of DeepSeek founder Liang Wenfeng, we know DeepSeek was explicitly an attempt to overcome China's unwillingness to innovate.
it is Llama 4 that obviously copied DeepSeek
DeepSeek-V3's MoE architecture is unusual in having high granularity, 8 active experts rather than the usual 1-2. Llama 4 Maverick doesn't do that[1]. The closest thing is the recent Qwen3-235B-A22B, which also has 8 active experts.
From the release blog post:
↩︎As an example, Llama 4 Maverick models have 17B active parameters and 400B total parameters. ... MoE layers use 128 routed experts and a shared expert. Each token is sent to the shared expert and also to one of the 128 routed experts.
its MoE model is pretty innovative
I would roughly punt it into the category of "optimization", not "innovation". "Innovation" is something like transformers, instruct-training, or RL-on-CoTs. MoE scaling is an incremental-ish improvement.
Or, to put it in other words: it's an innovation in the field of compute-optimal algorithms/machine learning. It's not an AI innovation.
But from interviews of DeepSeek founder Liang Wenfeng, we know DeepSeek was explicitly an attempt to overcome China's unwillingness to innovate
Yes, and we're yet to see them succeed. And with the CCP having apparently turned its sights on them, that attempt may be thoroughly murdered already.
Umm... I have already warned that inner troubles of the American administration might cost OpenBrain lots of compute, which can influence the AI race.
I have also made a comment where I tried to show that the US leadership would be undermined by the Taiwan invasion unless the US domestic chip production dominates the Chinese one. It would be especially terrifying to discover that both OpenBrain and DeepCent have similar amounts of compute while neither side[1] can increase these amounts faster than the other (and I did imply something similar in the comment I made!), since neither the US nor China can slow down without an international deal. And a hypothetical decay of the US makes the matters worse for the US.
Moreover, I have mentioned the possibility that the US administration realises that they have a misaligned AI, but without the AI-driven transformation of the economy the US will be unable to produce more chips and/or energy, leaving China with leadership. Then the US could be forced to let the AI transform the economy. Or threaten to unleash the misaligned AI unless China somehow surrenders its potential leadership...
Could you ask the AI-2027 team to reconsider the compute forecast and estimate the influence of the revised compute and power of the AIs on the other aspects of the scenario?
The slowdown ending of AI-2027.com had OpenBrain receive compute by merging with its rivals. The collapsed section about the Indo-Pakistan nuclear war (which would be equivalent to the Taiwan invasion in 2025) in my comment describes a situation where OpenBrain and its former rivals have done a number of computations similar to DeepCent and its former rivals.
To not only fail to robustly support and bring down regulatory and permitting barriers to the nuclear power we urgently need to support our data centers
Nuclear power for data centers is nice in theory, but if you have timelines ~8 years or less for superintelligence, then for practical purposes basically none of the relevant data centers will be powered by new nuclear plants in the US regardless. Of course if they get rid of enough government support they might force some of the old ones to shut down, but that seems unlikely.
Our government is determined to lose the AI race in the name of winning the AI race.
The least we can do, if prioritizing winning the race, is to try and actually win it.
It is one thing to prioritize ‘winning the AI race’ against China over ensuring that humanity survives, controls and can collectively steer our future. I disagree with that choice, but I understand it. This mistake is very human.
I also believe that more alignment and security efforts at anything like current margins not only do not slow our AI efforts, they would actively help us win the race against China, by enabling better diffusion and use of AI, and ensuring we can proceed with its development. So the current path is a mistake even if you do not worry about humanity dying or losing control over the future.
However, if you look at the idea of building smarter, faster, more capable, more competitive, freely copyable digital minds we don’t understand that can be given goals and think ‘oh that future will almost certainly stay under humanity’s control and not be a danger to us in any way’ (and when you put it like that, um, what are you thinking?) then I understand the second half of this mistake as well.
What is not an understandable mistake, what I struggle to find a charitable and patriotic explanation for, is to systematically cripple or give away many of America’s biggest and most important weapons in the AI race, in exchange for thirty pieces of silver and some temporary market share.
To continue alienating our most important and trustworthy allies with unnecessary rhetoric and putting up trading barriers with them. To attempt to put tariffs even on services like movies where we already dominate and otherwise give the most important markets, like the EU, every reason in their minds to put up barriers to our tech companies and question our reliability as an ally. And simultaneously in the name of building alliances put the most valuable resources with unreliable partners like Malaysia, Saudi Arabia, Qatar and the UAE.
Indeed, we have now scrapped the old Biden ‘AI diffusion’ rule with no sign of its replacement, and where did David Sacks gloat about this? Saudi Arabia, of course. This is what ‘trusted partners’ means to them. Meanwhile, we are warning sterny against use of Huawai’s AI chips, ensuring China keeps all those chips itself. Our future depends on who has the compute, who ends up with the chips. We seem to instead think the future is determined by the revenue from chip manufacturing? Why would that be a priority? What do these people even think is going on?
To not only fail to robustly support and bring down regulatory and permitting barriers to the nuclear power we urgently need to support our data centers, but to actively wipe out the subsidies on which the nuclear industry depends, as the latest budget aims to do with remarkably little outcry via gutting the LPO and tax credits, while China of course ramps up its nuclear power plant construction efforts, no matter what the rhetoric on this might say. Then to use our inability to power the data centers as a reason to put our strategically vital data centers, again, in places like the UAE, because they can provide that power. What do you even call that?
To fail to let our AI companies have the ability to recruit the best and brightest, who want to come here and help make America great, instead throwing up more barriers and creating a climate of fear I’m hearing is turning many of the best people away.
And most of all, to say that the edge America must preserve, the ‘race’ that we must ‘win,’ is somehow the physical production of advanced AI chips. So, people say, in order to maintain our edge in chip production, we should give that edge entirely away right now, allowing those chips to be diverted to China, as would be inevitable in the places that are looking to buy where we seem most eager to enable sales. Nvidia even outright advocates that it should be allowed to sell to China openly, and no one in Washington seems to hold them accountable for this.
And we are doing all this while many perpetuate the myth that our AI efforts are not very solidly ahead of China in the places that matter most, or threaten to lock in the world’s customers, because DeepSeek which is impressive but still very clearly substantially behind our top labs, or because TikTok and Temu exist while forgetting that the much bigger Amazon and Meta also exist.
Temu’s sales are less than a tenth of Amazon’s, and the rest of the world’s top four e-commerce websites are Shopify, Walmart.com and eBay. As worrisome as it is, TikTok is only the fourth largest social media app behind Facebook, YouTube and Instagram, and there aren’t signs of that changing. Imagine if that situation was reversed.
Earlier this week I did an extensive readthrough and analysis of the Senate AI Hearing.
Here, I will directly lay out my response to various claims by and cited by US AI Czar David Sacks about the AI Diffusion situation and the related topics discussed above.
There are multiple distinct forms of Obvious Nonsense to address, either as text or very directly implied, whoever you attribute the errors to:
David Sacks (US AI Czar): Writing in NYT, former Google CEO Eric Schmidt warns that “China Tech Is Starting to Pull Ahead”:
“China is at parity or pulling ahead of the United States in a variety of technologies, notably at the A.I. frontier. And it has developed a real edge in how it disseminates, commercializes and manufactures tech. History has shown us that those who adopt and diffuse a technology the fastest win.”
As he points out, diffusing a technology the fastest — and relatedly, I would add, building the largest partner ecosystem — are the keys to winning. Yet when Washington introduced an “AI Diffusion Rule”, it was almost 200 pages of regulation hindering adoption of American technology, even by close partners.
The Diffusion Rule is on its way out, but other regulations loom.
President Trump committed to rescind 10 regulations for every new regulation that is added.
If the U.S. doesn’t embrace this mentality with respect to AI, we will lose the AI race.
Sriram Krishnan: Something @DavidSacks and I and many others here have been emphasizing is the need to have broad partner ecosystems using American AI stack rather than onerous complicated regulations.
If the discussion was ‘a bunch of countries like Mexico, Poland and Portugal are in Tier 2 that should instead have been in Tier 1’ then I agree there are a number of countries that probably should have been Tier 1. And I agree that there might well be a simpler implementation waiting to be fond.
And yet, why is it that in practice, these ‘broad partner ecosystems using American AI’ always seem to boil down to a handful of highly questionably allied and untrustworthy Gulf States with oil money trying to buy global influence, perhaps with a side of Malaysia and other places that are very obviously going to leak to China? David Sacks literally seems to think that if you do not literally put the data center in specifically China, then that keeps it in friendly hands and out of China’s grasp, and that we can count on our great friendships and permanent alliances with places like Saudi Arabia. Um, no. Why would you think that?
That Eric Schmidt editorial quoted above is a royal mess. For example, you have this complete non-sequitur.
Eric Schmidt and Selina Xu: History has shown us that those who adopt and diffuse a technology the fastest win.
So it’s no surprise that China has chosen to forcefully retaliate against America’s recent tariffs.
China forcefully retaliated against America’s tariffs for completely distinct reasons. The story Schmidt is trying to imply here doesn’t make any sense. His vibe reports are Just So Stories, not backed up at all by economic or other data.
‘By some benchmarks’ you can show pretty much anything, but I mean wow:
Eric Schmidt and Selina Xu: Yet, as with smartphones and electric vehicles, Silicon Valley failed to anticipate that China would find a way to swiftly develop a cheap yet state-of-the-art competitor. Today’s Chinese models are very close behind U.S. versions. In fact, DeepSeek’s March update to its V3 large language model is, by some benchmarks, the best nonreasoning model.
Look. No. Stop.
He then pivots to pointing out that there are other ‘tech’ areas where China is competitive, and goes into full scaremonger mode:
Apps for the Chinese online retailers Shein and Temu and the social media platforms RedNote and TikTok are already among the most downloaded globally. Combine this with the continuing popularity of China’s free open-source A.I. models, and it’s not hard to imagine teenagers worldwide hooked on Chinese apps and A.I. companions, with autonomous Chinese-made agents organizing our lives and businesses with services and products powered by Chinese models.
As I noted above, ‘American online retailers like Amazon and Shopify and the social media platforms Facebook and Instagram are already not only among but the most used globally.’
There is a stronger case one can make with physical manufacturing, when Eric then pivots to electric cars (and strangely focuses on Xiaomi over BYD) and industrial robotics.
Then, once again, he makes the insane ‘the person behind is giving away their inferior tech so we should give away our superior tech to them, that’ll show them’ argument:
We should learn from what China has done well. The United States needs to openly share more of its A.I. technologies and research, innovate even faster and double down on diffusing A.I. throughout the economy.
When you are ahead and you share your model, you give your rivals that model for free, killing your lead and your business for some sort of marketing win, and also you’re plausibly creating catastrophic risk. When you are behind, and you share it, sure, I mean why not.
In any case, he’s going to get his wish. OpenAI is going to release an open weight reasoning model, reducing America’s lead in order to send the clear message that yes we are ahead. Hope you all think it was worth it.
The good AI argument is that China is doing a better job in some ways of AI diffusion, of taking its AI capabilities and using them for mundane utility.
Similarly, I keep seeing forms of an argument that says:
I’m sorry, what?
At lunch during Selina’s trip to China, when U.S. export controls were brought up, someone joked, “America should sanction our men’s soccer team, too, so they will do better.” So that they will do better.
It’s a hard truth to swallow, but Chinese tech has become better despite constraints, as Chinese entrepreneurs have found creative ways to do more with less. So it should be no surprise that the online response in China to American tariffs has been nationalistic and surprisingly optimistic: The public is hunkering down for a battle and thinks time is on Beijing’s side.
I don’t know why Eric keeps talking about the general tariffs or trade war with China here, or rather I do and it’s very obviously a conflation designed as a rhetorical trick. That’s a completely distinct issue, and I here take no position on that fight other than to note that our actions were not confined to China, and we very obviously shouldn’t be going after our trading partners and allies in these ways – including by Sacks’s logic.
The core proposal here is that, again:
It’s literal text. “America should sanction our men’s soccer team, too, so they will do better.” Should we also go break their legs? Would that help?
Then there’s a strange mix of ‘China is winning so we should become a centrally planned economy,’ mixed with ‘China is winning so we cannot afford to ever have any regulations on everything.’ Often both are coming from the same people. It’s weird.
So, shouting from the rooftops, once more with feeling for the people in the back:
Or, as Derek Thompson put it:
Derek Thompson: Trump’s new AI directive (quoted below from David Sacks) argues the US should take care to:
– respect our trading partners/allies rather than punish them with dumb rules that restrict trade
– respect “due process”
It’d be interesting to apply these values outside of AI!
Jordan Schneider: It’s an NVDA press release. Just absurd.
David Sacks continues to beat the drum that the diffusion rule ‘undermines the goal of winning the AI race,’ as if the AI race is about Nvidia’s market share. It isn’t.
If we want to avoid allocations of resources by governmental decision, overreach of our executive branch authorities to restrict trade, alienating US allies and lack of due process, Sacks’s key points here? Yeah, those generally sound like good ideas.
To that end, yes, I do believe we can improve on Biden’s proposed diffusion rules, especially when it comes to US allies that we can trust. I like the idea that we should impose less trade restrictions on these friendly countries, so long as we can ensure that the chips don’t effectively fall into the wrong hands. We can certainly talk price.
Alas, in practice, it seems like the actual plans are to sell massive amounts of AI chips to places like UAE, Saudi Arabia and Malaysia. Those aren’t trustworthy American allies. Those are places with close China ties. We all know what those sales really mean, and where they could easily be going. And those are chips we could have kept in more trustworthy and friendly hands, that are eager to buy them, especially if they have help facilitating putting those chips to good use.
The policy conversations I would like to be having would focus not only on how to best superchange American AI and the American economy, but also on how to retain humanity’s ability to steer the future and ensure AI doesn’t take control, kill everyone or otherwise wipe out all value. And ideally, to invest enough in AI alignment, security, transparency and reliability that there would start to be a meaningful tradeoff where going safer would also mean going slower.
Alas. We massively underinvesting in reliability and alignment and security purely from a practical utility perspective and we not even having that discussion.
Instead we are having a discussion about how, even if your only goal is ‘America must beat China and let the rest handle itself,’ to stop shooting ourselves in the foot on that basis alone.
The very least we can do is not shoot ourselves in the foot, and not sell out our future for a little bit of corporate market share or some amount of oil money.