re: 1b (development likely impossible to kept secret given scale required)
I'm remind of Dylan Patel's comments (semianalysis) on a recent episode of the Dwarkesh Podcast which goes something like:
if you're Xi Jinping and you're scaling pilled, you can just centralize all the compute and build all the substations for it. You can just hide it inside one of the factories you already have that's drawing power for steel production and re-purpose it as a data center.
Given the success we've seen in training SOTA models with constrained GPU resources (Deepseek), I don't think it's far fetched to think you can hide bleeding edge development. It turns out all you need is a few hundred of the smartest people in your country and a few thousand GPUs.
Hrm...sounds like the size of the Manhattan Project.
1We do not take a position on the likelihood of loss of control.
This seems worth taking a position on, the relevant people need to hear from the experts an unfiltered stance of "this is a real and perhaps very likely risk".
It seems to me that by saying this the authors wanted to communicate "this is not a place to discuss this". But I agree that the phrasing used may inaccurately (?) communicate that the authors are more uncertain/agnostic about this issue than they really are (or that they believe something like "both sides have comparably good arguments"), so I'd suggest to replace it with something like:
The probability of loss of control is beyond the scope of this report (for discussion, see: [sources]).
This argument neglects the option of racing -with-plausible-deniability. I would argue that both the US and China are already doing this. We haven't gone to war yet.
Is 'Stargate' not racing?
Many have argued that nationalizing the major AI companies would substantially slow down progress because of bureaucratic overhead, reorganization costs, loss of immigrant personnel, and stifling of creativity.
If this were my working model of the world, and I wanted to help the US win the race I might:
Invest in on-shoring the means of production for the AI vertical (in progress)
Invest in expanding supporting infrastructure (in progress)
Place plausibly state-disconnected but trusted people in positions of power, like board seats (some done, some in progress).
Award lucrative military contracts for not-explicitly-offensive-purposes. (Some done, probably more in progress).
Bring heads of the major labs to the head of government for secret meetings, and establish close working relationships (done, and on going).
Place secret operatives and surveillance tech within the key companies (unobservable unless caught).
Place restrictions on export of key materials and technological information (done and on going).
Create a national government org explicitly to monitor progress in AI (done).
Purchase large amounts of computer equipment under false pretenses, hide it in secret facilities, and refuse to discuss the location or purpose of confronted. (Done) https://www.lesswrong.com/posts/grvJay8Cv3TBhXz3a/secret-us-natsec-project-with-intel-revealed
Seems like a lot of evidence in favor of a plausibly deniable soft-nationalization race. Can you present any counter-evidence?
An ASI project would be highly distinguishable from civilian AI applications and not integrated with a state’s economy
Why? I think there's a smooth ramp from economically useful AI to superintelligence: AIs gradually become better at many tasks, and these tasks help more and more with improving AI in turn.
Pages 22-23:
Arms control for AI in general is therefore unlikely to succeed. The military and civilian applications of general-purpose systems are nearly indistinguishable, and AI will likely see wide use across military and civilian society.
However, the opposite may be true of ASI development control: ASI development would likely be distinguishable from most civilian AI development, and, so long as it is not developed, unintegrated in a state’s economy.
It's not obvious to me either.
At least in the current paradigm, it seems plausible that a state project of (or, deliberately aimed at) developing ASI would yield a lot of intermediate non-ASI products that would then be dispersed into the economy or military. That's what we've been seeing until now.
Are there reasons to expect this not to continue?
One reason might be that an "ASI Manhattan Project" would want to keep their development secrets so as to minimize information leakage. But would they keep literally all useful intermediate products to themselves? Even if they reveal some X, civilians play with this X, and conclude that X is useless for the purpose of developing ASI, this might still be a valuable negative result that closes off some until-then-plausible ASI development paths.
This is one reason, I think the Manhattan Project is a poor model for a state ASI project. Intermediate results of the original Manhattan Project didn't trickle down into the economy while the project was still ongoing. I'm not claiming that people are unaware of those disanalogies but I expect thinking in terms of an "ASI Manhattan Project" encourages overanchoring on it.
Contra 2:
ASI might provide a strategic advantage of a kind which doesn't negatively impact the losers of the race, e.g. it increases GDP by x10 and locks competitors out of having an ASI.
Then, losing control of the ASI could [not being able of] posing an existential risk to the US.
I think it's quite likely this is what some policymakers have in mind: some sort of innovation which will make everything better for the country by providing a lot cheap labor and generally improving productivity, the way we see AI applications do right now but on a bigger scale.
Comment on 3:
Not sure who your target audience is; I assume it would be policymakers, in which case I'm not sure how much weight that kind of argument has? I'm not a US citizen, but from international news I got the impression that current US officials would rather relish the option to undermine the liberal democracy they purport to defend.
ASI might provide a strategic advantage of a kind which doesn't negatively impact the losers of the race, e.g. it increases GDP by x10 and locks competitors out of having an ASI.
It does negatively impact the losers, to the extent that they're interested not only in absolute wealth but also relative wealth (which I expect to be the case, although I know ~nothing about SotA modeling of states as rational actors or whatever).
Very informative piece that does a lot in the right direction. Articles like this can have a real impact on policy demonstrating “there be dragons”.
A criticism would be that it doesn’t account for the state of the board in reality - the trust dilemma fails under circumstances where domestic commercial incentives overwhelm international cooperative concerns and collapses the situation to a prisoners dilemma, unfortunately, I think. I hope there are trust based solutions, and I’m mistaken.
I'm not sure about the trust dilemma analysis.
It seems to me like it switches between two levels of abstraction.
Cooperate-Cooperate may be more desirable for both states' citizens but at the same time Defect-Cooperate may be more desirable for state A as a state qua rational actor.
Below is the executive summary of our new paper, The Manhattan Trap. Please visit the link above to see the full paper. We also encourage discussion and feedback in the comments here.
This paper examines the strategic dynamics of international competition to develop Artificial Superintelligence (ASI). We argue that the same assumptions that might motivate the US to race to develop ASI also imply that such a race is extremely dangerous.
A race to develop ASI is motivated by two assumptions: that ASI provides a decisive military advantage (DMA) to the first state that develops it, and that states are rational actors aware of ASI's strategic implications.
However, these same assumptions make racing catastrophically dangerous for three reasons.
First, an ASI race creates a threat to strategic stability that could trigger a war between the US and its adversaries, particularly China. If ASI could provide a decisive military advantage, states would rationally view their adversaries' ASI development as an existential threat justifying military intervention and espionage. A state cannot race to ASI without incurring significant risk of great power conflict unless either (a) its adversaries are unaware of ASI's importance (contradicting the awareness assumption), (b) development can be kept secret (likely impossible given the scale required), or (c) adversaries believe they can win the race (requiring a close competition in which victory is not assured).
Second, racing heightens the risk of losing control of an ASI system once developed. We do not take a position on the likelihood of loss of control. Instead, we observe that the argument for racing assumes that ASI would wield a decisive advantage over the militaries of global superpowers; accordingly, losing control of such a technology would also present an existential threat to the state that developed it. We also argue that an ASI would only provide a DMA if its capabilities scaled extremely rapidly—precisely the scenario in which loss of control risk is theoretically highest. Finally, a race compounds this risk by creating competitive pressure to develop ASI before adequate control measures are in place.
Third, even a controlled ASI threatens to disrupt the internal power structures of the state that develops it. In the case of a US, a successful ASI project would likely undermine the liberal democracy it purports to defend. An ASI system that would grant a state international advantage would also grant its controllers unprecedented domestic power, creating an extreme concentration of power likely incompatible with democratic checks and balances. Racing exacerbates this risk by requiring development to occur quickly and in secret—and therefore without public input.
These three dangers—great power conflict, loss of control, and power concentration—represent three successive barriers a state would need to overcome to 'win' a race to ASI. Together, they imply that an ASI race poses an existential threat to the national security of the states involved. Assuming states are informed and rational, the strategic situation can be modeled as a trust dilemma: states would prefer mutual restraint to racing and will only race if they believe others will. States can therefore avoid an ASI race by establishing a verification regime—a set of mechanisms that can verify a state's compliance with an international agreement not to pursue an ASI project. An ASI project would be highly distinguishable from civilian AI applications and not integrated with a state's economy—precisely the conditions under which verification regimes have historically succeeded.
The paper concludes that, if an ASI race is motivated, then cooperation to avoid an ASI race is both preferable and strategically sound. The assumptions that make racing seem necessary are the very reasons it is unwise.