This is a linkpost for https://www.convergenceanalysis.org/research/the-manhattan-trap-why-a-race-to-artificial-superintelligence-is-self-defeating
The Manhattan Trap: Why a Race to Artificial Superintelligence is Self-Defeating
14t14n
8plex
3Mateusz Bagiński
8Nathan Helm-Burger
8cousin_it
2Mateusz Bagiński
5momom2
2Mateusz Bagiński
4Corin Katzke
3Convolutions
2Mateusz Bagiński
New Comment
re: 1b (development likely impossible to kept secret given scale required)
I'm remind of Dylan Patel's comments (semianalysis) on a recent episode of the Dwarkesh Podcast which goes something like:
if you're Xi Jinping and you're scaling pilled, you can just centralize all the compute and build all the substations for it. You can just hide it inside one of the factories you already have that's drawing power for steel production and re-purpose it as a data center.
Given the success we've seen in training SOTA models with constrained GPU resources (Deepseek), I don't think it's far fetched to think you can hide bleeding edge development. It turns out all you need is a few hundred of the smartest people in your country and a few thousand GPUs.
Hrm...sounds like the size of the Manhattan Project.
1We do not take a position on the likelihood of loss of control.
This seems worth taking a position on, the relevant people need to hear from the experts an unfiltered stance of "this is a real and perhaps very likely risk".
It seems to me that by saying this the authors wanted to communicate "this is not a place to discuss this". But I agree that the phrasing used may inaccurately (?) communicate that the authors are more uncertain/agnostic about this issue than they really are (or that they believe something like "both sides have comparably good arguments"), so I'd suggest to replace it with something like:
The probability of loss of control is beyond the scope of this report (for discussion, see: [sources]).
This argument neglects the option of racing -with-plausible-deniability. I would argue that both the US and China are already doing this. We haven't gone to war yet.
Is 'Stargate' not racing?
Many have argued that nationalizing the major AI companies would substantially slow down progress because of bureaucratic overhead, reorganization costs, loss of immigrant personnel, and stifling of creativity.
If this were my working model of the world, and I wanted to help the US win the race I might:
Invest in on-shoring the means of production for the AI vertical (in progress)
Invest in expanding supporting infrastructure (in progress)
Place plausibly state-disconnected but trusted people in positions of power, like board seats (some done, some in progress).
Award lucrative military contracts for not-explicitly-offensive-purposes. (Some done, probably more in progress).
Bring heads of the major labs to the head of government for secret meetings, and establish close working relationships (done, and on going).
Place secret operatives and surveillance tech within the key companies (unobservable unless caught).
Place restrictions on export of key materials and technological information (done and on going).
Create a national government org explicitly to monitor progress in AI (done).
Purchase large amounts of computer equipment under false pretenses, hide it in secret facilities, and refuse to discuss the location or purpose of confronted. (Done) https://www.lesswrong.com/posts/grvJay8Cv3TBhXz3a/secret-us-natsec-project-with-intel-revealed
Seems like a lot of evidence in favor of a plausibly deniable soft-nationalization race. Can you present any counter-evidence?
An ASI project would be highly distinguishable from civilian AI applications and not integrated with a state’s economy
Why? I think there's a smooth ramp from economically useful AI to superintelligence: AIs gradually become better at many tasks, and these tasks help more and more with improving AI in turn.
Pages 22-23:
Arms control for AI in general is therefore unlikely to succeed. The military and civilian applications of general-purpose systems are nearly indistinguishable, and AI will likely see wide use across military and civilian society.
However, the opposite may be true of ASI development control: ASI development would likely be distinguishable from most civilian AI development, and, so long as it is not developed, unintegrated in a state’s economy.
It's not obvious to me either.
At least in the current paradigm, it seems plausible that a state project of (or, deliberately aimed at) developing ASI would yield a lot of intermediate non-ASI products that would then be dispersed into the economy or military. That's what we've been seeing until now.
Are there reasons to expect this not to continue?
One reason might be that an "ASI Manhattan Project" would want to keep their development secrets so as to minimize information leakage. But would they keep literally all useful intermediate products to themselves? Even if they reveal some X, civilians play with this X, and conclude that X is useless for the purpose of developing ASI, this might still be a valuable negative result that closes off some until-then-plausible ASI development paths.
This is one reason, I think the Manhattan Project is a poor model for a state ASI project. Intermediate results of the original Manhattan Project didn't trickle down into the economy while the project was still ongoing. I'm not claiming that people are unaware of those disanalogies but I expect thinking in terms of an "ASI Manhattan Project" encourages overanchoring on it.
Contra 2:
ASI might provide a strategic advantage of a kind which doesn't negatively impact the losers of the race, e.g. it increases GDP by x10 and locks competitors out of having an ASI.
Then, losing control of the ASI could [not being able of] posing an existential risk to the US.
I think it's quite likely this is what some policymakers have in mind: some sort of innovation which will make everything better for the country by providing a lot cheap labor and generally improving productivity, the way we see AI applications do right now but on a bigger scale.
Comment on 3:
Not sure who your target audience is; I assume it would be policymakers, in which case I'm not sure how much weight that kind of argument has? I'm not a US citizen, but from international news I got the impression that current US officials would rather relish the option to undermine the liberal democracy they purport to defend.
ASI might provide a strategic advantage of a kind which doesn't negatively impact the losers of the race, e.g. it increases GDP by x10 and locks competitors out of having an ASI.
It does negatively impact the losers, to the extent that they're interested not only in absolute wealth but also relative wealth (which I expect to be the case, although I know ~nothing about SotA modeling of states as rational actors or whatever).
Very informative piece that does a lot in the right direction. Articles like this can have a real impact on policy demonstrating “there be dragons”.
A criticism would be that it doesn’t account for the state of the board in reality - the trust dilemma fails under circumstances where domestic commercial incentives overwhelm international cooperative concerns and collapses the situation to a prisoners dilemma, unfortunately, I think. I hope there are trust based solutions, and I’m mistaken.
I'm not sure about the trust dilemma analysis.
It seems to me like it switches between two levels of abstraction.
Cooperate-Cooperate may be more desirable for both states' citizens but at the same time Defect-Cooperate may be more desirable for state A as a state qua rational actor.
Curated and popular this week