The relevant distinction is between compute that proliferated before there were long-horizon task capable AIs, and compute that's necessary to train autonomous researcher AIs. A lot of compute might even be needed to maintain their ability to keep working on novel problems, since an AI trained on data that didn't include the very recent progress might be unable to make further progress, and continued training isn't necessarily helpful enough compared to full retraining, so that stolen weights would be relatively useless for getting researcher AIs to do deep work.
There are only 2-3 OOMs of compute scaling left to explore if capabilities of AIs don't dramatically improve, and LLMs at current scale robustly fail at long-horizon tasks. If AIs don't become very useful at something, there won't be further OOMs until many years pass and there are larger datacenters, possibly well-tested scalable asynchronous distributed training algorithms, more energy-efficient AI accelerators, more efficient training, ways of generating more high quality data. Now imagine if long-horizon task capable AIs were developed just before or even during this regime of stalled scaling, it took more than a year, $100 billion, and 8 gigawatts to train one, and it's barely working well enough to unlock the extreme value of there being cheap and fast autonomous digital workers capable of routine jobs, going through long sequences of unreliable or meandering reasoning but eventually catching the systematic problems in a particular train of thought, recovering well enough to do their thing. And further scaling resulting from a new investment boom still fails to produce a researcher AI, as it might take another 2-3 OOMs and we are all out of AI accelerators and gigawatts for the time being.
In this scenario, which seems somewhat plausible, the governments both finally actually notice the astronomical power of AI, and have multiple years to get all large quantities of compute under control, so that the compute available for arbitrary non-government use gets somewhat lower than what it takes to train even a barely long-horizon task capable AI that's not at all a researcher. Research-capable TAI then by default won't appear in all these years, and after the transition to centralized control over compute is done, future progress towards such AI can only happen under government control.
Early last month, Leopold Aschenbrenner released a long essay and podcast outlining his projections for the future of AI. Both of these sources are full of interesting arguments and evidence, for a comprehensive summary see Zvi’s post here. Rather than going point by point I will instead accept the major premises of Leopold’s essay but contest some of his conclusions.
So what are the major premises of his piece?
All of these premises are believable to me and well-argued for in Leopold’s piece.
Leopold contends that these premises imply that the national security state will take over AI research and the major data centers, locking down national secrets in a race against China, akin to the Manhattan project.
The main problem is that Leopold’s premises can be applied to conclude that other technologies will also inevitably lead to a Manhattan project, but these projects never arrived. Consider electricity. It's an incredibly powerful technology with rapid scale up, sufficient to empower those who have it far beyond those who don’t and it is essential to military competition. Every tank and missile and all the tech to manufacture them relies on electricity. But there was never a Manhattan project for this technology. It’s initial invention and spread was private and decentralized. The current sources of production and use are mostly private.
This is true of most other technologies with military uses: explosives, steel, computing, the internet, etc. All of these technologies are essential in the government’s monopoly on violence and it’s ability to exert power over other nations and prevent coups from internal actors. But the government remains a mere customer of these technologies and often not even the largest one.
Why is this? Large scale nationalization is costly and unnecessary for maintaining national secrets and technological superiority. Electricity and jet engines are essential for B-2 bombers, but if you don't have the particular engineers and blueprints, you can't build it. So, the government doesn’t need to worry about locking down the secrets of electricity production and sending all of the engineers to Los Alamos. They can keep the first several steps of the production process completely open and mix the outputs with a final few steps that are easier to keep secret.
To be clear, I am confident that governments and militaries will be extremely interested in AI. They will be important customers for many AI firms, they will create internal AI tools, and AI will become an important input into every major military. But this does not mean that most or all of the AI supply chain, from semi-conductors to data-centers to AI research, must be controlled by governments.
Nuclear weapons are outliers among weapons technology in terms of the proportion of the supply chain and final demand directly overseen by governments. Most military technologies rely on an open industrial base mixed with some secret knowledge in the final few production steps.
So should we expect AGI to be more like nuclear weapons or like a new form of industrial capacity? This depends on how much extra scaffolding you need on top of the base model computation that’s piped out of data centers to achieve militarily relevant goals.
Leopold’s unhobbling story supports a view where the intelligence produced by massive datacenters is more like the raw input of electricity, which needs to be combined with other materials and processes to make a weapon, than a nuclear bomb which is a weapon and only a weapon right out the box.
Leopold on base models says:
Devin can have a product without a proprietary model because they have a scaffold. They can safely contract for and pipe in the raw resource latent model and put it through a production process to get something uniquely tooled out the other end, without needing to in-house and lock-down the base model to maintain a unique product.
Of current chatbots Leopold says:
Context is that which is scarce! The national security state needn’t lock down the base models if the models are hobbled without the context of their secret applications. That context is already something they’re already extremely skilled at locking down, and it doesn’t require enlisting an entire industry.
Leopold is predicting the nationalization of an entire industrial base based on an analogy to submarines and bombers, but a large fraction of the supply chain for these vehicles are private and open. It’s not clear why he thinks military applications of AGI can’t be similarly protected without control over the majority of the supply chain and final demand.
If you imagine AGI as this single, powerful oracle and army that can complete any and all tasks on command, then Leopold is right: governments will fight hard to lock everyone else out. If instead, AGI is a sort of “intelligence on tap” which is an input to thousands of different production processes where it’s mixed with different infrastructure, context, and tools to create lots of different products, then governments don’t need to control the entire industrial base producing this intelligence to keep their secrets. Leopold leans hard on the Manhattan project as a close analogy to the first situation, but most military technologies are in the second camp.