Following my 3-week visit to Berkeley as described here and featured by ACX here, I told various people I would write in detail why I didn’t stick around to work in AI safety. In this post, I will dance around one object-level disagreement with AI safety and rationalists: why I no longer believe in explosive world transformation due to ASI. There will eventually be follow-up pieces on more object-level arguments and meta-level arguments against AI safety, but I prefer for this post to be mostly self-contained and not explicitly about AI safety.
When I worked in theoretical physics research, I was constantly reminded that if I were just that much smarter and had read that many more papers, I would be able to make the right connection to solve the problem. You are always the bottleneck; it’s always a skill issue. So if you’ve spent the bulk of your conscious life honing your ability to manipulate the pure abstraction of math and computer code, it is staggering to watch the problems you wrestled with for months or years get blown away in hours. You are left wondering how quickly all of the other problems in the world will be resolved - all of disease perhaps? Why not all of biology? Why not all the secrets of the universe?
In December 2025 when Claude Code + Opus 4.5 first made agents feel extremely useful, I was concerned about the job market being obliterated, so I took out “insurance” on rapid-onset white-collar unemployment. Thankfully, those predictions about increasingly capable agents causing rapid economic and labor transformation have been very wrong.
And so the question is, why? Why does the world largely look the same outside of highly verifiable domains like software engineering and math, and even in software engineering, why are there still millions employed? Why are models which can beat humans on nearly all benchmarks and are provided at unbelievably low prices not causing large economic shockwaves?
If I had to summarize my current perspective on AI in a single sentence, it would be that “the world is far too slow” to see the high double-digit or more economic growth and seismic transformations in the economy as predicted by variousessaysfromtherationalistblogo-instituto-sphere.
Amdahl’s Law
I suspect the reason I’ve felt less “capabilities whiplash” than my peers is that a lot of my experience is in experimental science where how smart you are is often not the actual bottleneck. To illustrate how enormous speedup in one component does not translate to enormous broader speedup due to other bottlenecks, let’s make a simple model:
If you have a process with some number of steps to be completed sequentially, each with rate i (ie each step takes time 1/to complete), your total rate is given by
If you send every rate but one to infinity, you are still left with
This is almost a trivial statement - your overall process is rate-limited by the slowest step, but it can be easily forgotten as you watch individual rates arc hyperexponentially upwards:
More realistically, suppose you have some hierarchy of rates
where r₀ = a is the slowest and each successive rate is b times faster than the previous. The total rate of this hierarchy is
If you send the first m bottleneck rates to infinity, entirely eliminating each bottleneck, the new rate is
The point is thatin hierarchically bottlenecked systems, even infinite speedup of the bottleneck results in a constant factor aggregate speedup. In order to see exponential aggregate speedup, you need explosive improvement at the current bottleneck over and over again. Overall progress is often brutally suppressed by this interplay between bottlenecks.
If during each time window you focus all of your efforts on accelerating the current bottleneck to ~infinity (or really just far beyond the next bottleneck or two), you see constant factor speedup. Repeated infinite individual factor speedup yields only exponential aggregate speedup.
To see diminishing returns, the intelligence curve need not flatten out; you just have to hit some other bottleneck. And so the question as it pertains to AI causing disruption in the world (for better and worse) is not “where is intelligence the rate-limiting step?” but “when intelligence becomes free, instant, and near-omniscient, what is the new rate-limiting step?” In computer science, this speedup analysis is known as Amdahl’s law and is used for understanding the limits of parallelism, but variants of it appear all across science.
Is Intelligence the Bottleneck?
I suspect the most common objection to the above analysis is that I am failing to imagine what ASI can accomplish. Instead of trying to abstractly rebut this, let’s look at a banal example of fairly bounded speedups even when you have access to ASI:
You’re the manager of a widget factory, and for some reason the defect rate has doubled in the past month. You’re able to use your process telemetry and slew of agents to figure out that tolerances during step 7 have blown out of the acceptable range. You haven’t changed anything about your process, so you have no idea what could be causing this. Four different precursors enter the process during step 7, but those haven’t changed either.
With the help of your ASI helper, you’re able to narrow down the possible causes of this specific kind of defect to inclusion of impurity X in material Y, but both precursors A and B contain Y. You reach out to the suppliers, but they shrug and say “sorry we haven’t changed anything”, so you need to diagnose which of these is the ultimate source. You order some tests on your new stock and the remnants of the old stock, and you find out that precursor A is causing your defect.
And now you have a decision to make: do you try to find a new supplier, remove the impurity yourself, or compensate for the impurity in other ways? You’ll need to run the tests on the new material or make a purification system or change your design. These all have tradeoffs and ASI will be able to help with all of these and pick the right one, but the fundamental problem here is that it takes time and money to install new systems to fix this problem. There is the time it takes to order new components, assemble them, install them. If you are integrating it into your telemetry system, you need all the sensors and wiring to feed into some database, and then finally you need to get your production up and running and make sure it’s all working.
The crux of AI disruption in my mind is these new bottlenecks which can be roughly split into two categories:
1. What are the speed limits in the world that intelligence does not accelerate?
The purely physical speed limits in the above example are things like sending out complicated tests to be run, ordering new components, installing new hardware into your factory, and manufacturing new widgets to verify your problem has been solved. There is a practical limit to how fast you can ship things across the country, and manufacturing steps involving chemical and biological processes have fundamental rates that can be on the order of days for things like culturing human cells and annealing metals.
Now suppose you also have humanoid robots to run your factory and install the new components, handling every step along the way. A very nontrivial percentage of the time in the above example is just dealing with the other companies, sending out materials for testing, having the new parts shipped to you, and installing the physical hardware and stress-testing it. This whole process could take a month in a current factory without superintelligence, but getting a 4x speedup to ~1 week seems about as fast as you can possibly go even with omniscient machines making perfect decisions at every step and working 24/7. At some point you’re quite literally bottlenecked on shipping time and physical throughput rate of tools in your factory.
There are also problems which are more or less solved - the steam engine has been around for 300 years, yet even fusion plants will likely end up turning a steam turbine. How many other problems in the world are in some sense solved and are very slow, expensive, and boring to improve? Other speed limits in the world are less physical but still important. What is the speed of regulation? What is the speed of political will?
It is worth disambiguating speedup and scale-up here. In something like the oil industry, a multitrillion dollar industry with enormous incentive to rapidly automate, it seems even a 3x speedup would be difficult. The oil comes out of the ground at some rate, is refined at some rate, and is transported at some rate, so the obvious answer if you want to double production is to parallelize everything and just have twice as much infrastructure. But massively expanding capacity in parallelization is itself a process that can be bottlenecked. Over the last year, strange scaling bottlenecks have looked like the monocrystalline turbine blades for natural gas plants and indium phosphide wafers for optical interconnects, and this has just been for running computers. If you believe in true industrial explosions, the demand for niche magnets and bearings and sensors and tool tips and alloys will explode, and any one of them can become the rate limiter.
2. What does it actually mean to be superintelligent in the presence of noise?
The more abstract speedup question is something like “could a superintelligence, given all the context of the situation, know right away that impurity X was the culprit and that it came from precursor A?” This would save you the time you spent running tests on A versus B and let you go straight to problem solving, but is this even possible? After all, the only thing that changed was the error bar in step 7.
In the aphorism “intelligence is compression”, there is a natural next question: What is the maximum amount of compression you can achieve in the real world? In order to realize that precursor A and not B had the impurity, you need to acquire 1 bit of information. Some bits, however, are unbelievably expensive to obtain and create a similar kind of Amdahl calculus. If key bits are orders of magnitude more expensive and slower to acquire (say, for example, long-term side effect analysis of new drugs on actual humans), these will set your overall rate.
It seems most people in AI consider the world to be far less noisy than it actually is because they mostly live in a world of flawlessly repeatable execution at staggering scale. Scaling experiments is as simple as spinning up more Docker containers, but the notion of superintelligence in domains like biology is ill-defined: There are large variations between nominally identical experiments, and the rate at which experiments produce informative bits is unfathomably low compared to anything in ML or verifiable like math. In other words, even information theory-optimal exploration of some problem landscape could be agonizingly slow if experiments are slow or noisy, let alone both. (To those who believe AI would be able to simulate any process accurately enough, it’s worth appreciating how poorly understood many multi-scale interactions are and how ground truth experimentation is often your best option. See also my blog post from April.)
Outlook
Intelligence is far from the only technology curve to go quasi-infinite in the last century and a half: The world electrified. The cost of communication has gone from couriers and weeks to ~free and gigabytes per second. Computers have made calculations which were once infeasible executable on a cheap computer near-instantly. Transportation costs have fallen by an order of magnitude. From 1880 to 2026, the share of the US workforce in agriculture fell from 51% to 1.2% as we were freed to work in other fields.
I think the widget story changes quite a bit if one takes the assumption of cheap superhuman intelligence and tries to propagate consequences a bit further.
The owners of the factories producing A and B also have their own AI assistants.
These assistants negotiated the supply contracts with your assistant. Because the AIs can work quickly and cheaply, they invested effort into writing a long contract that tightly specifies what your widget production process requires. Your AI understands the process well, and was able to put nearly all relevant considerations into the contract.
One clause requires a bound on the concentration of impurity X in material Y.
A new factory near supplier A's factory finishes construction. It releases small particles of X into the air as a byproduct of its activities. This happens not to be prohibited by environmental regulations. (In a world of cheap intelligence, perhaps this is because X is harmless to living things, and only inconvenient for certain delicate technological processes.)
A subagent belonging to supplier A has been tracking the construction progress of that factory. It notes that 5% of supplier A's customers have provisions in their contracts regarding the concentration of impurity X, and that factories of such a kind often produce particles of X. A quick email exchange with an AI assistant associated with that factory confirms that this is that case.
Supplier A upgrades their air filters so that they are capable of filtering out impurity X. This is accomplished before the neighbouring factory even starts up production.
The USA's surprisingly stable 2% growth per capita could be due to presence of other bottlenecks like governance issues or educating new generations. I would advise you to study the growth rates of countries like Japan and the USSR which didn't demonstrate a consistent 2% growth per capita.
Following my 3-week visit to Berkeley as described here and featured by ACX here, I told various people I would write in detail why I didn’t stick around to work in AI safety. In this post, I will dance around one object-level disagreement with AI safety and rationalists: why I no longer believe in explosive world transformation due to ASI. There will eventually be follow-up pieces on more object-level arguments and meta-level arguments against AI safety, but I prefer for this post to be mostly self-contained and not explicitly about AI safety.
When I worked in theoretical physics research, I was constantly reminded that if I were just that much smarter and had read that many more papers, I would be able to make the right connection to solve the problem. You are always the bottleneck; it’s always a skill issue. So if you’ve spent the bulk of your conscious life honing your ability to manipulate the pure abstraction of math and computer code, it is staggering to watch the problems you wrestled with for months or years get blown away in hours. You are left wondering how quickly all of the other problems in the world will be resolved - all of disease perhaps? Why not all of biology? Why not all the secrets of the universe?
In December 2025 when Claude Code + Opus 4.5 first made agents feel extremely useful, I was concerned about the job market being obliterated, so I took out “insurance” on rapid-onset white-collar unemployment. Thankfully, those predictions about increasingly capable agents causing rapid economic and labor transformation have been very wrong.
And so the question is, why? Why does the world largely look the same outside of highly verifiable domains like software engineering and math, and even in software engineering, why are there still millions employed? Why are models which can beat humans on nearly all benchmarks and are provided at unbelievably low prices not causing large economic shockwaves?
If I had to summarize my current perspective on AI in a single sentence, it would be that “the world is far too slow” to see the high double-digit or more economic growth and seismic transformations in the economy as predicted by various essays from the rationalist blogo-instituto-sphere.
Amdahl’s Law
I suspect the reason I’ve felt less “capabilities whiplash” than my peers is that a lot of my experience is in experimental science where how smart you are is often not the actual bottleneck. To illustrate how enormous speedup in one component does not translate to enormous broader speedup due to other bottlenecks, let’s make a simple model:
If you have a process with some number of steps to be completed sequentially, each with rate
If you send every rate but one to infinity, you are still left with
This is almost a trivial statement - your overall process is rate-limited by the slowest step, but it can be easily forgotten as you watch individual rates arc hyperexponentially upwards:
From the new METR report
More realistically, suppose you have some hierarchy of rates
where r₀ = a is the slowest and each successive rate is b times faster than the previous. The total rate of this hierarchy is
If you send the first m bottleneck rates to infinity, entirely eliminating each bottleneck, the new rate is
The point is that in hierarchically bottlenecked systems, even infinite speedup of the bottleneck results in a constant factor aggregate speedup. In order to see exponential aggregate speedup, you need explosive improvement at the current bottleneck over and over again. Overall progress is often brutally suppressed by this interplay between bottlenecks.
If during each time window you focus all of your efforts on accelerating the current bottleneck to ~infinity (or really just far beyond the next bottleneck or two), you see constant factor speedup. Repeated infinite individual factor speedup yields only exponential aggregate speedup.
To see diminishing returns, the intelligence curve need not flatten out; you just have to hit some other bottleneck. And so the question as it pertains to AI causing disruption in the world (for better and worse) is not “where is intelligence the rate-limiting step?” but “when intelligence becomes free, instant, and near-omniscient, what is the new rate-limiting step?” In computer science, this speedup analysis is known as Amdahl’s law and is used for understanding the limits of parallelism, but variants of it appear all across science.
Is Intelligence the Bottleneck?
I suspect the most common objection to the above analysis is that I am failing to imagine what ASI can accomplish. Instead of trying to abstractly rebut this, let’s look at a banal example of fairly bounded speedups even when you have access to ASI:
You’re the manager of a widget factory, and for some reason the defect rate has doubled in the past month. You’re able to use your process telemetry and slew of agents to figure out that tolerances during step 7 have blown out of the acceptable range. You haven’t changed anything about your process, so you have no idea what could be causing this. Four different precursors enter the process during step 7, but those haven’t changed either.
With the help of your ASI helper, you’re able to narrow down the possible causes of this specific kind of defect to inclusion of impurity X in material Y, but both precursors A and B contain Y. You reach out to the suppliers, but they shrug and say “sorry we haven’t changed anything”, so you need to diagnose which of these is the ultimate source. You order some tests on your new stock and the remnants of the old stock, and you find out that precursor A is causing your defect.
And now you have a decision to make: do you try to find a new supplier, remove the impurity yourself, or compensate for the impurity in other ways? You’ll need to run the tests on the new material or make a purification system or change your design. These all have tradeoffs and ASI will be able to help with all of these and pick the right one, but the fundamental problem here is that it takes time and money to install new systems to fix this problem. There is the time it takes to order new components, assemble them, install them. If you are integrating it into your telemetry system, you need all the sensors and wiring to feed into some database, and then finally you need to get your production up and running and make sure it’s all working.
The crux of AI disruption in my mind is these new bottlenecks which can be roughly split into two categories:
1. What are the speed limits in the world that intelligence does not accelerate?
The purely physical speed limits in the above example are things like sending out complicated tests to be run, ordering new components, installing new hardware into your factory, and manufacturing new widgets to verify your problem has been solved. There is a practical limit to how fast you can ship things across the country, and manufacturing steps involving chemical and biological processes have fundamental rates that can be on the order of days for things like culturing human cells and annealing metals.
Now suppose you also have humanoid robots to run your factory and install the new components, handling every step along the way. A very nontrivial percentage of the time in the above example is just dealing with the other companies, sending out materials for testing, having the new parts shipped to you, and installing the physical hardware and stress-testing it. This whole process could take a month in a current factory without superintelligence, but getting a 4x speedup to ~1 week seems about as fast as you can possibly go even with omniscient machines making perfect decisions at every step and working 24/7. At some point you’re quite literally bottlenecked on shipping time and physical throughput rate of tools in your factory.
There are also problems which are more or less solved - the steam engine has been around for 300 years, yet even fusion plants will likely end up turning a steam turbine. How many other problems in the world are in some sense solved and are very slow, expensive, and boring to improve? Other speed limits in the world are less physical but still important. What is the speed of regulation? What is the speed of political will?
It is worth disambiguating speedup and scale-up here. In something like the oil industry, a multitrillion dollar industry with enormous incentive to rapidly automate, it seems even a 3x speedup would be difficult. The oil comes out of the ground at some rate, is refined at some rate, and is transported at some rate, so the obvious answer if you want to double production is to parallelize everything and just have twice as much infrastructure. But massively expanding capacity in parallelization is itself a process that can be bottlenecked. Over the last year, strange scaling bottlenecks have looked like the monocrystalline turbine blades for natural gas plants and indium phosphide wafers for optical interconnects, and this has just been for running computers. If you believe in true industrial explosions, the demand for niche magnets and bearings and sensors and tool tips and alloys will explode, and any one of them can become the rate limiter.
2. What does it actually mean to be superintelligent in the presence of noise?
The more abstract speedup question is something like “could a superintelligence, given all the context of the situation, know right away that impurity X was the culprit and that it came from precursor A?” This would save you the time you spent running tests on A versus B and let you go straight to problem solving, but is this even possible? After all, the only thing that changed was the error bar in step 7.
In the aphorism “intelligence is compression”, there is a natural next question: What is the maximum amount of compression you can achieve in the real world? In order to realize that precursor A and not B had the impurity, you need to acquire 1 bit of information. Some bits, however, are unbelievably expensive to obtain and create a similar kind of Amdahl calculus. If key bits are orders of magnitude more expensive and slower to acquire (say, for example, long-term side effect analysis of new drugs on actual humans), these will set your overall rate.
It seems most people in AI consider the world to be far less noisy than it actually is because they mostly live in a world of flawlessly repeatable execution at staggering scale. Scaling experiments is as simple as spinning up more Docker containers, but the notion of superintelligence in domains like biology is ill-defined: There are large variations between nominally identical experiments, and the rate at which experiments produce informative bits is unfathomably low compared to anything in ML or verifiable like math. In other words, even information theory-optimal exploration of some problem landscape could be agonizingly slow if experiments are slow or noisy, let alone both. (To those who believe AI would be able to simulate any process accurately enough, it’s worth appreciating how poorly understood many multi-scale interactions are and how ground truth experimentation is often your best option. See also my blog post from April.)
Outlook
Intelligence is far from the only technology curve to go quasi-infinite in the last century and a half: The world electrified. The cost of communication has gone from couriers and weeks to ~free and gigabytes per second. Computers have made calculations which were once infeasible executable on a cheap computer near-instantly. Transportation costs have fallen by an order of magnitude. From 1880 to 2026, the share of the US workforce in agriculture fell from 51% to 1.2% as we were freed to work in other fields.
Yet through it all, we’ve seen nearly 150 years of 2% GDP per capita growth. These are all general-purpose technologies, so what needs to give for ASI to be that different?