My biggest problem with AI 2027 is I don't think it is science-fictional enough. That is, the end of the scenario seems optimized for respectability over accuracy - here I refer to the "special economic zones" and "robot economy" parts. Their industrial explosion assumes human-scale robots will be building robot factories to build more human-scale robotics factories. This is a respectable assumption and one that is fun to model, but seems likely false to me.

I applaud Kokotajlo and his team for making it more legible that AI corps will be vertically integrating to the point they can manufacture their own customers. But they [their mainline narrative] almost entirely elide[s] that there are extreme incentives to miniaturize all the way to the physical limits of miniaturization, that there is immense power in making the unit of self-replication as small as possible, and that the construction of a self-replicator dependent only on environmental inputs (rather than inputs from the human economy) is likely to be something of a discontinuity.

Even if we make the fashionable choice of pretending synthetic biology and nanotechnology won't be the first path to sovereign infrastructure, the unit of replication is unlikely to be a conventional factory or anything close to a conventional factory. It's going to be the smallest sufficiently useful replicator the "army of geniuses" can construct. To put it crudely and insultingly, if we arbitrarily pretend biological and nanotechnological infrastructure will be impossible, we should at least model the unit of self-replication as being as small as critics allow it to be before they arbitrarily dismiss it as impossible by virtue of being the product of biotech or nanotechnology.

But why should we ignore biology and nanotechnology again? It's commonly said it's just really, really hard. It's worth making explicit that this is a claim that millions of superintelligences running tens or hundreds of times faster than human brains will be incapable of getting purchase in these domains for years or decades. This is a very strong claim!

Often, people justify this by pointing to limits to the speed of running experiments and collecting data. And there are limits there but they don't look very limiting. We can imagine automated bio labs focused on designing sensors and actuators that are partially alive, modified multicellular life - living microscopes, little empirical fingers granting a window into the realm of the very small, that have the advantage of being extremely cheap to replicate. All of this as a supplement to advances in conventional microscopy and the sheer manufacturing capacity to construct vastly more of what we already have.

This should allow experiments at a massive scale and parallelism, collecting data that is used to perfect simulations and autoregressive models trained on same.

You can postulate enormous serial constraints. But is that a conservative assumption? The greater the degree of miniaturization, the faster the empirical feedback loop. And we should expect great progress in simulations, as I mentioned. Importantly, one can stick to the part of the design space that is both useful and accurately simulated by the simulators one has.

2027's assumption of human-level robotics implies automated wet labs unconstrained by shortages of skilled human labour - both physical and intellectual. This implies much faster progress in biotechnology which, in turn, implies synthetic organisms (including multicellular organisms) that can, themselves, accelerate biotechnology. And if Drexler-style nanotechnology is possible, advanced biotechnology will be very useful in getting there.

There is also the spectre of quantum computers, which from what I understand are likely to be extremely useful for certain physical calculations, including some of use in biology, nanotechnology, and improving conventional molecular simulations. Given we seem to be expecting quantum computers in the 2030s even absent AGI, we should assume ASIs will be able to construct them.

I read AI 2027 as being partially a political document, less so than 2040 but still heavily constrained by political considerations. So maybe this is an intended compromise. But to the extent it is treated as pure prediction, I expect it will lead many readers to be predictably surprised by self-replicators, macroscopic and otherwise.