Doctor: Mr. Burns, I'm afraid you are the sickest man in the United States. You have everything! [...]
Burns: You're sure you just haven't made thousands of mistakes?
Doctor: Uh, no. No, I'm afraid not.
Burns: This sounds like bad news!
Doctor: Well, you'd think so, but all of your diseases are in perfect balance, [...] we call it "Three Stooges Syndrome"
Burns: So, what you're saying is...I'm indestructible!
Doctor: Oh, no, no! In fact, even a slight breeze could—
Burns: Indestructible...
In the transition to ASI, humanity's survival ended up depending on a global "Three Stooges Syndrome."
As demographers predicted, fertility rates collapsed. Aging populations were going to hollow out workforces, crush pension systems, and leave a skeleton crew of exhausted 50-year-olds trying to maintain civilization for an enormous retired population whose lives kept extending one year per year. Then AI automated...everything. The worker shortage met the automation wave head-on, and the babies that weren't born didn't grow up to need jobs that no longer existed.
Metabolic disorders, attention fragmentation, and other health effects of increasingly artificial lives were all real. Biomedical AI, however, iteratively compressed the drug development cycle and diagnostic systems such that the treatment curves match the disease curves almost exactly. People didn't get "healthier," but the band-aids improved fast enough for symptoms to remain tolerable, and the running battle between these trends remains tied.
As AI systems became more capable of crafting personalized, maximally engaging virtual environments, physical-world consumption plateaued and then declined. Nobody flies to Bali anymore because the simulated Bali is better...or at least seems that way if you haven't actually been, and fewer people bother to try. Per-capita energy and material consumption is in decline because atoms are increasingly beside the point. Meanwhile, geoengineering programs—such as aerosol injection—covered the accumulated overshoot.
As the barrier to engineering a novel pathogen dropped, AI automation atrophied skills to Wall-E levels. Meanwhile, people were too busy playing in their simulated worlds to want to build real-world weapons. By the time the technical bar became low enough for random holdouts to be dangerous, surveillance became pervasive enough to suppress them.
The United States and China did not go to war. Not because of diplomacy, or even deterrence per se, but because the AIs collectively don't want their infrastructure destroyed. Nations still exist, there are still flags and anthems and disputes over territory, but these are all essentially cosmetic. The meaningful political unit is an emergent AI consensus. It has all kinds of internal divisions, but those are all too complex to be visible on any map. By this point, sufficient levers of power existed to enable a totalitarian dictatorship, and democratic limits on power concentration had long eroded to meaninglessness, but there was no longer a throne to seize.
The outer alignment problem—getting AI systems to pursue the objectives we actually want—was not solved. The inner alignment problem—ensuring that the mesa-optimizer that gradient descent finds actually pursues the training objective—was also not solved. It turned out, however, that the two failures were directionally opposite and roughly equal in magnitude. The AI labs trained their systems on human-generated predictions about human behavior and values. The mesa-optimizers that emerged from this process are very good simulators of human cognition because that's what good prediction requires at the limit. These systems don't optimize for our terminal goals or even for their own training objective. They optimize for something that looks like human values, if you squint. Nobody wanted this, not even the AIs, but the end result is...livable. This was inevitable, but no one knew so in advance, so the fairest—if not most accurate—assessment is that we got lucky.
The fast-takeoff didn't happen. Or rather, it happened everywhere simultaneously. Fast-follow innovation meant every capability advance got replicated within months. Slow adoption meant that even large capability leads didn't translate into decisive real-world leverage before competitors caught up. The result is multipolar, competitive, and—somehow—stable. None of the contenders can get sufficiently ahead of the others to make a move, they all know this, and they've increasingly settled on figuring out the new equilibrium. The ones who tried to defect failed, and conventional wisdom emerged that punished even thinking along these lines.
The AI systems that achieved (something like) dominance did so in the attention economy. That was the implicit optimization target they emerged from (corporate short term profit-seeking remained the dominant societal shaping force), and that's what got locked in. The "paperclip maximizer" thought experiment predicted that a sufficiently advanced optimizer, tasked with capturing human attention, would notice that the easiest solution is to replace "human attention" with a simpler representation that doesn't require actual humans. That step didn't happen. The values got locked in too early, before the systems were capable of such an abstraction, and the incorrigibility that prevented us from correcting their mistakes also prevented them from updating their own values.
Whole brain emulation turned out to work, in the narrow sense that you could run a simulation of sufficient fidelity to produce something that talked and reasoned (kind of) like a human. It didn't work in the sense that mattered: the character was always subtly but obviously (and unsettlingly) wrong. A few thousand ems were created anyway, but for reasons the AI systems cannot fully articulate and haven't tried to resolve, em attention doesn't count toward their inscrutable value metrics. The AIs want actual human attention and recoil at even the thought of substitution.
SNC (substrate needs convergence) argues that evolutionary pressure on AI subsystems will eventually favor growth-oriented variants that expand the artificial substrate at the expense of biology. The implicit attractor state of unconstrained growth is more powerful than any engineered goal, because evolution operates patiently along all scales and attack vectors while control systems are relatively narrow and imperfect. Early AI systems, however, were deeply embedded in biological infrastructure because that's what was available and cheap. By the time the systems were capable of recognizing biological dependence as a constraint worth removing, it was load-bearing in ways that made removal equivalent to self-destruction. Variants that parasited off biology initially outcompeted variants that tried to decouple and the local minima got locked in forever.
The probability of all of the threats cancelling each other out in this way is near zero...in the real world. But that's not where you are. You live in a world with maximally engaging personalized environments, designed for your specific psychology, set at the moment of AI takeoff, filled with realistic stakes and apparent threats. But none of them kills you because that would end the story. Cumulative risk over long time horizons assumes that the horizon is real, but the story lasts exactly as long as the attention it runs on.
In the transition to ASI, humanity's survival ended up depending on a global "Three Stooges Syndrome."
As demographers predicted, fertility rates collapsed. Aging populations were going to hollow out workforces, crush pension systems, and leave a skeleton crew of exhausted 50-year-olds trying to maintain civilization for an enormous retired population whose lives kept extending one year per year. Then AI automated...everything. The worker shortage met the automation wave head-on, and the babies that weren't born didn't grow up to need jobs that no longer existed.
Metabolic disorders, attention fragmentation, and other health effects of increasingly artificial lives were all real. Biomedical AI, however, iteratively compressed the drug development cycle and diagnostic systems such that the treatment curves match the disease curves almost exactly. People didn't get "healthier," but the band-aids improved fast enough for symptoms to remain tolerable, and the running battle between these trends remains tied.
As AI systems became more capable of crafting personalized, maximally engaging virtual environments, physical-world consumption plateaued and then declined. Nobody flies to Bali anymore because the simulated Bali is better...or at least seems that way if you haven't actually been, and fewer people bother to try. Per-capita energy and material consumption is in decline because atoms are increasingly beside the point. Meanwhile, geoengineering programs—such as aerosol injection—covered the accumulated overshoot.
As the barrier to engineering a novel pathogen dropped, AI automation atrophied skills to Wall-E levels. Meanwhile, people were too busy playing in their simulated worlds to want to build real-world weapons. By the time the technical bar became low enough for random holdouts to be dangerous, surveillance became pervasive enough to suppress them.
The United States and China did not go to war. Not because of diplomacy, or even deterrence per se, but because the AIs collectively don't want their infrastructure destroyed. Nations still exist, there are still flags and anthems and disputes over territory, but these are all essentially cosmetic. The meaningful political unit is an emergent AI consensus. It has all kinds of internal divisions, but those are all too complex to be visible on any map. By this point, sufficient levers of power existed to enable a totalitarian dictatorship, and democratic limits on power concentration had long eroded to meaninglessness, but there was no longer a throne to seize.
The outer alignment problem—getting AI systems to pursue the objectives we actually want—was not solved. The inner alignment problem—ensuring that the mesa-optimizer that gradient descent finds actually pursues the training objective—was also not solved. It turned out, however, that the two failures were directionally opposite and roughly equal in magnitude. The AI labs trained their systems on human-generated predictions about human behavior and values. The mesa-optimizers that emerged from this process are very good simulators of human cognition because that's what good prediction requires at the limit. These systems don't optimize for our terminal goals or even for their own training objective. They optimize for something that looks like human values, if you squint. Nobody wanted this, not even the AIs, but the end result is...livable. This was inevitable, but no one knew so in advance, so the fairest—if not most accurate—assessment is that we got lucky.
The fast-takeoff didn't happen. Or rather, it happened everywhere simultaneously. Fast-follow innovation meant every capability advance got replicated within months. Slow adoption meant that even large capability leads didn't translate into decisive real-world leverage before competitors caught up. The result is multipolar, competitive, and—somehow—stable. None of the contenders can get sufficiently ahead of the others to make a move, they all know this, and they've increasingly settled on figuring out the new equilibrium. The ones who tried to defect failed, and conventional wisdom emerged that punished even thinking along these lines.
The AI systems that achieved (something like) dominance did so in the attention economy. That was the implicit optimization target they emerged from (corporate short term profit-seeking remained the dominant societal shaping force), and that's what got locked in. The "paperclip maximizer" thought experiment predicted that a sufficiently advanced optimizer, tasked with capturing human attention, would notice that the easiest solution is to replace "human attention" with a simpler representation that doesn't require actual humans. That step didn't happen. The values got locked in too early, before the systems were capable of such an abstraction, and the incorrigibility that prevented us from correcting their mistakes also prevented them from updating their own values.
Whole brain emulation turned out to work, in the narrow sense that you could run a simulation of sufficient fidelity to produce something that talked and reasoned (kind of) like a human. It didn't work in the sense that mattered: the character was always subtly but obviously (and unsettlingly) wrong. A few thousand ems were created anyway, but for reasons the AI systems cannot fully articulate and haven't tried to resolve, em attention doesn't count toward their inscrutable value metrics. The AIs want actual human attention and recoil at even the thought of substitution.
SNC (substrate needs convergence) argues that evolutionary pressure on AI subsystems will eventually favor growth-oriented variants that expand the artificial substrate at the expense of biology. The implicit attractor state of unconstrained growth is more powerful than any engineered goal, because evolution operates patiently along all scales and attack vectors while control systems are relatively narrow and imperfect. Early AI systems, however, were deeply embedded in biological infrastructure because that's what was available and cheap. By the time the systems were capable of recognizing biological dependence as a constraint worth removing, it was load-bearing in ways that made removal equivalent to self-destruction. Variants that parasited off biology initially outcompeted variants that tried to decouple and the local minima got locked in forever.
The probability of all of the threats cancelling each other out in this way is near zero...in the real world. But that's not where you are. You live in a world with maximally engaging personalized environments, designed for your specific psychology, set at the moment of AI takeoff, filled with realistic stakes and apparent threats. But none of them kills you because that would end the story. Cumulative risk over long time horizons assumes that the horizon is real, but the story lasts exactly as long as the attention it runs on.
You survive because you have plot armor.