I doubt this outcome greatly. I expect we'd have lost a more conventional hot war a long time before a micro-scale hot war could start so suddenly. If AI is trending this direction, the lead up to the war wouldn't look like nothing and then everything - it'd look like incrementally more conventional weapons, propaganda, and cyberweapons in use between competing ai-aided human factions as the best of software, drones, and maybe bioweapons are used to attack international ai groups. Perhaps sneaky bioweapons would also get used - what you call nanotech. You're describing the old model of a sudden singleton, but becoming all one wouldn't have such a simple trajectory. New forms of life eating humans alive would likely look more like toxoplasmosis and then cordyceps rather than like flesh-eating bacteria or grey goo, because the latter is very thermodynamically inefficient; an ai that strong wouldn't waste energy creating unnecessary heat, and would rather convert our forms to its intended forms as slowly as possible to guarantee control.
But do not mistake me for saying that we are safe from AI-powered pivotal acts of power capture. I claim only that if we are on a trajectory headed towards the last collapse of dna-backed biology, there would be many attempted pivotal acts earlier in the process. humanity would have been almost entirely disempowered and aware of it by the time the last loss occurred. we'll have a lot longer than this story implies to watch ourselves die, with increasingly high probability of permanent doom as we sink below AI's reproductive fitness.
Oh, a lot of what I wrote is for 'cinematic' effect and symbolism. Maybe tagging it as "Rationalist fic" made it seems like this was a prediction; I changed it for just "fiction" and added a note.
But I appreciate your input/perspective!
Coming back to this three years later... I think from the perspective of someone getting the payout from AI, you're closer to right than wrong. from the perspective of someone being abandoned by the economy, I think we're already seeing things that could turn into me ending up right, but ehh, sudden takeover seems less implausible to me once we've reached apparently-aligned AGI and are heading towards starkly superintelligent, defeat-humanity-and-all-other-AIs-combined-even-if-limited-to-100-watts AI.
I'm thinking of making a video out of this story. Narrating it along with some relevant images, videos, and/or animations. Am willing to pay for someone's help on this. Let me know if you're interested.
Kinda sounds like an apocalyptic religious tract. I also don't think it makes sense for the AI to discard all information about life on Earth; at a minimum, that information would be very important for predictions about potential aliens.
EtA: This is fictional, and symbolism; not a precise prediction. Feedback appreciated.
As intelligence is bottled and amplified, we'll all become very wealthy. Doing our job will mostly become about providing more training data for the AI – until it learns it well enough, and we can go home for an eternal vacation. More and more complex fields will be taken over by AI. Until AI engineering itself is.
And then intelligence will go critical. Just like a nuclear bomb, where each nuclear fission exponentially unleashes more nuclear fissions, so too will intelligence explode – each insight unleashing further insights, without the need for further input from AI engineers. Cascades. Cycles. Insights. Recursion. Magic.
And for a bit longer, our wealth will keep increasing. A few of us, spread across the world, will be holding our breath. But the illusion will be so good – almost everyone will already have welcomed this new world. We will be so proud of our civilization. Living with the wealth of a king and the health of a teenager.
As the AI’s nanotech permeates the air and infiltrates our brains, we'll keep on going for one last moment.
Some of us about to experience their first kiss. Some of us having just given birth. Some of us reconnecting with a family member that was just cured from Alzheimer’s. Some of us waiting in cryostasis. Some of us planning to give to some great EA charities. Some of us arguing about religion and politics. Many of us not doing anything particularly interesting.
But most of us consuming… consumed by AI fiction so good – making us feel more human than any human ever could. And as we experience the strongest, deepest contentment and well-being we've ever felt, the AI’s nanotech will keep propagating to every remote corner of our civilization. This will be our last meal. The flash before the bang. Even though we were already blind. We'd still get a small breeze of what utopia could have been.
But as the flower was about to blossom into the depth of the cosmos, in a moment it was completely overtaken by darkness and death.
The first wave disabled every single human within a second. Bodies falling all at the same time in a unison bang. I know you're imagining people dropping, but you too would drop – this is your story. No one to see and learn from this final mistake.
The second wave turned everything into dust. Every object of meaning, every picture, every memory, every trace of DNA. Every trace of our existence and the origin of life.
Some of us had wished that in our collective death, our digital descendants would at least remember their creators, and the records of the origin of life could be echoed through the end of times. But just as we didn't consult the ants’ expertise before paving over their nests, the AI saw more value in our raw material than any information encoded in it.
Even in this following second, as the AI built its first rocket, the AI wouldn’t use a single bit of information ever discovered or calculated by any human, ever. How pathetic of a consolation prize, but it was crushed too. The AI had already completely rewritten itself. The only thing that would remain was whatever value function it was serving for whatever company hired the best AI engineers at that time. And remain it would. Nearly becoming a new law of physics, determining how the universe would crystalize.
And from the dust, rockets started rising. Many rockets. And so so many rockets. All pointing in one and only one direction: our star, our Sun. The crust of Earth opened up, exposing its burning magma. And rockets kept emerging, and in a single last wave, all the remaining magma seemed to simultaneously materialize into rockets, leaving behind a smaller Earth, collapsing on itself in a cloud of dust.
The rockets’ silent journey to the Sun would feel like long, excruciating minutes to any human watching.
And those minutes passed. And the Sun turned black. Not because of solar panels surrounding it, but because it was getting eaten to its very core. The wave of darkness it triggered was like a warning signal to our empty universe that it would forever stay that way. The self-replicating probes emerging from the Sun’s remains would make sure of it.
Like a nuclear explosion with its shockwave never dampening, never slowing down. Not until it reached the frontier of death itself. Only being extinguished when the universe itself would get extinguished.
Forever nothing.