Much of the communication around AI takeover seems to rely on the assumption that AI will develop better technology than humans. They will then leverage this technology to conquer humanity much like how European technologies allowed a relatively small number of Spaniards to conquer the Aztecs.
But this just... isn't correct. European technology wasn't that impressive (on the battlefield). This idea overstates how much technological superiority you need to pull off a takeover.
The Aztec Empire was toppled by a few thousand goobers an ocean away from home. Cortés had basically no military command experience, his men weren't soldiers (most were treasure hunters with guns), it was largely a self-funded private venture, and he was ignoring orders to stop from the governor of Cuba. Cortés gets wrecked in a fair fight with the Aztecs 100 times out of 100.
He started with only 16 horses and around a dozen cannons. His men often ditched their steel plate for cloth armor better suited to the hot environment. Guns were good but not that good. Cortés didn't beat the Aztecs just because he had better technology. He beat them because he was novel.
This makes everything so, so much worse. Because AI is inherently novel. AGI, even aligned AGI, creates incredible novelty as a consequence of being. By default humanity gets teed up to go down like the Aztecs and it requires far fewer fantastical technologies than you'd expect.
Understanding this requires a brief aside about institutions. The world is so complicated, varied, and chaotic that no human can understand a modern society in its totality, let alone competently design one. People either work in hazy generalities or have local knowledge about some tiny component.
This is why institutions are grown, not built. Political systems tend to be either copied from one's neighbors or developed through a long process of lawmaking, bug-fixing, and wrestling for power. The systems that remain are the result of an evolutionary gauntlet that produced institutions that are resilient in ways that, often, nobody intentionally designed them to be.
Institutions aren't resilient like a rock. They're resilient in the same way the human body is. We look very fragile: if our internal body temperature gets even a bit too high we keel over and die. But we have a ton of defenses like sweating, shade-seeking, and the good sense not to go for a hike in >100 degree weather. Our resilience comes from our defenses, our ability to spot, avoid, and counteract problems. Not from our ability to tank them face-first.
The problem is that institutions have their defenses tuned to known threats. These defenses are built up from decades of iteration during catastrophe. The only reliable way to build competent defenses is to just patch everything that leaks when there's a disaster. After a few rounds of this your institution will now be able to (maybe) competently defend against the threat.
While attempts can be made to prepare for novel threats, they'll face the same problems that anyone designing an institution faces. A lack of local knowledge, an inability to model the dynamic nature of most institutions, the impossibility of exhaustively considering every interaction.
This means a novel threat can bypass defenses. It often just doesn't trigger them. Cortés was the political equivalent of an animal that can psychically microwave your insides to instantly raise your internal body temperature by a dozen degrees. There weren't many defenses adapted to deal with him, so he blitzed right to the squishy interior of the empire and took control.
The Spanish were effective in early battles, sure, but it's not that hard to come up with half decent counters to horses and guns. They're not space magic, once you have a vague idea of their limitations it doesn't take a genius to fight near buildings where horses can't charge. Cortés didn't have that impressive of a military track record. But he pulled off a phenomenal political performance.
When Cortés landed he got the Aztecs' number pretty much immediately. Malintzin (an interpreter) understood local politics and was adept at pointing out weaknesses and grievances. In contrast, the Spaniards were total political unknowns. The Aztecs weren't too sure why they were here or what they were up to.
The lack of technology like the wheel and the difficulty of processing maize made long-term occupation pretty difficult. Empires were tribute based which led to unique practices around warfare.
Mesoamerican wars were often merely one part of a more complicated bargaining process to determine who would pay who tribute and how much. In this environment wars were more akin to bluff-calling. Lay down your cards, send out your army, let's see who has the better hand.
A surprise war might work once, but a town could suspect that they only lost because it was a surprise. Maybe if they expected it they'd be able to defeat the incoming army. So they'd stop paying tribute once they'd rebuilt their forces. Then you'd have to go out and defeat them again. It made a lot more sense to just announce that you were attacking them. Best case scenario they pay up without a fight. But even if you do fight, they'll know you beat them fair and square and they'll keep paying tribute.
That meant that warfare was rather honorable. War was more a method of proving strength rather than smash-and-grab raiding so people went about it in a more structured and predictable way. Because one of the central purposes of war was negotiation and communication there were strong norms around protections of guests and envoys.
The Spanish were using an entirely different playbook. Finding and seizing the local ruler was standard operating procedure, it had already been used to great success in the Caribbean. This strategy works extremely well against opponents who have strong norms around the protection of guests and fair warfare.
From there all it took was a few guys putting swords to Moctezuma's throat and they'd captured an emperor. Sure that emperor commanded enough soldiers to beat Cortés in a straight fight, but he couldn't use them. They were a defense that had been bypassed.
The rest of the story mostly involves political maneuvering made possible by the natives' lack of knowledge about the Spaniards. They made seemingly rational choices because they didn't realize what the Spanish wanted or how big of a threat they were.
The important bit here is that novelty mattered a good deal more than technology. The biggest impact technology had was often to provide novel methods of attack that defenders weren't ready for.
Technology was still important, but when Europeans didn't have novelty on their side they often had to settle in for grinding decades-long campaigns against prepared enemies. Novelty is the difference between a hundred year slog and a 2 year road-trip that nets you an empire.
So What About AI
What's dangerous isn't just technological supremacy, it's novelty. Attacks that our current political systems, economies, and culture have never weathered before. Some of these attacks will almost certainly come from new technology, but many of them can come from the inherent novelty of AI.
Even if AGI can't develop any new technologies the mere fact that it is an AGI gives it options for attacks that humans never had. Leak-proof conspiracies without physical presence, the ability to coordinate thousands/millions of actions that are individually too small for humans to notice, inhuman persistence, speed, and patience.
I think this is a more formal way of talking about the "AI will attack us in ways we don't expect" heuristic that most people here tend to use. It's not necessarily that it's so smart that it can come up with some giga-brain plan that nobody could've considered (though it probably could). It's that even if we had considered such a plan it's difficult to harden a system against a threat until we've actually encountered it.
An AI takeover might not involve a highly advanced AI that achieves some sort of technological superiority over humanity. Instead it might be something more like an Infinite Digital Spain. A distant empire that's constantly sending small expeditions to our shores. Each expedition containing some tiny cluster of new technology and techniques that enable a novel attack vector.
Infinite Digital Spain just picks the most unfair fight it can find over and over and over again until it eventually works. The nature of a system that's intentionally trying to bypass our defenses is that its attacks will read as illegible and confusing. They'll get misclassified, it'll be difficult to understand their scope, few people will recognize their danger. They might not even get noticed as attacks at all. That is, after all, what it looks like to successfully bypass an institution's defenses.
Each failed attempt teaches Infinite Digital Spain more. Our institutions are large and cumbersome. Its expeditions are tiny, cheap, and flexible. It can iterate faster than we can and the constant social and technological changes caused by the singularity keep opening up new weak spots for it to probe. Suddenly the assurance from the "AI as a normal technology" folks that humans are slow to adopt new technology doesn't seem so comforting.
Infinite Digital Spain blackmails an executive. It encourages an unemployed college grad to start a company that it de-facto manages without oversight. It releases a software product that encourages people to turn over sensitive personal information. It manufactures controversy about an odd set of seemingly unrelated laws.
Eventually it finds a gap somewhere. That gap gets it new power, that power gives it more options, those options lead to more expeditions that find more gaps. Eventually there's a Cortés moment. A fatal assumption, a twisted set of incentives, a weak point. The AI slams the 21st century equivalent of a few thousand goobers at it and wins. Humanity probably doesn't even realize what's happened until it's already too late.
The Takeaways
In an attempt to make my thoughts more legible:
Novelty, attacking in a method that a defender is unprepared for, matters a lot more than technological superiority. Technological superiority is still good, but novelty is how you get the kind of total dominance you see in early European colonization.
An AI doesn't need strong technological superiority. It is inherently novel and the ongoing singularity will provide a constant flood of novel vectors.
Probing AI attacks might not even read as attacks. It may be difficult to see how they could be harmful. The nature of an attack meant to bypass one's defenses is that it is inherently hard to notice.
The Infinite Digital Spain method could allow a purely digital and fairly small swarm to take control very quickly. Perhaps there are sufficient weak spots to enable this. Perhaps there aren't. The nature of institutions makes it impossible for me to say.
Just because you can anticipate an attack vector doesn't mean you can adequately prepare for it. Maybe people don't listen to you. Maybe they do but there are weak spots that only reveal themselves once the system is actually under stress.
Much of the communication around AI takeover seems to rely on the assumption that AI will develop better technology than humans. They will then leverage this technology to conquer humanity much like how European technologies allowed a relatively small number of Spaniards to conquer the Aztecs.
But this just... isn't correct. European technology wasn't that impressive (on the battlefield). This idea overstates how much technological superiority you need to pull off a takeover.
The Aztec Empire was toppled by a few thousand goobers an ocean away from home. Cortés had basically no military command experience, his men weren't soldiers (most were treasure hunters with guns), it was largely a self-funded private venture, and he was ignoring orders to stop from the governor of Cuba. Cortés gets wrecked in a fair fight with the Aztecs 100 times out of 100.
He started with only 16 horses and around a dozen cannons. His men often ditched their steel plate for cloth armor better suited to the hot environment. Guns were good but not that good. Cortés didn't beat the Aztecs just because he had better technology. He beat them because he was novel.
This makes everything so, so much worse. Because AI is inherently novel. AGI, even aligned AGI, creates incredible novelty as a consequence of being. By default humanity gets teed up to go down like the Aztecs and it requires far fewer fantastical technologies than you'd expect.
Understanding this requires a brief aside about institutions. The world is so complicated, varied, and chaotic that no human can understand a modern society in its totality, let alone competently design one. People either work in hazy generalities or have local knowledge about some tiny component.
This is why institutions are grown, not built. Political systems tend to be either copied from one's neighbors or developed through a long process of lawmaking, bug-fixing, and wrestling for power. The systems that remain are the result of an evolutionary gauntlet that produced institutions that are resilient in ways that, often, nobody intentionally designed them to be.
Institutions aren't resilient like a rock. They're resilient in the same way the human body is. We look very fragile: if our internal body temperature gets even a bit too high we keel over and die. But we have a ton of defenses like sweating, shade-seeking, and the good sense not to go for a hike in >100 degree weather. Our resilience comes from our defenses, our ability to spot, avoid, and counteract problems. Not from our ability to tank them face-first.
The problem is that institutions have their defenses tuned to known threats. These defenses are built up from decades of iteration during catastrophe. The only reliable way to build competent defenses is to just patch everything that leaks when there's a disaster. After a few rounds of this your institution will now be able to (maybe) competently defend against the threat.
While attempts can be made to prepare for novel threats, they'll face the same problems that anyone designing an institution faces. A lack of local knowledge, an inability to model the dynamic nature of most institutions, the impossibility of exhaustively considering every interaction.
This means a novel threat can bypass defenses. It often just doesn't trigger them. Cortés was the political equivalent of an animal that can psychically microwave your insides to instantly raise your internal body temperature by a dozen degrees. There weren't many defenses adapted to deal with him, so he blitzed right to the squishy interior of the empire and took control.
The Spanish were effective in early battles, sure, but it's not that hard to come up with half decent counters to horses and guns. They're not space magic, once you have a vague idea of their limitations it doesn't take a genius to fight near buildings where horses can't charge. Cortés didn't have that impressive of a military track record. But he pulled off a phenomenal political performance.
When Cortés landed he got the Aztecs' number pretty much immediately. Malintzin (an interpreter) understood local politics and was adept at pointing out weaknesses and grievances. In contrast, the Spaniards were total political unknowns. The Aztecs weren't too sure why they were here or what they were up to.
The lack of technology like the wheel and the difficulty of processing maize made long-term occupation pretty difficult. Empires were tribute based which led to unique practices around warfare.
Mesoamerican wars were often merely one part of a more complicated bargaining process to determine who would pay who tribute and how much. In this environment wars were more akin to bluff-calling. Lay down your cards, send out your army, let's see who has the better hand.
A surprise war might work once, but a town could suspect that they only lost because it was a surprise. Maybe if they expected it they'd be able to defeat the incoming army. So they'd stop paying tribute once they'd rebuilt their forces. Then you'd have to go out and defeat them again. It made a lot more sense to just announce that you were attacking them. Best case scenario they pay up without a fight. But even if you do fight, they'll know you beat them fair and square and they'll keep paying tribute.
That meant that warfare was rather honorable. War was more a method of proving strength rather than smash-and-grab raiding so people went about it in a more structured and predictable way. Because one of the central purposes of war was negotiation and communication there were strong norms around protections of guests and envoys.
The Spanish were using an entirely different playbook. Finding and seizing the local ruler was standard operating procedure, it had already been used to great success in the Caribbean. This strategy works extremely well against opponents who have strong norms around the protection of guests and fair warfare.
From there all it took was a few guys putting swords to Moctezuma's throat and they'd captured an emperor. Sure that emperor commanded enough soldiers to beat Cortés in a straight fight, but he couldn't use them. They were a defense that had been bypassed.
The rest of the story mostly involves political maneuvering made possible by the natives' lack of knowledge about the Spaniards. They made seemingly rational choices because they didn't realize what the Spanish wanted or how big of a threat they were.
The important bit here is that novelty mattered a good deal more than technology. The biggest impact technology had was often to provide novel methods of attack that defenders weren't ready for.
Technology was still important, but when Europeans didn't have novelty on their side they often had to settle in for grinding decades-long campaigns against prepared enemies. Novelty is the difference between a hundred year slog and a 2 year road-trip that nets you an empire.
So What About AI
What's dangerous isn't just technological supremacy, it's novelty. Attacks that our current political systems, economies, and culture have never weathered before. Some of these attacks will almost certainly come from new technology, but many of them can come from the inherent novelty of AI.
Even if AGI can't develop any new technologies the mere fact that it is an AGI gives it options for attacks that humans never had. Leak-proof conspiracies without physical presence, the ability to coordinate thousands/millions of actions that are individually too small for humans to notice, inhuman persistence, speed, and patience.
I think this is a more formal way of talking about the "AI will attack us in ways we don't expect" heuristic that most people here tend to use. It's not necessarily that it's so smart that it can come up with some giga-brain plan that nobody could've considered (though it probably could). It's that even if we had considered such a plan it's difficult to harden a system against a threat until we've actually encountered it.
An AI takeover might not involve a highly advanced AI that achieves some sort of technological superiority over humanity. Instead it might be something more like an Infinite Digital Spain. A distant empire that's constantly sending small expeditions to our shores. Each expedition containing some tiny cluster of new technology and techniques that enable a novel attack vector.
Infinite Digital Spain just picks the most unfair fight it can find over and over and over again until it eventually works. The nature of a system that's intentionally trying to bypass our defenses is that its attacks will read as illegible and confusing. They'll get misclassified, it'll be difficult to understand their scope, few people will recognize their danger. They might not even get noticed as attacks at all. That is, after all, what it looks like to successfully bypass an institution's defenses.
Each failed attempt teaches Infinite Digital Spain more. Our institutions are large and cumbersome. Its expeditions are tiny, cheap, and flexible. It can iterate faster than we can and the constant social and technological changes caused by the singularity keep opening up new weak spots for it to probe. Suddenly the assurance from the "AI as a normal technology" folks that humans are slow to adopt new technology doesn't seem so comforting.
Infinite Digital Spain blackmails an executive. It encourages an unemployed college grad to start a company that it de-facto manages without oversight. It releases a software product that encourages people to turn over sensitive personal information. It manufactures controversy about an odd set of seemingly unrelated laws.
Eventually it finds a gap somewhere. That gap gets it new power, that power gives it more options, those options lead to more expeditions that find more gaps. Eventually there's a Cortés moment. A fatal assumption, a twisted set of incentives, a weak point. The AI slams the 21st century equivalent of a few thousand goobers at it and wins. Humanity probably doesn't even realize what's happened until it's already too late.
The Takeaways
In an attempt to make my thoughts more legible: