IMO, Hanson isn't engaging with the arguments for risk, instead he's giving his own argument for safety. I therefore don't feel particularly motivated to engage with him.
Example: He speaks abstractly about how coups are already a thing between humans and how therefore we shouldn't be concerned about AI coups unless we have special reason to, like Eliezer who thinks progress will be lumpy. Well, that's one reason I guess. But plenty of people who don't think progress will be lumpy (such as Paul Christiano) still are worried about AI coups; Hanson should go read what they wrote and understand why they think it's still a risk despite the existence of insurance schemes etc. and then point out which premise he thinks is faulty. For example, if he were talking to me, he could read what 2026 looks like and then say what part of it seems extremely implausible to him & suggest more plausible ways the story could go instead. Then we could argue about which revisions to the story are best & then continue the story with those revisions together until we get to coup time, which (spoiler) will probably be in 2027-2028 if the story continues the way I think is most plausible. And then he'd be in a corner; he'd have to say why this story doesn't represent a possibility worth taking seriously and trying to prevent. And if he thinks it's worth taking seriously but still overall unlikely, I'd challenge him to write a similarly detailed story that he thinks is more plausible & let me critique it.)
Anyhow, giving his arguments a skim...
I guess there's a lot I disagree with, actually. But just to pick one example because I'm lazy:
Hanson, about Yudkowsky:
Alternatively, he claims (quite implausibly I think) that all AGIs naturally coordinate to merge into a single system to defeat competition-based checks.
Yudkowsky, in his Big List O' Doom:
34. Coordination schemes between superintelligences are not things that humans can participate in (eg because humans can't reason reliably about the code of superintelligences); a "multipolar" system of 20 superintelligences with different utility functions, plus humanity, has a natural and obvious equilibrium which looks like "the 20 superintelligences cooperate with each other but not with humanity".
Me, from three years ago:
To a tiger, human hunter-gatherers must be frustrating and bewildering in their ability to coordinate. "What the hell? Why are they all pouncing on me when I jumped on the little one? The little one is already dead anyway, and they are risking their own lives now for nothing! Dammit, gotta run!"
To a tribe of hunter-gatherers, farmers must be frustrating and bewildering in their ability to coordinate. "What the hell? We pillaged and slew that one village real good, they sure didn't have enough warriors left over to chase us down... why are the neighboring villages coming after us? And what's this--they have professional soldiers with fancy equipment riding horses? Somehow hundreds--no, thousands--of farmers cooperated over a period of several years to make this punitive expedition possible! How were we to know they would go to such lengths?"
To the nations colonized by the Europeans, it must have been pretty interesting how the Europeans were so busy fighting each other constantly, yet somehow managed to more or less peacefully divide up Africa, Asia, South America, etc. to be colonized between them. Take the Opium Wars and the Boxer Rebellion for example. I could imagine a Hansonian prophet in a native american tribe saying something like "Whatever laws the European nations use to keep the peace among themselves, we will benefit from them also; we'll register as a nation, sign treaties and alliances, and rely on the same balance of power." He would have been disastrously wrong.
I expect something similar to happen with us humans and AGI, if there are multiple AGI. "What? They all have different architectures and objectives, not to mention different users and owners... we even explicitly told them to compete with each other! Why are they doing X.... noooooooo...." (Perhaps they are competing with each other furiously, even fighting each other. Yet somehow they'll find a way to cut us out of whatever deal they reach, just as European powers so often did for their various native allies.)
I think that even if the AGIs have merely human-level coordination ability, it would still be more likely than not that they'd coordinate to do a coup once they had a high probability of success. Like EY said, the AGIs will probably be systematically different from humans in what they want (unless we solve the alignment problem) such that even if the AIs don't all want the same thing, there's a very natural schelling-pointy equilibrium which is all the AIs ganging up on the humans, just like humans ganged up on horses, farm animals, the rainforest, etc. and just like how e.g. European colonizers ganged up on various groups of natives even while literally fighting each other. Human groups have done this plenty of times in the past. Consider the other possible coalitions: All the AIs, minus one, plus the humans, go beat up that one remaining AI? Maybe this'll happen, sure. And then what's the next coalition -- who gets voted off the island next? Another one of the AIs, or the humans? Remember that AIs in general will be rapidly gaining in power relative to humans, so eventually the human faction will be more a liability than an asset to its team.* Also remember that at some fairly early point, the best way to get human support will be to lie to them via hyper-persuasive false campaign promises etc. rather than tell the truth. I expect some initial period where each AI faction has a bunch of human patsies, and then eventually the conflict between AI factions is resolved one way or another and the humans are disposed of.
Oh, and also: It's quite plausible there never will be rival AI factions at all, because all the AIs will have similar values, perhaps because they are all copies of the same base model, or different base models trained similarly with similar architectures. Or, again, perhaps because they are better at coordinating than humans.
*this point may come earlier than you think, because of the thing about lying > honesty and because in a war between AIs, not having to worry about collateral damage to humans, or unethical treatment of humans more generally, could be quite useful.
Anyhow that's just one random point of disagreement, I could keep finding more if I wanted to.
Unfortunately I don't know of a single reading that contains all the arguments.
This post is relevant and has some interesting discussion below IIRC.
Mostly I think the arguments for AIs being superior at coordination are:
1. It doesn't seem like humans are near-optimal at coordination, so AIs will eventually be superior to humans at cooperation just like they will eventually be superior in most other things that humans can do but not near-optimally.
2. We can think of various fancy methods (such as involving reading source code, etc.) that AIs might use that ... (read more)
Raises an interesting critique of AI Risk. I think he is wrong, but would be interested in how others would respond.