The version of this argument that really bugs me, and that your AI examples demonstrate, is when the "good" end of the comparison is also really bad. I have heard someone say, in earnest, a near paraphrase of "OK, maybe AI could kill everyone, but humans commit genocide too...."
In 2012, Scott Alexander wrote about what he considered to be “the worst argument in the world”. Essentially, the form of this argument, as he laid out, is to use the inherent characteristics of a category to criticize a noncentral or marginal member of that category. In doing so, you would attempt to create associations between central members of that category and noncentral members, in order to impugn a noncentral member by implicitly ascribing characteristics that don’t actually apply to it.
For instance, consider the category “criminals”. If you think of a criminal without specific context, people like Al Capone or Philip Sackler probably come to mind: someone who cares nothing for the laws that uphold civilized society, and lies, cheats, and steals his way to personal benefit without letting those laws bind his conscience or actions. Martin Luther King - the main example Scott uses in his essay - is also a criminal, in that he broke the law, and was sent to jail for it (hence the famous Letter from Birmingham Jail). However, King, while technically a criminal, is far from the qualities we traditionally attribute to criminals (and which apply to such people as Capone and Sackler). King deliberately chose to break the law in an act of civil disobedience, in which one chooses to defy the law and accept the consequences in order to bring attention to the unjust nature of that law. This was not someone who ran around robbing banks or committing fraud or swindling people into getting addicted to deadly drugs; King was an otherwise law-abiding citizen who, in one instance, made the calculated choice to violate an unjust law in order to force people to consider whether that law should really remain in force. Today, almost everyone would agree that what King did in Birmingham was good and right, that the law he broke was profoundly unjust, and that he was a criminal in only the narrow technical sense.
Or, as Scott wrote in his original post:
Having looked at that example closely, we can now zoom out and consider the broader contours of category theory. Decades of research have shown that categories, as we perceive them, are not fixed Platonic absolutes in which all members are equally central and equally relevant. Rather, a category is more like a fuzzy conceptual ‘bucket’ with imprecise borders, and with gradients of applicability within those borders.[1] The noncentral fallacy exploits these gradients in one direction. I suggest that the second-worst argument in the world - let’s call it the central fallacy - is that which exploits these gradients in the other direction. Whereas the function and purpose of the noncentral fallacy is to attack an idea, concept, or individual by associating it with core members of a category it only marginally belongs to, the purpose of the central fallacy is to defend an idea, concept, or individual. It proceeds by observing that this idea, concept, or individual (let’s call it X) belongs to a category (of which it is a core member); that some other thing Y also belongs to that category (though Y is only a marginal member of said category); and then arguing that since Y isn’t bad (or even is good), it therefore follows that X must be fine too. Much as the noncentral fallacy takes advantage of the fact that the thing being attacked technically belongs to a category while simultaneously lacking core traits of that category, the central fallacy takes advantage of the fact that Y lacks core traits of a category it only marginally belongs to. It uses Y’s (relative or absolute) acceptable-ness to insist that X is also acceptable, ignoring the fact that Y has crucial characteristics that X lacks.
Continuing to run with the example of Martin Luther King, an application of the central fallacy might be: “Well, you say criminals are bad, but Martin Luther King was a criminal, and he was good, so people like Al Capone must not be that bad either.”
The exercise of identifying hot-button political issues where this fallacy is commonly used is left to the reader. However, I do note that in discussions about AI, especially its potential future impacts, I see it especially often. What follows are few examples of common forms it takes:
This one is a facile example, one that you can easily see right through. Despite large absolute numbers of car crashes, cars are famously steerable - they have a massive wheel for that exact purpose, as well as gas pedals, brakes, and various other control inputs. The crashes occur only on a tiny minority of drives, and fatal crashes are a tiny portion of crashes altogether. A car crash harms only a small handful of people at most, whereas an AI loss-of-control incident could gravely harm the entire human population. And even so, car crashes are far from costless: we have a tremendous amount of training, insurance, and policing infrastructure directed toward preventing drivers who can’t control their vehicles from getting behind the wheel in the first place, catching them on the roads before they can do harm, and managing the financial costs imposed by the crashes that do occur.
Today’s dictatorships and autocracies are both less durable and less repressive than the newly robust forms of dictatorial lock-in that superhuman AI would enable. They are quite often overturned by internal revolutions, such as was the fate of the regimes of Nicolae Ceausescu and Ferdinand Marcos. AI-controlled nanobot drone swarms, combined with mind-reading of the populace (which, as sci-fi as it sounds, is already being actively worked on by technologists, who predict success within a decade) would enable surveillance and control at a level that even North Korea dreams of. Even in that most repressive of all dictatorships, the people at least have the inner sanctum of their own mind - internal freedom of thought - left untrammeled. A dictatorship that can the minds of dissidents as well as ordinary citizens, and use superhuman AI to feed this information into omnipresent drones ready to fire off a deadly kill shot at any moment, is vastly worse than even the horrors of totalitarianism the past century has given us.
This one, especially common among those engaged with Nick Land’s thought, is a sleight-of-hand that relies on conflating two very different types of loss of control. The form discussed in relation to potential future AI systems means total human lack of influence: every single person could agree on something, and put all their energy into trying to achieve that thing, but the machine overlords could easily ensure that this would not be allowed to occur, if they disagreed and wanted to prevent it. To my understanding, the idea that agriculture (or the industrial revolution, or what have you) entailed loss of control operates at a more mundane level: these systems created new incentives which take on a life of their own and thus constrain realistic paths of action by creating coordination problems very difficult to break. But difficult is not impossible, and any great power could quite easily engage in deus ex machina on a number of seemingly intractable problems, should they wish to burn an unusual amount of political capital. Some abstracted telos pushed along by human institutions that take on lives of their own is very different than the prospect of humans simply being entirely out of the loop.
The “worst argument in the world” is fairly well known in certain intellectual circles. I hope that by naming and identifying its converse (the central fallacy), people will more readily spot it when it takes place and avoid falling into the trap, as well as resisting the urge to use this fallacious line of argument in the first place.
George Lakoff’s book “Women, Fire, and Dangerous Things: What Categories Reveal about the Mind” is an excellent book covering this topic.