Suppose there is a threshold of capability beyond which an AI may pose a non-negligible existential risk to humans.
What is the argument against this reasoning: If one AI passes or seems likely to pass this threshold, then humans, to lower x-risk, ought to push other AI past this threshold in light of the following.
1) If only one AI passes this threshold and it works to end humanity either directly or indirectly, humanity has zero chance of survival. If there are other AIs, there is a non-zero chance that they support humanity directly or indirectly, and thus humanity's chance of survival is above zero.
2) Even if, at some point, there is only one... (read more)
Thanks for the replies.
w/r to zero not being a probability: obviously. The probability is extremely low, not zero, such that the chance of a benevolent AI existing is greater than the chance of humanity surviving a single malevolent AI. If that's not the case, then 1 and 2 are useless.
Peter, thanks. After reading that Drexler piece, the linked Christiano piece, the linked Eliezer post, and a few others, especially the conversation between Eliezer and Drexler in the comments of Drexler's post, I agree with you. TBH I am surprised that there's no better standard argument in support of inevitable anti-human collusion than this from Eliezer: "They [AIs] cooperate with each other but not you because they can do a spread of possibilities on each other modeling probable internal thought processes of each other; and you can’t adequately well-model a spread of possibilities on them, which is a requirement on being able to join an LDT coalition." As Christiano says, that makes a lot of assumptions.