YonatanK

Announcing a $500 bounty for work that meaningfully engages with the idea of asymmetric existential AI risk. Background Existential risk has been defined by the rationalist/Effective Altruist sphere as existential relative to the human species, under the premise that the continuation of the species has very high value. This provided a strong rationality (or effectiveness) grounding for big investments in AI alignment research when the risks still seemed to most people remote and obscure. However, as an apparent side-effect, "AI risk" and "risk of a misaligned AI destroying humanity" have become nearly conflated. Over the past couple of years I have attempted to draw attention to highly asymmetric AI risks, where a small number of controllers of "aligned" (from their point of view) AI employ it to kill the rest of the human population. From the point of view of the average person, who would stand to be killed along with their children and approximately everyone they personally know, this ought to count meaningfully as existential risk. Arguably, by a similar logic to the one used to justify early alignment research, even with a low probability of such an outcome its badness justifies investment in its prevention. Furthermore, prevention by way of arresting AI development conveniently provides a two-for-one solution, also addressing the misalignment problem. Conversely, investments in ensuring successful AI "alignment" without evaluating the full destructive potential of aligned AI potentially makes the investor complicit in genocide. These points suggest a strong interest by members of the rationalist/Effective Altruist sphere (at least, based on my understanding of their stated commitments) in asymmetric existential AI risk. But so far my efforts have revealed no evidence of such interest. This bounty is an attempt to stimulate engagement through small monetary reward(s). More concretely, the goal is to promote broadly changing the status of this risk from "unac