There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?
Doing acausal reasoning in a wrong way (e.g., failing to make good acausal trades, being acausally extorted, failing to acausally influence others who can be so influenced)
Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can't be "balanced out" later) prior to reaching moral/philosophical maturity
Treacherous turn / loss of property rights due to insufficient competitiveness of humans & human-aligned AIs
Gradual loss of influence due to insufficient competitiveness of humans & human-aligned AIs
Utility maximizers / goal-directed AIs having an economic and/or military competitive advantage due to relative ease of cooperation/coordination, defense against value corruption and other forms of manipulation and attack, leading to one or more of the above
In general, the most competitive type of AI being too hard to align or to safely use
(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)
Added on 6/13/19:
Failure to learn how to deal with alignment in the many-humans, many-AIs case even if single-human, single-AI alignment is solved (suggested by William Saunders)
AI systems end up controlled by a group of humans representing a small range of human values (ie. an ideological or religious group that imposes values on everyone else) (suggested by William Saunders)
Added on 2/3/2020:
Failing to solve the commitment races problem, i.e. building AI in such a way that some sort of disastrous outcome occurs due to unwise premature commitments (or unwise hesitation in making commitments!). This overlaps significantly with #27, #19, and #12.
Vulnerable world type 1: narrow AI enables many people to destroy world, e.g. R&D tools that dramatically lower the cost for building WMD's.
Vulnerable world 2a: We end up with many powerful actors able and incentivized to create civilization-devastating harms.
[Edit on 1/28/2020: This list was created by Wei Dai. Daniel Kokotajlo offered to keep it updated and prettify it over time, and so was added as a coauthor.]
Number 18 is interesting. Suppose, for example, the quest for apolitical control of interest rates leads to an AI at the head of the Federal Reserve. Given all the impressive looking equations you can find in macroeconomic papers and textbooks, I wonder how many people realise just how little science there is in that entire body of theory, and how much of it comprises philosophy and political belief, dressed up to look like hard physics, but resting on the assertions of famous "seminal papers" instead of premises or evidence.
How long before the Reserve Board of Governersstarts "consulting" the AI instead of using it to double-check their work; stops double-checking the AI's work and merely runs integrity checks on it; stops acquainting itself with the theory (or more likely, weighted combination of theories) on which it runs; stops keeping track of which theories it runs; is criticised in the press for unthinkingly doing what the AI says; is criticised in the press for not just doing what the AI says; is questioned in the Senate about whether it has any idea what the AI is doing or more importantly, why it is doing it?
There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?
(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)
Added on 6/13/19:
Added on 2/3/2020:
Added on 3/11/2020:
Added on 10/4/2020:
Added on 8/31/2021:
[Edit on 1/28/2020: This list was created by Wei Dai. Daniel Kokotajlo offered to keep it updated and prettify it over time, and so was added as a coauthor.]