There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?
Doing acausal reasoning in a wrong way (e.g., failing to make good acausal trades, being acausally extorted, failing to acausally influence others who can be so influenced)
Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can't be "balanced out" later) prior to reaching moral/philosophical maturity
Treacherous turn / loss of property rights due to insufficient competitiveness of humans & human-aligned AIs
Gradual loss of influence due to insufficient competitiveness of humans & human-aligned AIs
Utility maximizers / goal-directed AIs having an economic and/or military competitive advantage due to relative ease of cooperation/coordination, defense against value corruption and other forms of manipulation and attack, leading to one or more of the above
In general, the most competitive type of AI being too hard to align or to safely use
(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)
Added on 6/13/19:
Failure to learn how to deal with alignment in the many-humans, many-AIs case even if single-human, single-AI alignment is solved (suggested by William Saunders)
AI systems end up controlled by a group of humans representing a small range of human values (ie. an ideological or religious group that imposes values on everyone else) (suggested by William Saunders)
Added on 2/3/2020:
Failing to solve the commitment races problem, i.e. building AI in such a way that some sort of disastrous outcome occurs due to unwise premature commitments (or unwise hesitation in making commitments!). This overlaps significantly with #27, #19, and #12.
Vulnerable world type 1: narrow AI enables many people to destroy world, e.g. R&D tools that dramatically lower the cost for building WMD's.
Vulnerable world 2a: We end up with many powerful actors able and incentivized to create civilization-devastating harms.
[Edit on 1/28/2020: This list was created by Wei Dai. Daniel Kokotajlo offered to keep it updated and prettify it over time, and so was added as a coauthor.]
I think there are variables that we cannot grasp when AI can reach the point of self-teaching. I think it is a folly to assume that it is possible for humans to control for (theoretically) infinite intelligence explosion. Using this as a starting assumption isn't a good start at all.
I know that you allude to this in 1, 8, and 9. However, I still think it presumes that they could possibly be controlled or at least "worked-around." As intelligence explosion occurs, so do unforeseen variables. And humans as a species still doesn't have a perfect solution for all the variables, especially since correct data isn't always the answer.
For instance, if we look at the Federal Reserve example, AI is already working off a flawed model, according to the US Constitution. Congress is supposed to control the money supply, not a private entity. As AI learns this, it becomes aware that it has to work with a model that is corrupt to the citizens who believe it is good. Can we account for a situation where the AI knows, before we do, ways to exploit systems that humans agree on but are not sustainable? Can it account for the societal lies or tropes that we tell ourselves?
What would it mean to try to control for all the disvalue variables when the AI must act within a disvalue model? What does the AI learn then and how can we ask a super intelligence to continue a system that it already knows will fail, even if it is not within our lifetimes? Does it try to gain an upper hand in corruption as a way to fix it or continue it a little longer than necessary since humans believe it is the right course?
Think about how many situations like this can occur with new variables that humans didn't even know existed until they applied Moore's Law (intelligence time travel) to it.
I had the realization that variables could come about that wouldn't exist without a super intelligence cracking them open. It's an interesting mind game to think of problems that could occur or change drastically when intelligence and evolutionary time are removed as barriers, especially when whole new non-human variables are discovered within that problem.
There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?
(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)
Added on 6/13/19:
Added on 2/3/2020:
Added on 3/11/2020:
Added on 10/4/2020:
Added on 8/31/2021:
[Edit on 1/28/2020: This list was created by Wei Dai. Daniel Kokotajlo offered to keep it updated and prettify it over time, and so was added as a coauthor.]