Many people complain that the Singularity Institute's "Big Scary Idea" (AGI leads to catastrophe by default) has not been argued for with the clarity of, say, Chalmers' argument for the singularity. The idea would be to make explicit what the premise-and-inference structure of the argument is, and then argue about the strength of those premises and inferences.
Here is one way you could construe one version of the argument for the Singularity Institute's "Big Scary Idea":
At some point in the development of AI, there will be a very swift increase in the optimization power of the most powerful AI, moving from a non-dangerous level to a level of superintelligence. (Fast takeoff)
This AI will maximize a goal function.
Given fast takeoff and maximizing a goal function, the superintelligent AI will have a decisive advantage unless adequate controls are used.
Adequate controls will not be used. (E.g. Won’t box/boxing won’t work)
Therefore, the superintelligent AI will have a decisive advantage
Unless that AI is designed with goals that stably align with ours, if the superintelligent AI has a decisive advantage, civilization will be ruined. (Friendliness is necessary)
Unless the first team that develops the superintelligent AI makes adequate preparations, the superintelligent AI will not have goals that stably align with ours.
Therefore, unless the first team that develops the superintelligent AI makes adequate preparations, civilization will be ruined shortly after fast takeoff
The first team that develops the superintelligent AI will fail to make adequate preparations
Therefore, civilization will be ruined shortly after fast takeoff.
Edit to add: premises should be read as assuming the truth of all above premises. E.g., (9) is assuming that we've created an artificial agent with a decisive advantage.
My questions are:
Have I made any errors in the argument structure?
Can anyone suggest an alternative argument structure?
I created a new page on the wiki to collect links like the Big Scary Idea one: Criticism of the sequences. If anyone knows of more links to intelligent disagreement with ideas prevailing on Less Wrong, please add them!
Well, people like user:wedrifid disagree but I don't think that the following posts are fallacious (or at least I haven't heard counterarguments that would render those posts obsolete):
* Is an Intelligence Explosion a Disjunctive or Conjunctive Event?
* Why an Intelligence Explosion might be a Low-Priority Global Risk
(I am currently writing up a post for my personal blog where I list all requirements that need to be true in conjunction for SIAI to be the best choice when it comes to charitable giving.)
Many people complain that the Singularity Institute's "Big Scary Idea" (AGI leads to catastrophe by default) has not been argued for with the clarity of, say, Chalmers' argument for the singularity. The idea would be to make explicit what the premise-and-inference structure of the argument is, and then argue about the strength of those premises and inferences.
Here is one way you could construe one version of the argument for the Singularity Institute's "Big Scary Idea":
My questions are: