Inspired by the talk by Anna Salamon I decided to do my own calculations about the future. This post is a place for discussion about mine and others calculations.
To me there are two possible paths for the likely development of intelligence, that I can identify.
World 1) Fast and conceptually clean. Intelligence is a concrete value like the number of neutrons in a reactor. I assign a 20% chance of this.
World 2) Slow and messy. Intelligence is contextual, much like say fitness in evolutionary biology. Proofs of intelligence of a system are only doable by a much higher intelligence entity, as it will involve discussing the complex environment. I'd assign about an 60% chance to this.
Worlds 3) Other. The other 20% chance is the rest of the scenarios that are not either of these two.
Both types of AI have the potential to change the world, both possibly destroying humanity if we don't use them correctly. So they both have the same rewards.
So for world 1, I'll go with the same figures as Anna Salamon, because I can't find strong arguments against them (and it will serve as a refresher )
Probability of an eventual AI (before humanity dies otherwise) = 80%
Probability that AI will kill us = 80%
Probability that we manage safeguards = 40%
Probability that current work will save us = 30%
So we get 7%*20%. Gives us 1.4%
So for world 2. Assume we have an SIAI that is working on the problem of how to make messy AI Friendly or at least as Friendly as possible. It seems less likely we would make AI and harder to create safeguards as they have to act over longer time.
Probability of an eventual AI (before humanity dies otherwise) = 70%
Probability that AI will kill us (and/or we will have to give up humanity due to hard scrapple evolution) = 80%
Probability that we manage safeguards = 30%
Probability that current work will save us = 20%
So we get a factor of 3% times 60% give a 1.8%.
Both have the factor of 7billion lives times n, so that can be discounted. They pretty much weigh the same. Or as near as dammit for a back of the envelope calcs, considering my meta-uncertainty is high as well.
They do however interfere. The right action in world 1 is not the same as the right action in world 2. Working on Friendliness of conceptually clean AI and suppressing all work and discussion on messy AI hurts world 2 as it increases the chance we might end up with messy UFAI. There is no Singularity Institute for messy AI in this world, and I doubt there will be if SIAI becomes somewhat mainstream in AI communities, so giving money to SIAI hurts world 2, it might have a small negative expected life cost. Working on Friendliness for Messy AI wouldn't intefere with the Clean AI world, as long as it didn't do stupid tests until the messy/clean divide became solved. This tips the scales somewhat towards working on messy FAI and how it is deployed. World 3 is so varied I can't really say much about.
So for me the best information I should seek is getting more information on the messy/clean divide. Which is why I always go on about whether SIAI has a way of making sure it is on the right track with the Decision Theory/conceptually clean path.
So how do the rest of you run the numbers on the singularity?