Sep 27, 2013
Part of the series AI Risk and Opportunity: A Strategic Analaysis. Previous articles on this topic: Some Thoughts on Singularity Strategies, Intelligence enhancement as existential risk mitigation, Outline of possible Singularity scenarios that are not completely disastrous.
Below are my quickly-sketched thoughts on intelligence amplification and FAI, without much effort put into organization or clarity, and without many references. But first, I briefly review some strategies for increasing the odds of FAI, one of which is to work on intelligence amplification (IA).
Suppose you find yourself in a pre-AGI world, and you’ve been convinced that the status quo world is unstable, and within the next couple centuries we’ll likely settle into one of four stable outcomes: FAI, uFAI, non-AI extinction, or a sufficiently powerful global government which can prevent AGI development. And you totally prefer the FAI option. What should you do to get there?
Below are some key considerations about the IA route. I’ve numbered them so they’re easy to refer to later. My discussion assumes MIRI’s basic assumptions, including timelines similar to my own AGI timelines.
Below are my thoughts about all this. These are only my current views: other MIRI personnel (including Eliezer) disagree with some of the points below, and I wouldn’t be surprised to change my mind about some of these things after extended discussion (hopefully in public, on Less Wrong).
I doubt (1) is true. I think IQ 130–170 humans could figure out FAI in 50–150 years if they were trying to solve the right problems, and if FAI development wasn’t in a death race with the strictly easier problem of uFAI. If normal smart humans aren’t capable of building FAI in that timeframe, that’s probably for lack of rationality and philosophical skill, not for lack of IQ. And I’m not confident that rationality and philosophical skill predictably improve with IQ after about IQ 140. It’s a good sign that atheism increases with IQ after IQ 140, but on the other hand I know too many high-IQ people who think that (e.g.) an AI that maximizes K-complexity is a win, and also there’s Stanovich’s research on how IQ and rationality come apart. For these reasons, I’m also not convinced (4) would be a large positive effect on our FAI chances.
Can we train people in rationality and philosophical skill beyond that of say, the 95th percentile Less Wronger? CFAR has plans to find out, but they need to grow a lot first to execute such an ambitious research program.
(2) looks awfully hard, unless we can find a powerful IA technique that also, say, gives you a 10% chance of cancer. Then some EAs devoted to building FAI might just use the technique, and maybe the AI community in general doesn’t.
(5) seems right, though I doubt it’ll be a big enough effect to make a difference for the final outcome.
I think (3) is the dominant consideration here, along with the worry about lacking the philosophical skill (but not IQ) to build FAI at all. At the moment, I (sadly) lean toward the view that slower Earths have a better chance at FAI. (Much of my brain doesn’t know this, though: I remember reading the Summers news with glee, and then remembering that on my current model this was actually bad news for FAI.)
I could say more, but I’ll stop for now and see what comes up in discussion.
Not counting civilizations that might be simulating our world. This matters, but I won’t analyze that here. ↩
There are other possibilities. For example, there could be a global nuclear war that kills all but about 100,000 people, which could set back social, economic, and technological progress by centuries, thus delaying the crucial point in Earth’s history in which it settles into one of the four stable outcomes. ↩
And perhaps also advanced nanotechnology, intelligence amplification technologies, and whole brain emulation. ↩
Thanks to Carl Shulman for making this point. ↩