The approach taken in some of these questions, particularly Q3, seems unlikely to yield helpful responses and likely to make you seem uninformed. It would probably be better to ask directly about one or a few relevant inputs:
More generally, while I think that getting a better sense of AI researchers' views does have value, I am afraid that the primary effect of presenting these questions in this way may be to make the marginal researcher less receptive to serious arguments or discussions about AI risk. In light of this I would recommend condensing questions 2, 4, 5, 6 and presenting them in a way that seems less loaded, if you are set on approaching a significant fraction of all AI researchers.
(Though I would also suggest applying a little more deliberation, particularly in revising or varying questions / explanations / background between rounds, if you are going to ask more people.)
Yes.
Also, it may be important to clarify what is meant by "intelligence", as many researchers seem to be confused because they're not sure what's meant by "intelligence."
XiXiDu: You should clarify this "human-level intelligence" concept, it seems to be systematically causing trouble. For example:
"By AI having 'human-level intelligence' we mean that it's a system that's about as good or better (perhaps unevenly) than humans (or small groups of humans) at activities such as programming, engineering and research."
The idea of "human-level intelligence" inspired by science fiction or naive impressions from AI that refers to somewhat human-like AIs is pervasive enough that when better-informed people hear a term like "human-level intelligence", they round up to this cliche and proceed with criticizing it.
Agreed. But not all respondents trash the question just because it's poorly phrased. Nils Nilsson writes:
I'll rephrase your question to be: When will AI be able to perform around 80% of these jobs as well or better than humans perform?
I really like this guy.
I agree. I think the most important use of the concept is in question 3, and so for timeline purposes we can rephrase "human-level intelligence" as "human-level competence at improving its source code, combined with a structure that allows general intelligence."
Question 3 would then read "What probability do you assign to the possibility of an AGI with human-level competence at improving its source code being able to self-modify its way up to massively superhuman skills in many areas within a matter of hours/days/< 5 years?"
I don't think most AI researchers think of "improving its source code" as one of the benchmarks in an AI research program. Whether or not you think it is, asking them to identify a benchmark that they've actually thought about (I really like Nilson's 80% of human jobs, especially since it jives well with a Hansonian singularity) seems more likely to get an informative response.
Might be worth specifying whether "human-level competence at improving its source code" here means "as good at improving source code as an average professional programmer," "as good at improving source code as an average human," "as good at improving source code as the best professional programmer," or something else.
Some remarks:
I've been enjoying reading the discussion it has generated.
...
Given the comments, I might change my mind and not post anything. :-)
Oh, well. You asked, I answered, they think I'm an idiot. Another day in the life.
Yes, I think it would be much better if the comments were not so negative (in many cases with little justification). I would request that in the future you explicitly discourage negative comments unless they genuinely add something new to the discussion (i.e. a direct criticism of something the interviewee said that is not simply rehashing a standard argument on LW).
EDIT: Also, thanks for taking the initiative to do this, though like paulfchristiano noted, I think it's important to be careful to make a good impression in general. Perhaps try to get more detailed feedback if possible before sending out the next batch?
I think it's important to be careful to make a good impression in general. Perhaps try to get more detailed feedback if possible before sending out the next batch?
SI knows how the emails look that I send and I contact SI for help if there are questions or the need to reply to an expert. What do you mean by 'detailed feedback'? ETA: Never mind, this was the first comment I read.
I got the impression that Peter J. Bentley crafted his responses like a climate scientist responding to a climate change denier: patronizing and dismissive. I think you may be able to avoid that in future with the newly added Question 8. It should not only act as a shibboleth but also as effective in-group signaling. Might want to make it the first question, though.
The signaling uses of Q8 seem like a bad idea to me, although it seems a worthwhile thing to ask for Steve Rayhawk's reasons. If someone is all prepared to be patronizing and dismissive, going "Are you familiar with Goedel machines, AIXI, etc.?" may help if they know what those things are, but seems likely to do more harm if they don't know them, or regard them negatively - and those people are, by hypothesis, the ones who are going to be patronizing and dismissive.
In fact, I'd prefer it if Q8 started out with the less-shibbolethy "How much have you read about, or used the concepts of..." or something like that, which replaces a dichotomy with a continuum.
In fact, I'd prefer it if Q8 started out with the less-shibbolethy "How much have you read about, or used the concepts of..." or something like that, which replaces a dichotomy with a continuum.
Yeah... I wanted to make the suggested question less loaded, but it would have required more words, and I was unthinkingly preoccupied with worry about a limit on the permitted complexity of a single-sentence question. Maybe I should have split the question across more sentences.
The signaling uses of Q8 seem like a bad idea to me, although it seems a worthwhile thing to ask for Steve Rayhawk's reasons.
My reasons for suggesting Q8 were mostly:
First, I wanted to make it easier to narrow down hypotheses about the relationship between respondents' opinions about AI risk and their awareness of progress toward formal, machine-representable concepts of optimal AI design (also including, I guess, progress toward practically efficient mechanized application of those concepts, as in Schmidhuber's Speed Prior and AIXI-tl).
Second, I was imagining that many respondents would be AI practitioners who thought mostly in terms of architectures with a machine-learning flavor. Those architectures usually have a very specific and limited structure in their hypothesis space or policy space by construction, such that it would be clearly silly to imagine a system with such an architecture self-representing or self-improving. These researchers might have a conceptual myopia by which they imagine "progress in AI" to mean only "creation of more refined machine-learning-style architectures", of a sort which of course wouldn't lead towards passing a threshold of capability of self-improvement anytime soon. I wanted to put in something of a conceptual speed bump to that kind of thinking, to reduce unthinking dismissiveness in the answers, and counter part of the polarizing/consistency effects that merely receiving and thinking about answering the survey might have on the recipients' opinions. (Of course, if this had been a survey which were meant to be scientific and formally reportable, it would be desirable for the presence of such a potentially leading question to be an experimentally controlled variable.)
With those reasons on the table, someone else might be able to come up with a question that fulfills them better. I also agree with paulfchristiano's comment.
may help if they know what those things are, but seems likely to do more harm if they don't know them, or regard them negatively - and those people are, by hypothesis, the ones who are going to be patronizing and dismissive.
I had considered that. Here's my assumptions:
If true, I think the first assumption is especially important. It's the difference between answering a journalist's question and answering that same question at a professional conference. In the former case, I would have to consider the variety of ways they're likely to misunderstand or skew my answer, and, really, I just want to give them the answer that produces the belief I want. E.g., don't freak about A.I. because you know nothing about it and we do. We're not worried, really. Now, look at this cute dancing robot and give us more funding.
Edit: I forgot to add that I agree with you on changing the wording of Q8. Although, I don't think it makes it any less shibbolethy, just less obviously a shibboleth. Sneaky, I like it.
Even some of the more open-minded machine learning researchers regard AIXI, etc. as silly. I think if anything this signals that you belong to the out-group.
So, split the silly things into another question, or remove them entirely, or replace them with similarly functional in-group signaling equivalents. My main thrust is that in-group signaling is important in this situation, but I think paulfchristiano got at that better than I did.
Very impressed by the sanity level of Nilsson's response. He's a very respected name in the area, and this is a pretty warm endorsement for SIAI's project, despite placing it below global warming and giving 0.01% probability of extinction event due to AGI. Some people would call 0.01% a small risk (particularly AGI researchers) and move right along.
Curious what Minsky has to say - I assume you're going after him at some point?
Very impressed by the sanity level of Nilsson's response.
I am not sure "sanity level" is the phrase I would use here, since the difference seems to be "Nilsson agrees with SIAI" rather than "Nilsson isn't crazy."
Especially since it seems rather discongruent to assign a a 50% probability for AI to be developed by 2050 and a 90% probability for self-modification that leads to superintelligence within 5 years of AI being developed... and then only a 0.01% probability for AI-caused human extinction within this century.
Of course, he may be presuming that superintelligence alone won't help much.
Curious what Minsky has to say - I assume you're going after him at some point?
He was the first person I wrote. He never replied.
I generally agree with paulfchristiano here. Regarding Q2, Q5 and Q6 I'll note that that aside from Nils Nilsson, the researchers in question do not appear to be familiar with the most serious existential risk from AGI: the one discussed in Omohundro's The Basic AI Drives. Researchers without this background context are unlikely to deliver informative answers on Q2, Q5 and Q6.
What bothers me in The Basic AI Drives is a complete lack of quantitativeness.
Temporal discount rate isn't even mentioned. No analysis of self-improvement/getting-things-done tradeoff. Influence of explicit / implicit utility function dichotomy on self-improvement aren't considered.
I find some of your issues with the piece legitimate but stand by my characterization of the most serious existential threat from AI being of the type described in the therein.
- I have a draft book on "beliefs" that deals with (among other things) how to evaluate them. See:
That looks interesting, I will bookmark this for further reading. Anyone else planning on reading the draft?
Nilsson's response feels a little odd. It seems rather discongruent to assign a a 50% probability for AI to be developed by 2050 and a 90% probability for self-modification that leads to superintelligence within 5 years of AI being developed... and then only a 0.01% probability for AI-caused human extinction within this century.
Of course, he may be presuming that superintelligence alone won't make an AI very powerful.
Notice that his probabilities are for "AI will be able to do 80% of the jobs a human can do." That's much more limited than "general intelligence" (whatever that is) and thus much more likely.
As well, if the research focus is on "how do we automate production?" rather than "how do we create mentally superior beings that will take over reality?" it seems that the chance of the machine intelligences making us go extinct is much lower. The Staples supply bots want to fulfill orders for paperclips, not maximize the number of paperclips that exist. From that perspective, estimating the chance of a FOOM by 2100 at one in ten thousand doesn't sound unreasonably low to me.
Actually, the more I think about the "80% of the jobs a human can do" metric, the more I wonder about it.
I mean, a particularly uncharitable interpretation starts counting jobs like "hold this door open", in which case it's possible that existing computers can do 80% of the jobs a human can do. (Possibly even without being turned on.)
I mean, a particularly uncharitable interpretation starts counting jobs like "hold this door open"
Well, 'charitable' is hard to judge there. That interpretation makes it easier for computers to meet that standard- is the threshold more meaningful when it's easy or hard? Hard to say.
Even if by jobs he means "things people get paid to do full-time," you have the question of weighting jobs equally (if even one person gets paid to floss horse teeth, that goes on the list of things an AI has to be able to do) or by composition (only one person doing the job means it's a tiny fraction of jobs). But the second is a fluid thing, especially as jobs are given to machines rather than people!
The signaling uses of Q8 seem like a bad idea to me, although it seems a worthwhile thing to ask for Steve Rayhawk's reasons. If someone is all prepared to be patronizing and dismissive, going "Are you familiar with Goedel machines, AIXI, etc.?" may help if they know what those things are, but seems likely to do more harm if they don't know them, or regard them negatively - and those people are, by hypothesis, the ones who are going to be patronizing and dismissive.
In fact, I'd prefer it if Q8 started out with the less-shibbolethy "How much have you read about, or used the concepts of..." or something like that, which replaces a dichotomy with a continuum.
In fact, I'd prefer it if Q8 started out with the less-shibbolethy "How much have you read about, or used the concepts of..." or something like that, which replaces a dichotomy with a continuum.
Yeah... I wanted to make the suggested question less loaded, but it would have required more words, and I was unthinkingly preoccupied with worry about a limit on the permitted complexity of a single-sentence question. Maybe I should have split the question across more sentences.
...The signaling uses of Q8 seem like a bad idea to me, although it seems a
[Click here to see a list of all interviews]
I am emailing experts in order to raise and estimate the academic awareness and perception of risks from AI.
(Note: I am also asking Reinforcement Learning / Universal AI researchers, teams like the one that build IBM Watson, organisations like DARPA and various companies. Some haven't replied yet while I still have to write others.)
Nils John Nilsson is one of the founding researchers in the discipline of Artificial intelligence. He is the Kumagai Professor of Engineering, Emeritus in Computer Science at Stanford University. He is particularly famous for his contributions to search, planning, knowledge representation, and robotics. [Wikipedia] [Homepage] [Google Scholar]
Peter J. Bentley, a British author and computer scientist based at University College London. [Wikipedia] [Homepage]
David Alan Plaisted is a computer science professor at the University of North Carolina at Chapel Hill. [Wikipedia] [Homepage]
Hector Levesque is a Canadian academic and researcher in artificial intelligence. He does research in the area of knowledge representation and reasoning in artificial intelligence. [Wikipedia] [Homepage]
Nils Nilsson: I did look at the lesswrong.com Web page. Its goals are extremely important! One problem about blogs like these is that I think they are read mainly by those who agree with what they say -- the already converted. We need to find ways to get the Fox News viewers to understand the importance of what these blogs are saying. How do we get to them?
Before proceeding to deal with your questions, you might be interested in the following:
1. An article called "Rationally-Shaped Artificial Intelligence" by Steve Omohundro about robot "morality." It's at:
http://selfawaresystems.com/2011/10/07/rationally-shaped-artificial-intelligence/
2. I have a draft book on "beliefs" that deals with (among other things) how to evaluate them. See:
http://ai.stanford.edu/~nilsson/Beliefs.html
(Comments welcome!)
3. Of course, you must know already about David Kahneman's excellent book, "Thinking, Fast and Slow."
Here are some (maybe hasty!) answers/responses to your questions.
David Plaisted: Your questions are very interesting and I've had such questions for a long time, actually. I've been surprised that people are not more concerned about such things.
However, right now the problems with AI seem so difficult that I'm not worried about these issues.
Peter J. Bentley: I think intelligence is extremely hard to define, even harder to measure, and human-level intelligence is a largely meaningless phrase. All life on Earth has human-level intelligence in one sense for we have all evolved for the same amount of time and we are equally able to survive in our appropriate niches and solve problems relevant to us in highly effective ways.
There is no danger from clever AI - only from stupid AI that is so bad that it kills us by accident. I wish we were on track to create something as clever as a frog. We have a long, long way to go. I agree with Pat Hayes on this subject.
I actually find the discussion a little silly. We are *much* more likely to all become a bunch of cyborgs completely reliant on integrated tech (including some clever computers) in the next 100 years. Computers won't be external entities, they will be a part of us. Many of us already can't live our modern lives without being plugged into their gadgets for most of their waking lives. Worry about that instead :)
I think your questions are very hard to answer in any rigorous sense, for they involve prediction of future events so far ahead that anything I say is likely to be quite inaccurate. I will try to answer some below, but these are really just my educated guesses.
Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?
Explanatory remark to Q1:
P(human-level AI by (year) | no wars ∧ no disasters ∧ beneficially political and economic development) = 10%/50%/90%
Nils Nilsson: Because human intelligence is so multi-faceted, your question really should be divided into each of the many components of intelligence. For example, on language translation, AI probably already exceeds the performance of many translators. On integrating symbolic expressions in calculus, AI (or computer science generally) is already much better than humans. AI does better on many planning and scheduling tasks. On chess, same! On the Jeopardy! quiz show, same!
A while back I wrote an essay about a replacement for the Turing test. It was called the "Employment Test." (See: http://ai.stanford.edu/~nilsson/OnlinePubs-Nils/General_Essays/AIMag26-04-HLAI.pdf) How many of the many, many jobs that humans do can be done by machines? I'll rephrase your question to be: When will AI be able to perform around 80% of these jobs as well or better than humans perform?
10% chance: 2030
50% chance: 2050
90% chance: 2100
David Plaisted: It seems that the development of human level intelligence is always later than people think it will be. I don't have an idea how long this might take.
Peter J. Bentley: That depends on what you mean by human level intelligence and how it is measured. Computers can already surpass us at basic arithmetic. Some machine learning methods can equal us in recognition of patterns in images. Most other forms of "AI" are tremendously bad at tasks we perform well. The human brain is the result of a several billion years of evolution at molecular scales to macro scales. Our evolutionary history spans unimaginable numbers of generations, challenges, environments, predators, etc. For an artificial brain to resemble ours, it must necessarily go through a very similar evolutionary history. Otherwise it may be a clever machine, but its intelligence will be in areas that do not necessarily resemble human intelligence.
Hector Levesque: No idea. There's a lot of factors beyond wars etc mentioned. It's tough to make these kind of predictions.
Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?
Explanatory remark to Q2:
P(human extinction | badly done AI) = ?
(Where 'badly done' = AGI capable of self-modification that is not provably non-dangerous.)
Nils Nilsson: 0.01% probability during the current century. Beyond that, who knows?
David Plaisted: I think people will be so concerned about the misuse of intelligent computers that they will take safeguards to prevent such problems. To me it seems more likely that disaster will come on the human race from nuclear or biological weapons, or possibly some natural disaster.
Peter J. Bentley: If this were ever to happen, it is most likely to be because the AI was too stupid and we relied on it too much. It is *extremely* unlikely for any AI to become "self aware" and take over the world as they like to show in the movies. It's more likely that your pot plant will take over the world.
Hector Levesque: Low. The probability of human extinction by other means (e.g. climate problems, micro biology etc) is sufficiently higher that if we were to survive all of them, surviving the result of AI work would be comparatively easy.
Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?
Explanatory remark to Q3:
P(superhuman intelligence within hours | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
Nils Nilsson: I'll assume that you mean sometime during this century, and that my "employment test" is the measure of superhuman intelligence.
hours: 5%
days: 50%
<5 years: 90%
David Plaisted: This would require a lot in terms of robots being able to build hardware devices or modify their own hardware. I suppose they could also modify their software to do this, but right now it seems like a far out possibility.
Peter J. Bentley: It won't happen. Has nothing to do with internet connections or speeds. The question is rather silly.
Hector Levesque: Good. An automated human level intelligence is achieved, it ought to be able to learn what humans know more quickly.
Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?
Explanatory remark to Q4:
How much money is currently required to mitigate possible risks from AI (to be instrumental in maximizing your personal long-term goals, e.g. surviving this century), less/no more/little more/much more/vastly more?
Nils Nilsson: Work on this problem should be ongoing, I think, with the work on AGI. We should start, now, with "little more," and gradually expand through the "much" and "vastly" as we get closer to AGI.
David Plaisted: Yes, some kind of ethical system should be built into robots, but then one has to understand their functioning well enough to be sure that they would not get around it somehow.
Peter J. Bentley: Humans are the ultimate in killers. We have taken over the planet like a plague and wiped out a large number of existing species. "Intelligent" computers would be very very stupid if they tried to get in our way. If they have any intelligence at all, they will be very friendly. We are the dangerous ones, not them.
Hector Levesque: It's always important to watch for risks with any technology. AI technology is no different.
Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?
Explanatory remark to Q5:
What existential risk (human extinction type event) is currently most likely to have the greatest negative impact on your personal long-term goals, under the condition that nothing is done to mitigate the risk?
Nils Nilsson: I think the risk of terrorists getting nuclear weapons is a greater risk than AI will be during this century. They would certainly use them if they had them -- they would be doing the work of Allah in destroying the Great Satan. Other than that, I think global warming and other environmental problems will have a greater negative impact than AI will have during this century. I believe technology can save us from the risks associated with new viruses. Bill Joy worries about nano-dust, but I don't know enough about that field to assess its possible negative impacts. Then, of course, there's the odd meteor. Probably technology will save us from that.
David Plaisted: There are risks with any technology, even computers as we have them now. It depends on the form of government and the nature of those in power whether technology is used for good or evil more than on the nature of the technology itself. Even military technology can be used for repression and persecution. Look at some countries today that use technology to keep their people in subjection.
Peter J. Bentley: No.
Hector Levesque: See above Q2. I think AI risks are smaller than others.
Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?
Nils Nilsson: Not as high as it should be. Some, like Steven Omohundro, Wendell Wallach and Colin Allen ("Moral Machines: Teaching Robots Right from Wrong"), Patrick Lin ("Robot Ethics"), and Ronald Arkin ("Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture") are among those thinking and writing about these problems. You probably know of several others.
David Plaisted: Probably not as high as it should be.
Peter J. Bentley: The whole idea is blown up out of all proportion. There is no real risk and will not be for a very long time. We are also well aware of the potential risks.
Hector Levesque: Low. Technology in the area is well behind what was predicted in the past, and so concern for risks is correspondingly low.
Q7: Can you think of any milestone such that if it were ever reached you would expect human-level machine intelligence to be developed within five years thereafter?
Nils Nilsson: Because human intelligence involves so many different abilities, I think AGI will require many different technologies with many different milestones. I don't think there is a single one. I do think, though, that the work that Andrew Ng, Geoff Hinton, and (more popularly) Jeff Hawkins and colleagues are doing on modeling learning in the neo-cortex using deep Bayes networks is on the right track.
Thanks for giving me the opportunity to think about your questions, and I hope to stay in touch with your work!
David Plaisted: I think it depends on the interaction of many different capabilities.
Peter J. Bentley: Too many advances are needed to describe here...
Hector Levesque: Reading comprehension at the level of a 10-year old.
Q8: Are you familiar with formal concepts of optimal AI design which relate to searches over complete spaces of computable hypotheses or computational strategies, such as Solomonoff induction, Levin search, Hutter's algorithm M, AIXI, or Gödel machines?
David Plaisted: My research does not specifically relate to those kinds of questions.