Buying off AGI startups and then letting the relevant programmers program smart cars seems to me a quite good move to stall UFAI.
...has someone had a polite word with them about not killing all humans by sheer accident?
Shane Legg is familiar with AI risks. So is Jaan Tallinn, a top donor of MIRI, who is also associated with DeepMind. I suppose they will talk about their fears with Google.
Actually, there does seem to have been a very quiet press release about this acquisition resulting in a DeepMind ethics board.
So that's a relief.
Not to mention of course the Google employees that post on LW.
I didn't know there were any. My guess is that you have to be pretty high in the hierarchy to actually steer Google into a direction that would suit MIRI (under the assumption that people who agree with MIRI are in the minority).
I didn't know there were any.
Greetings from Dublin! You're right that the average employee is unlikely to matter, though.
Eliezed specifically mentioned Google in his Intelligence Explosion Microeconomics paper as the only named organization that could potentially start an intelligence explosion.
Larry Page has publicly said that he is specifically interested in “real AI” (Artificial General Intelligence), and some of the researchers in the field are funded by Google. So far as I know, this is still at the level of blue-sky work on basic algorithms and not an attempt to birth The Google in the next five years, but it still seems worth mentioning Google specifically.
In these interviews Larry Page gave years ago he constantly said that he wanted Google to become "the ultimate search engine" that would be able to understand all the information in the world. And to do that, Larry Page said, it would need to be 'true' artificial intelligence (he didn't say 'true', but it comes clear what he means in the context).
Here's a quote by Larry Page from the year 2007:
...We have some people at Google who are really trying to build artificial intelligence and to do it on a large scale and so on, and in fact, to make search better, to do the perfect job of search you could ask any query and it would give
It could be, like lukeprog said in October 2012, that Google doesn't even have "an AGI team".
Not that I know of, anyway. Kurzweil's team is probably part of Page's long-term AGI ambitions, but right now they're focusing on NLP (last I heard). And Deep Mind, which also has long-term AGI ambitions, has been working on game AI as an intermediate step. But then again, that kind of work is probably more relevant progress toward AGI than, say, OpenCog.
IIRC the Deep Mind folks were considering setting up an ethics board before Google acquired them, so the Google ethics board may be a carryover from that. FHI spoke to Deep Mind about safety standards a while back, so they're not totally closed to taking Friendliness seriously. I haven't spoken to the ethics board, so I don't know how serious they are.
Update: DeepMind will work under Jeff Dean at Google's search team.
And, predictably:
“Things like the ethics board smack of the kind of self-aggrandizement that we are so worried about,” one machine learning researcher told Re/code. “We’re a hell of a long way from needing to worry about the ethics of AI.”
...despite the fact that AI systems already fly planes, drive trains, and pilot Hellfire-carrying aerial drones.
NYTimes also links to LessWrong.
Quote:
Mr. Legg noted in a 2011 Q&A with the LessWrong blog that technology and artificial intelligence could have negative consequences for humanity.
So, to summarize, Google wants to build a potentially dangerous AI, but they believe they can keep it as an Oracle AI which will answer questions but not act independently. They also apparently believe (not without some grounding) that true AI is so computationally expensive in terms of both speed and training data that we will probably maintain an advantage of sheer physical violence over a potentially threatening unboxed oracle for a long time.
Except that they are also blatant ideological Singulatarians, so they're working to close that gap.
has someone had a polite word with them about not killing all humans by sheer accident?
Why do you think you have a better idea of the risks and solutions involved than they do, anyway? Superior AI expertise? Some superior expert-choosing talent of yours?
My suggestion to Google is to free up their brightest minds and tell them to talk to MIRI for 2 weeks, full-time. After the two weeks are over, let each of them write a report on whether Google should e.g. give them more time to talk to MIRI, accept MIRI's position and possibly hire them, or ignore them. MIRI should be able to comment on a draft of each of the reports.
I think this could finally settle the issue, if not for MIRI itself then at least for outsiders like me.
Well, that's sort of like having the brightest minds at CERN spend two weeks full time talking to some random "autodidact" who's claiming that LHC is going to create a blackhole that will devour the Earth. Society can't work this way.
Does that mean there is a terrible ignored risk? No, when there is a real risk, the brightest people of extreme and diverse intellectual accomplishment are the ones most likely to be concerned about it (and various "autodidacts" are most likely to fail to notice the risk).
Well, that's sort of like having the brightest minds at CERN spend two weeks full time talking to some random "autodidact" who's claiming that LHC is going to create a blackhole that will devour the Earth.
This is an unusual situation though. We have a lot of smart people who believe MIRI (they are not idiots, you've to grant them that). And you and me are not going to change their mind, ever, and they are hardly going to convince us. But if a bunch of independent top-notch people were to accept MIRI's position, then that would certainly make me assign a high probability to the possibility that I simply don't get it and that they are right after all.
Society can't work this way.
In the case of the LHC, independent safety reviews have been conducted. I wish this was the case for the kinds of AI risk scenarios imagined by MIRI.
Well, the other issue is also that people's opinions tend to be more informative of their own general plans than about the field in general.
Imagine that there's a bunch of nuclear power plant engineering teams - before nuclear power plants - working on different approaches.
One of the teams - not a particularly impressive one either - claimed that any nuclear plant is going to blow up like a hundred kiloton nuclear bomb, unless fitted with a very reliable and fast acting control system. This is actually how nuclear power plants were portrayed in early science fiction ("Blowups Happen", by Heinlein).
So you look at the blueprints, and you see that everyone's reactor is designed for a negative temperature coefficient of reactivity, in the high temperature range, and can't blow up like a nuke. Except for one team whose reactor is not designed to make use of a negative temperature coefficient of reactivity. The mysterious disagreement is explained, albeit in a very boring way.
Except for one team whose reactor is not designed to make use of a negative temperature coefficient of reactivity.
Except that this contrarian team, made of high school drop-outs, former theologians, philosophers, mathematicians and coal power station technicians, never produce an actual design, instead they spend all their time investigating arcane theoretical questions about renormalization in quantum field theory and publish their possibly interesting results outside the scientific peer review system, relying on hype to disseminate them.
"the brightest minds that build the first AI failed to understands some argument that even former theologians can follow"
This is related to something that I am quite confused about. There are basically 3 possibilities:
(1) You have to be really lucky to stumble across MIRI's argument. Just being really smart is insufficient. So we should not expect whoever ends up creating the first AGI to think about it.
(2) You have to be exceptionally intelligent to come up with MIRI's argument. And you have to be nowhere as intelligent in order to build an AGI that can take over the world.
(3) MIRI's argument is very complex. Only someone who deliberately thinks about risks associated with AGI could come up with all the necessary details of the argument. The first people to build an AGI won't arrive at the correct insights in time.
Maybe there is another possibility on how MIRI could end up being right that I have not thought about, let me know.
It seems to me that what all of these possibilities have in common is that they are improbable. Either you have to be (1) lucky or (2) exceptionally bright or (3) be right about a highly conjunctive hypothesis.
Comment by Juergen Schmidhuber:
Our former PhD student Shane Legg is co-founder of deepmind (with Demis Hassabis and Mustafa Suleyman), just acquired by Google for ~$500m. Several additional ex-members of the Swiss AI Lab IDSIA have joined deepmind, including Daan Wierstra, Tom Schaul, Alex Graves.
Peter Norvig is at least in principle aware of some of the issues; see e.g. this article about the current edition of Norvig&Russell's AIAMA (which mentions a few distinct way in which AI could have very bad consequences and cites Yudkowsky and Omohundro).
I don't know what Google's attitude is to these things, but if it's bad then either they aren't listening to Peter Norvig or they have what they think are strong counterarguments, and in either case an outsider having a polite word is unlikely to make a big difference.
Microsoft seems to focus on AI as well:
Q: You are in charge of more than 1000 research labs around the world.
What kind of thing are you focusing on?
Microsoft: A big focus right now, really on point for this segment, is artificial intelligence.
We have been very focused.
It is our largest investment area right now.
That interpretation seems tenuous. The sentence in which "pushed" is used:
The DeepMind-Google ethics board, which DeepMind pushed for, will devise rules for how Google can and can't use the technology.
suggests nothing more than that the proposal originated with DeepMind. I may as well imagine that Google's apparent amenability to the arrangement augurs well.
(Unless the article goes on to explain further? Not a subscriber.)
A Weyland-Yutani style outcome is a far bigger risk. EDIT: Does this mean anti-trust laws probably should've hit them a long time ago?
I was somewhat concerned when Google hired Kurzweil because he comes across as very Pollyanna-ish in his popular writings.
Now they're buying a company founded by the guy who created this game.
So I know we've already seen them buying a bunch of ML and robotics companies, but now they're purchasing Shane Legg's AGI startup. This is after they've acquired Boston Dynamics, several smaller robotics and ML firms, and started their own life-extension firm.
Is it just me, or are they trying to make Accelerando or something closely related actually happen? Given that they're buying up real experts and not just "AI is inevitable" prediction geeks (who shall remain politely unnamed out of respect for their real, original expertise in machine learning), has someone had a polite word with them about not killing all humans by sheer accident?
