Perhaps you've covered this, but what exactly does "friendly" mean? Do what I say? Including enslave my fellow humans?

Make me do what I should? Including forcing me to exercise regularly, and preventing me from eating what I want?

Support human improvement? Including reengineering my body/mind into whatever I want?

Make me happy? Including just inventing a really great drug?

I don't know what I want AI to do for me, so I have no idea how you would build a system that knows. And no idea how to resolve differences in opinion between myself and other humans.

The last student should use a fuctional language. He's right, the computer could easily be programmed to handle any order (as long as the IO is in the right sequence, and each variable is only assigned one value). So it's reasonable for him to expect that it would be.

Michael: Eliezer has at least in the past supported coherent extrapolated volition. I don't know if this is up-to-date with his current views.

I'd be curious to hear the top initial reactions.

Personally I'd be going with unforeseen consequences of however happiness is defined. But that's cause I live and breathe computers so I imagine early AI having a personality that is computerish and I understand at a very fundamental level what can happen when seemingly innocent instructions are carried out with infinitely relentless and pedantic determination.

I have thought for a while now if we want to survive the transition then the first really successful AI's are going to have to dramatically more tolerant and understanding than the average human, because you can pretty much guarantee that they are going to be subject to the irrational fear and hatred that humans generally inflict on anything new or different.

Gordon, humans respond in kind to hatred because we are programmed to by evolution, not because it is a universal response of all "ghosts". But of course an AI won't have compassion etc. either unless programmed to do so.

It also seems that if you take a utilitarian point of view, so none of this preserving stuff because it's unique or interesting only if it's useful, then once you have strong AI and decent robotics is there any use for humans or are they just technically obsolete?

And if the answer is that we're not really useful enough to justify the cost of our continued existence then should we try and define friendly to include preserving us or should we just swallow that metaphorical bullet and be happy that our creations will carry our legacy forward?

Simon, what I was meaning to get at was just that it/they are going to be on the receiving end of that human response and if they deal with it about as well as the average human would then we could be in big trouble.

"From a utilitarian perspective", where does the desire to do things better than can be done with the continued existence of humans come from? If it comes from humans, should not the desire to continue to exist also be given weight?

Also, if AI researchers anchor their expectations for AI on the characteristics of the average human then we could be in big trouble.

Gordon, I think you misunderstand what "utilitarian" means.

Here's my vision of this, as a short scene from a movie. Off my blog: The Future of AI

Minimal, very imprecise description of Friendly AI:

"A Friendly AI is one that does what we ought to want it to do."

Unfortunately, we humans have enough trouble figuring out what we do want, let alone what we should want. Hence, even if we could make a Really Powerful Optimization Process that was better than people at achieving goals, we don't know what goals to make it try to achieve.

Eliezer taught you rationality, so figure it out!

If I understand the research program under discussion, certain ideas are answered "somebody else will". e.g.

Don't build RSI, build AI with limited improvement capabilities (like humans) and use Moore's law to get speedup. "but somebody else will"

Build it so that all it does is access a local store of data (say a cache of the internet) and answer multiple choice questions (or some other limited function). Don't build it to act. "but somebody else will"

etc. every safety suggestion can be met with "somebody else will build an AI that does not have this safety feature".

So: make it Friendly. "but somebody else won't".

This implies: make it Friendly and help it take over the world to a sufficient degree that "somebody else" has no opportunity to build non-Friendly AI.

I think it is hugely unlikely that intelligence of the level being imagined is possible in anything like the near future, and "recursive self improvement" is very likely to be a lot more limited than projected (there's a limit to how much code can be optimized, P!=NP which severely bounds general search optimization, there's only so much you can do with "probably true" priors, and the physical world itself is too fuzzy to support much intellegent manipulation). But I could be wrong.

So, if you guys are planning to take over the world with your Friendly AI, I hope you get it right. I'm surprised there isn't an "Open Friendliness Project" to help answer all the objections and puzzles that commenters on this thread.

If Friendliness has already been solved, I'm reminded of Dr. Strangelove: it does no good to keep it a secret!

If it isn't, is it moral to work on more dangerous aspects (like reflectivity) without Friendliness worked out beforehand?

Here's my vision of this, as a short scene from a movie. Off my blog: The Future of AI

Problem with these AI's is that, in order to do something useful for us, they will for sure have some goals to attain, and be somewhat based on today's planning algorithms. Typical planning algorithms will plan with an eye for the constraints given and, rightfully, ignore everything else.

A contrived example, but something to consider: imagine a robot with artificial vision capabilities and capable of basic manipulation. You tell it to do something, but a human would be harmed by doing that action (by pushing an object, for example). One of the constraints is "do not harm humans", which was roughly translated as "if you see a human, don't exert big forces on it". The robot then happily adjusts its vision software to deliberately not see the human, as it would under any other condition adjust it the other way around to actively look for a human when it is not seen (you can imagine this as an important adjustable threshold for 3D object recognition or whatever, one that has to be adjusted to look for different kinds of objects). Yes this is a contrived example, but it's easy to imagine loopholes in AI design, and anyone who has worked closely with planning algorithms knows that if these loopholes exist, the algorithm will find them. As sad as it might sound, in medieval times, the way that religious people found to justify using slave labor was to classify slaves as "not actual people and thus, our laws are not applicable". It's only reasonable to assume that an AI can do this as well.

"Lets say we, as humans, placed some code on every server on the net that mimics a neuron. Is that going to become sentient? I have no idea. Probably not."

Ooo, even better, have the code recreate a really good hockey game. Have the code play the game in the demolished Winnipeg Arena, but make the sightlines better. And have the game between Russia and the Detroit Redwings. Have Datsyuk cloned and play for both teams. Of course, programs only affect the positions of silicon switches in a computer. To actually undemolish a construction site you need an actuator (magic) that affects the world outside the way lines of computer code flip silicon switches. The cloning the player part might be impossible, but at least it seems more reasonable than silicon switches that are conscious.

Well, I'm a noob, but I don't really understand why AI is so dangerous. If we created a superintelligent AI on supercomputer (or a laptop or whatever)even if it was malevolent, how could it take over the world/kill all humans or whatever? It would be a software program on a computer. Assuming we didn't give it the capacity to manufacture, how would it make all those microscopic killbots anyway?

kevin: Eliezer has written about that already. The AI could convice any human to let it out. See the AI box experiment ( http://yudkowsky.net/essays/aibox.html ). If it was connected to the Internet, it could crack the protein folding problem, find out how to build protein nanobots (to, say, build other nanobots), order the raw material (such as DNA strings) online) and convice some guy to mix it ( http://www.singinst.org/AIRisk.pdf ). It could think of something we can't even think of, like we could use fire if we were kept in a wooden prison (same paper).

An AI could screw us up just by giving bad advice. We'll be likely to trust it, because it's smart and we're too lazy to think. A modern GPS receiver can make you drive into a lake. An evil AI could ruin companies, start wars, or create an evil robot without lifting a finger.

Besides, it's more fun to create FAI and let it do what it wants than to build Skynet and then try to confine it forever. You'll still have only one chance to test it, whenever you decide to do that.

the physical world itself is too fuzzy to support much intellegent manipulation

I'm going to call mind projection on that one bambi. The world looks fuzzy to us, but only when we've got our human hats on. Put your AI programmer hat on and there's just 'stuff that could be used to compute'.

Right or wrong, the real answer is that we have no idea what a superhuman, self-improving intelligence would be like. Humans have transformed the face of the world in a certain time window. An intelligence orders of magnitude higher could...well, who knows? Whatever, saying '"recursive self improvement" is very likely to be a lot more limited than projected' is very, very dangerous indeed, even if accurate. The rest of your comment does take this into account.

I'm gobsmacked that it was a full 18 minutes before someone slam-dunked kevin's comment. Come on people, get it together.

Ben, you could be right that my "world is too fuzzy" view is just mind projection, but let me at least explain what I am projecting. The most natural way to get "unlimited" control over matter is a pure reductionist program in which a formal mathematical logic can represent designs and causal relationships with perfect accuracy (perfect to the limits of quantum probabilities). Unfortunately, combinatorial explosion makes that impractical. What we can actually do instead is redescribe collections of matter in new terms. Sometimes these are neatly linked to the underlying physics and we get cool stuff like f=ma but more often the redescriptions are leaky but useful "concepts". The fact that we have to leak accuracy (usually to the point where definitions themselves are basically impossible) to make dealing with the world tractable is what I mean by "the world is too fuzzy to support much intelligent manipulation". In certain special cases we come up with clever ways to bound probabilities and produce technological wonders... but transhumanist fantasies usually make the leap to assume that all things we desire can be tamed in this way. I think this is a wild leap. I realize most futurists see this as unwarranted pessimism and that the default position is that anything imaginable that doesn't provably violate the core laws of physics only awaits something smart enough to build it.

My other reasons for doubting the ultimate capabilities of RSI probably don't need more explanation. My skepticism about the imminence of RSI as a threat (never mind the overall ability of RSI itself) is more based on the ideas that 1) The world is really damn complicated and it will take a really damn complicated computer to make sense of it (the vast human data sorting machinery is well beyond Roadrunner and is not that capable anyway), and 2) there is still no beginning of a credible theory of how to make sense of a really damn complicated world with software.

I agree it is "very dangerous" to put a low probability on any particular threat being an imminent concern. Many such threats exist and we make this very dangerous tentative conclusion every day... from cancer in our own bodies to bioterror to the possibility that our universe is a simulation designed to measure how long it takes us to find the mass of the Higgs, after which we will be shut off.

That is all just an aside though to my main point, which was that if I'm wrong, the only conclusion I can see is that an explicit program to take over the world with a Friendly AI is the only reasonable option.

I approve of such an effort. If my skepticism is correct it will be impossible for decades at least; if I'm wrong I'd rather have an RSI that at least tried to be Friendly. It does seem that the Friendliness bit is more important than the RSI part as the start of such an effort.

kevin:

If we created a superintelligent AI on supercomputer (or a laptop or whatever)even if it was malevolent, how could it take over the world/kill all humans or whatever?

My attempt at answering that question can be found at http://www.saunalahti.fi/~tspro1/whycare.html . See also http://www.saunalahti.fi/~tspro1/objections.html#tooearly - it's also important to realize that there is a threat.

Along the lines of some of the commenters above, it's surely not telling Eliezer anything he doesn't already know to say that there are lots of reasons to be scared that a super-smart AI would start doing things we wouldn't like even without believing that an AI is necessarily a fundamentally malevolent ghost that will wriggle out of whatever restraints we put it in.

Like the "program" is giving instructions to a little ghost that sits inside the machine, which will look over your instructions and decide whether it likes them or not.

Fry: If you're programmed to jump off a bridge, would you do it?

Bender: Let me check my program... Yep.

(I am not linking to this, because it is a fearsome time-trap; you can Google if you dare.)

I dared.

I can't stop laughing now.

My intro to programming instructor did a pretty good exercise: he had us pair up, and we'd each write pseudo-code for the other person instructing them on how to make a peanut butter & jelly sandwich, step by step from a certain starting position (walk forward 5 steps, move hand out X inches, grasp jar, twist lid, etc). The person acting out the "code" had to do it exactly as written without making logical leaps (as refereed by the rest of the class) in order to simulate a computer.

Needless to say not a lot of sandwiches got completed. The point was well made though, I think.

To be honest, I do not understand at all how people can think that a very smart ghost is sitting in a computer, after all, they are not trying to break the processor in order to release a genie or give verbal commands to a turned off monitor. It's just very obvious to me that there is only a very stupid "ghost" in the computer who can only add bytes and that anything more complicated does not fit into his tiny calculator mind. However, I have two explanations why this is so obvious to me, firstly, my reduced empathy, which makes it easier for me to imagine a mechanism than someone's mind, and secondly, what I started programming at 9 years old is true , I don't remember if I already knew about zeros and ones, transistors, bytes and so on, but I probably knew, because even as a child I did not try to refer to a computer as intelligent, realizing that the only mind in a computer is the algorithm itself , the computer is just a big calculator that will not give the result of calculations until you press the "equal" button, even if it is obvious to you that the expression is complete, the calculator simply does not have the mind to understand this, this is just a mechanism, like a car or abacus , it would be strange to expect that they will start pressing the necessary buttons themselves if you write an explanation on a piece of paper.

I tutored college students who were taking a computer programming course. A few of them didn't understand that computers are not sentient.  More than one person used comments in their Pascal programs to put detailed explanations such as, "Now I need you to put these letters on the screen."  I asked one of them what the deal was with those comments. The reply:  "How else is the computer going to understand what I want it to do?"  Apparently they would assume that since they couldn't make sense of Pascal, neither could the computer.

There's been a phase change with the release of copilot, where this suddenly appears to work-- at least, for tasks like putting letters on the screen or assembling cookie recipes. "Waiter, there's a ghost in my machine!"