I guess these "how stupid I have been" posts are a welcome change to the "how smart I am" posts.

I've been stupid.

More generally, I'd like to see Overcoming Bias bloggers writing more about their current biases, either ones they struggle against, though not always successfully; or ones they have decided to surrender to.

Roko:

So allow me to object: not all configurations of matter worthy of the name "mind" are optimization processes. For example, my mind doesn't implement an optimization process as you have described it here.

I would actually say the opposite: Not all optimisation processes are worthy of the name "mind". Furthermore, your mind (I hope!) does indeed try to direct the future into certain limited supersets which you prefer. Unfortunately, you haven't actually said why you object to these things.

My problem with this post is simply that, well... I don't see what the big deal is. Maybe this is because I've always thought about AI problems in terms of equations and algorithms.

So do you now think that engineers can create a "Completely Alien Mind Design"? Do you have a feasible CAMD yourself?

I don't know if Eliezer2002 invented this reply on his own, or if he read it somewhere else. What about the concept of "optimization process"? Did you come to that idea yourself, or read about it elsewhere?

Writing fiction is a really useful tool for biting philosophical bullets. You can consider taboo things in a way your brain considers "safe", because it's just fiction, after all.

Eliezer, if you have time writing your book, one thing I'd really like to see is some sort of "Poor Richard's Almanack" style terse list of rationalist aphorisms. You've generated many, but have you collected them?

People could memorize them like SF geeks memorize the "litany against fear" from Dune ;-)

I guess these "how stupid I have been" posts are a welcome change to the "how smart I am" posts. I personally find the "how stupid I have been" posts useful because they demonstrate one path from stupid to smart, which is useful when knowing that I will probably run into similar realizations in the future. But I learn a lot more from the "how smart I am" posts because.. well, I'm not going to learn much by seeing that someone else made mistakes similar to the ones I used to (or still do) make, without seeing what they do about instead. This post wouldn't mean much to me without having actually learned about optimization processes, or knowing what the Outcome Pump was, etc. Like Eliezer said - "This may seem like an obvious point, if you've been following Overcoming Bias this whole time..." In other words - if you haven't read all the "how smart I am" posts, the "how stupid I have been posts" won't be nearly as useful.

That said... I do find myself in more suspense waiting for the next post in this series than the average post, though I suspect that's due more to the story-like nature of it than the actual material. And really, I don't know that I can say I look forward to the next installment in this series all that much more than posts in other long series like the quantum series or the series on words.

Bertrand Russell felt that such thought processes are native to humans:

What a man believes upon grossly insufficient evidence is an index into his desires -- desires of which he himself is often unconscious. If a man is offered a fact which goes against his instincts, he will scrutinize it closely, and unless the evidence is overwhelming, he will refuse to believe it. If, on the other hand, he is offered something which affords a reason for acting in accordance to his instincts, he will accept it even on the slightest evidence. The origin of myths is explained in this way.

Perhaps any reasoning one readily accepts is evidence of bias, and bears deeper examination. Could this be the value of educated criticism, the willingness of others to "give it to me straight", the impetus to fight against the unconscious tendencies of intelligence?

Roko, will you please exhibit a mind that you believe is not a utility maximizer? I am having trouble imagining one. For example, I consider a mind that maximizes the probability of some condition X coming to pass. Well, that is a utility maximizer in which possible futures satifying condition X have utility 1 whereas the other possible futures have utility 0. I consider a mind produced by natural selection, e.g., a mammalian mind or a human mind. Well, I see no reason to believe that that mind is not a utility maximizer with a complicated utility function that no one can describe completely, which to me is a different statement than saying the function does not exist.

Shane: Furthermore, your mind (I hope!) does indeed try to direct the future into certain limited supersets which you prefer.

Yes, it does. But I think we have to distinguish between "an agent who sometimes acts so as to produce a future possible world which is in a certain subset of possible states" and "an agent who has a utility function and who acts as an expected utility maximizer with respect to that utility function". The former is applicable to any intelligent agent, the latter is not. Yes, I am aware of the expected utility theorem of von Neumann and Morgenstern, but I think that decision theory over a fixed set of possible world states and a fixed language for describing properties of those states is not applicable to a situations where, due to increasing intelligence, that fixed set of states quickly becomes outmoded. But this really deserves a good, thorough post of it's own, but you can get some idea of what I am trying to say by reading ontologies, approximations and fundamentalists

Unfortunately, you haven't actually said why you object to these things.

So, my first objection, stated more clearly, says that we can usefully consider agents who are not expected utility maximizers. Clearly there are agents who aren't expected utility maximizers. It strikes me as dangerous to commit to building a superintelligent utility maximizer right now. I have my reasons for not liking utility maximizing agents; other people have their reasons for liking them, but at least let us keep the options open.

My second objection requires no further justification, and my third is really the same as the above: let us keep our options a bit more open.

Richard: Roko, will you please exhibit a mind that you believe is not a utility maximizer?

Consider the following toy universe U, which has 2 possible states - A and B, and where time is indexed by the natural numbers. The following pseudo-code does not embody a utility maximizing agent:

10: motor-output{A} 20: motor-output{B} 30: GOTO{10}

The following agent

10: motor-output{A} 20: END

does embody a utility maximizer with utility function U(A) = 1, U(B) = 0

Roko, why not:

U( alternating A and B states ) = 1 U( everything else ) = 0

"I've been stupid".

Come now. It's fine to realise you've made a mistake. But in itself this does not make you as smart as a Protector.

I guess these "how stupid I have been" posts are a welcome change to the "how smart I am" posts.

They're just another variation on that theme. The underlying assumption is that we can learn more from Eliezer's old mistakes, than from anyone else's current thinking.

Shane: "Roko, why not"

Let make Shane's reply more formal, so that Roko has something concrete to attack.

I did not have time to learn how to indent things on this blog, so I use braces to indicate indentation and semicolon to indicate the start of a new line.

Let state be a vector of length n such that for every integer time, (state[time] == A) or (state[time] == B).

U(state) == (sum as i goes from 0 to n in steps of 2) {2 if (state[i] == B) and (state[i+1] == A); 1 if (state[i] == B) or (state[i+1] == A); 0 otherwise}

If I remember correctly this mostly happened about a year earlier. I remember my intense relief when CAFAI came out in late 2000 or early 2001 in any event.

Eliezer, why do you call this awakening "naturalistic"? I don't see where your previous view was not "naturalistic".

I saw a dichotomy between them, the blindness of natural selection and the lookahead of intelligent foresight, [...] yet it was natural selection that created human intelligence, so that our brains, though not our thoughts, are entirely made according to the signature of natural selection.

Humans are the product of choices by intelligent agents. It would indeed be a shattering insight to discover that "blind" forces forged humanity - but that's not how it happened, the agents responsible posessed both vision and foresight - and were not "blind" in any reasonable sense of the word. See: http://alife.co.uk/essays/evolution_sees/

it drives me up the wall when people lump together natural selection and intelligence-driven processes as "evolutionary"

That's perfectly correct, according to the definition of evolution. Evolution is about variation and selection in populations of entities. There is no specification that variation should be random - or that selection should be unthinking. Evolution thus includes intelligent design among its fundamental mechanisms, by its very definition. For example, genetic engineering is a type of evolution. Check any evolution textbook for the definition of evolution.

@ shane: I was specifically talking about utility functions from the set of states of the universe to the reals, not from spacetime histories. Using the latter notion, trivially every agent is a utility maximizer, because there is a canonical embedding of any set X (in this case the set of action-perception pair sequences) into the set of functions from X to R. I'm attacking the former notion - where the domain of the utility function is the set of states of the universe.

Roko,

Who advocates that? Standard frameworks talk about world-histories, e.g. Omohundro's paper, which you use a lot. A hedonistic utilitarian wouldn't value a single state (a snapshot in time) of the universe, since experience of pleasure and the like are processes that take time to occur.

Nontransitive preferences don't translate to a utility function, and it would seem that a mind can have nontransitive preferences. Therefore, not all minds are utility maximizers.

(Does that make sense?)

@ carl: perhaps I should have checked through the literature more carefully. Can you point me to any other references on ethics using world-history utility functions with domain {world histories} ?

Roko, not all minds are good optimizers, not everything with a mind has everything within it proceeding according to optimization, and humans in particular are hacks. I think it should be obvious that I do not regard humans as expected utility maximizers, all bloggings considered, and I've written before about structurally difficult Friendly AI problems that are hard to interpret in terms of EU maximizing (e.g. "preserve individual self-determination"). Still, the insight is the insight.

Come now. It's fine to realise you've made a mistake. But in itself this does not make you as smart as a Protector.

If I'm as smart as Larry Niven, I'm as smart as a Protector. Vinge's Law: No character can be realistically depicted as being qualitatively smarter than the author. Next you'll be telling me that I'm not as smart as that over-enthusiastic child, Kimball Kinnison.

Eliezer,

What did you think of Blindsight (Peter Watts)? Pretty much the entire book is a depiction of humans or aliens much smarter than the author. (Myself, I enjoyed the story quite a bit, but wasn't impressed by the philosophizing about consciousness, which was rather trite and rang true not at all.)

Expected utility maximization is a powerful framework for modelling all intelligent agents - including humans.

Expected utility maximization as a framework is about as powerful as folk psychology. They both break down when you actually need to understand the inner workings of an intelligent agent, AI or human.

It seems a bit like complaining that microeconomics breaks down at the cellular level. Uh huh, but that's not the level at which microeconomics is intended to act as an explanatory framework.

Julian: "[O]ne thing I'd really like to see is some sort of 'Poor Richard's Almanack' style terse list of rationalist aphorisms. You've generated many, but have you collected them? [new graf] People could memorize them [...]"

Aren't the "Twelve Virtues" good enough?

EY kid,

I know you're deleting my posts, but you will see it first.

The actual true definition of 'intelligence' is the ability to form effective representations of the environment. My definition encompasses yours, but my definition is more general.

Don't say I didn't tell you so...

"It seems a bit like complaining that microeconomics breaks down at the cellular level. Uh huh, but that's not the level at which microeconomics is intended to act as an explanatory framework."

All the issues of how future AIs will actually perform in the real world depend on how far they diverge from utility maximizers. If they don't you'll get paper clippers, if they do they'll be more error prone human like and less likely to hard take off (due to error prone-ness).

Your comment struck me, as someone interested in the nuts and bolts of AI and also the future of the world, as someone saying to a bunch of quantum physicists, "Newtonian Dynamics is a really powerful framework". Which it is, but not a useful statement to make to a few quantum physicists. As most people, at the moment, are interested in prediction of divergence from utility maximizing and creation of AI, your statement was also not so helpful to the general discussion of intelligent agents, IMO.

All the issues of how future AIs will actually perform in the real world depend on how far they diverge from utility maximizers.

That seems highly inaccurate to me. AIs will more closely approximate rational utilitarian agents than current organisms - so the expected utility maximisation framework will become a better predictor of behaviour as time passes.

Obviously, the utility function of AIs will not be to produce paper clips.

Roko: Well, my thesis would be a start :-) Indeed, pick up any text book or research paper on reinforcement learning to see examples of utility being defined over histories.

If I'm as smart as Larry Niven, I'm as smart as a Protector. Vinge's Law: No character can be realistically depicted as being qualitatively smarter than the author.

Ridiculous. Is it not written "Anyone can find the right answer in thirty years"? Well then, anyone can find the right answer in the time it takes to write a book, and then portray a smarter character finding it in two seconds, sufficiently fast to kill their enemies.

Rolf, that was Niven's claim, but that seems to me as weak a form of faked genius as having the character invent neat gadgets. I'm not going to get an aura of scary formidability off that character.

And yet you were apparently impressed by a man getting the right answer to difficult problems in a book, where he had plenty of time to think about it. Had you read about someone performing such feats of mathematical rigour in mere minutes, would that not impress?

[i]That seems highly inaccurate to me. AIs will more closely approximate rational utilitarian agents than current organisms - so the expected utility maximisation framework will become a better predictor of behaviour as time passes.[/i]

The AI that I think humanity is likely to produce first will be a mass of hacks that work, that also hacks itself in a manner that works. There will be masses of legacy code, that it finds hard to get rid of, much as humans find ideas we have relied upon for reasoning for a long time hard to get rid of, if we can at all.

This isn't based on the fact that I think that we should build human like machines. But that only those can win in the the real world. There is no neat clean way of specifying a utility maximizer that eventually always wins, without infinite computing resources and supposing the computation done has no affect on the outside world. So we and other intelligent agents have to take mental short cuts, guess, make mistakes, get stuck in psychological cul-de-sacs. While AIs might up the number of ideas they play with to avoid those traps, it would be a trade off with looking at the links between ideas more thoroughly. For example you could devote more memory and processing time to finding cross correlations between inputs 1 - 1 million and the acquisition of utility, and looking at inputs 2million to 4 million as well. Either could be the right thing to do, so another hack is needed to decide which is done.

Unless you decide to rigorously prove which is the right thing to do, but then you are using up precious processing time and resources doing that. In short I see hacks everywhere in the future, especially towards the beginning, unless you can untangle the recursive knot caused by asking the question, "How much resources should I use, deciding how much resources I should use".

[i]Obviously, the utility function of AIs will not be to produce paper clips.[/i]

And obviously, I was referring to the single minded, focussed utility maximizer that Eliezer often uses in his discussions about AI.

The idea that superintelligences will more closely approximate rational utilitarian agents than current organisms is based on the idea that they will be more rational, suffer from fewer resource constraints, and be less prone to problems that cause them to pointlessly burn through their own resources. They will improve in these respects as time passes. Of course they will still use heuristics - nobody claimed otherwise.

I was referring to the single minded, focussed utility maximizer that Eliezer often uses in his discussions about AI.

This still sounds needlessly derogatory. Paper-clip maximisers have a dumb utility function, that's all. An expected utility maximiser is not necessarily "single minded": e.g. it may be able to focus on many things at once.

Optimisation is key to understanding intelligence. Criticising optimisers is criticising all intelligent agents. I don't see much point to doing that.

Antropic principle is also optimization process, different from evolution and human mind.

Also collective unconcsiones mind of human population which has created for example languages - is also some kind of optimization process.

Also we shoud mention science.

So in fact we live inside many mind-like processes - and often even do not mention it.

"The idea that superintelligences will more closely approximate rational utilitarian agents than current organisms is based on the idea that they will be more rational, suffer from fewer resource constraints, and be less prone to problems that cause them to pointlessly burn through their own resources."

But this in my book is not a firm basis. A firm basis requires a theory of AI. Only then can you talk about whether they will pointlessly burn through resources or not.

There are lots of things that are not obviously pointlessly burning through resources, but might still be doing so. Things like trying to prove P == NP if this happens to be improvable either way, modelling the future of the earth's climate without being able to take into account new anthropogenic influences (such as the change in albedo of solar cells, arcologies, or even unfolding nanoblooms).

Even spending time on this blog might be burning brain and computer cycles, perhaps we should be laying down our thoughts we want to survive in stone, in case we bomb ourselves back into the stone age.

A firm basis requires a theory of AI. Only then can you talk about whether they will pointlessly burn through resources or not.

We have the theory of evolution, we have hundreds of years of man-machine symbiosis to work from - and AI is probably now no longer terribly far off. IMHO, we have enough information to address this issue. Irrational AIs that run about in circles will sell poorly - so we probably won't build many like that.

I believe you have a problem with transparency here. You did not adequately link your revelation with the refutation of your previous thoughts. It may seem obvious to you that "squeezing the future into a narrow region" means that your old ideas "would have converted its future light cone into generic tools."

And, no, I do not think it is reasonable to ask me to read everything else you've ever said on this blog just to figure out the answer. Perhaps the explanation is too long for this post, but I would at least like some links.

When you say “a science-fiction story”, I am curious if it ever was finished. Is it HPMOR?