Lest anyone get the wrong impression, I'm juggling multiple balls right now and can't give the latest Singularity debate as much attention as it deserves.  But lest I annoy my esteemed co-blogger, here is a down payment on my views of the Singularity - needless to say, all this is coming way out of order in the posting sequence, but here goes...

Among the topics I haven't dealt with yet, and will have to introduce here very quickly, is the notion of an optimization process.  Roughly, this is the idea that your power as a mind is your ability to hit small targets in a large search space - this can be either the space of possible futures (planning) or the space of possible designs (invention).  Suppose you have a car, and suppose we already know that your preferences involve travel.  Now suppose that you take all the parts in the car, or all the atoms, and jumble them up at random.  It's very unlikely that you'll end up with a travel-artifact at all, even so much as a wheeled cart; let alone a travel-artifact that ranks as high in your preferences as the original car.  So, relative to your preference ordering, the car is an extremely improbable artifact; the power of an optimization process is that it can produce this kind of improbability.

You can view both intelligence and natural selection as special cases of optimization:  Processes that hit, in a large search space, very small targets defined by implicit preferences.  Natural selection prefers more efficient replicators.  Human intelligences have more complex preferences.  Neither evolution nor humans have consistent utility functions, so viewing them as "optimization processes" is understood to be an approximation.  You're trying to get at the sort of work being done, not claim that humans or evolution do this work perfectly.

This is how I see the story of life and intelligence - as a story of improbably good designs being produced by optimization processes.  The "improbability" here is improbability relative to a random selection from the design space, not improbability in an absolute sense - if you have an optimization process around, then "improbably" good designs become probable.

Obviously I'm skipping over a lot of background material here; but you can already see the genesis of a clash of intuitions between myself and Robin.  Robin's looking at populations and resource utilization.  I'm looking at production of improbable patterns.

Looking over the history of optimization on Earth up until now, the first step is to conceptually separate the meta level from the object level - separate the structure of optimization from that which is optimized.

If you consider biology in the absence of hominids, then on the object level we have things like dinosaurs and butterflies and cats.  On the meta level we have things like natural selection of asexual populations, and sexual recombination.  The object level, you will observe, is rather more complicated than the meta level.  Natural selection is not an easy subject and it involves math.  But if you look at the anatomy of a whole cat, the cat has dynamics immensely more complicated than "mutate, recombine, reproduce".

This is not surprising.  Natural selection is an accidental optimization process, that basically just started happening one day in a tidal pool somewhere.  A cat is the subject of millions of years and billions of years of evolution.

Cats have brains, of course, which operate to learn over a lifetime; but at the end of the cat's lifetime, that information is thrown away, so it does not accumulate.  The cumulative effects of cat-brains upon the world as optimizers, therefore, are relatively small.

Or consider a bee brain, or a beaver brain.  A bee builds hives, and a beaver builds dams; but they didn't figure out how to build them from scratch.  A beaver can't figure out how to build a hive, a bee can't figure out how to build a dam.

So animal brains - up until recently - were not major players in the planetary game of optimization; they were pieces but not players.  Compared to evolution, brains lacked both generality of optimization power (they could not produce the amazing range of artifacts produced by evolution) and cumulative optimization power (their products did not accumulate complexity over time).  For more on this theme see Protein Reinforcement and DNA Consequentialism.

Very recently, certain animal brains have begun to exhibit both generality of optimization power (producing an amazingly wide range of artifacts, in time scales too short for natural selection to play any significant role) and cumulative optimization power (artifacts of increasing complexity, as a result of skills passed on through language and writing).

Natural selection takes hundreds of generations to do anything and millions of years for de novo complex designs.  Human programmers can design a complex machine with a hundred interdependent elements in a single afternoon.  This is not surprising, since natural selection is an accidental optimization process that basically just started happening one day, whereas humans are optimized optimizers handcrafted by natural selection over millions of years.

The wonder of evolution is not how well it works, but that it works at all without being optimized.  This is how optimization bootstrapped itself into the universe - starting, as one would expect, from an extremely inefficient accidental optimization process.  Which is not the accidental first replicator, mind you, but the accidental first process of natural selection.  Distinguish the object level and the meta level!

Since the dawn of optimization in the universe, a certain structural commonality has held across both natural selection and human intelligence...

Natural selection selects on genes, but generally speaking, the genes do not turn around and optimize natural selection.  The invention of sexual recombination is an exception to this rule, and so is the invention of cells and DNA.  And you can see both the power and the rarity of such events, by the fact that evolutionary biologists structure entire histories of life on Earth around them.

But if you step back and take a human standpoint - if you think like a programmer - then you can see that natural selection is still not all that complicated.  We'll try bundling different genes together?  We'll try separating information storage from moving machinery?  We'll try randomly recombining groups of genes?  On an absolute scale, these are the sort of bright ideas that any smart hacker comes up with during the first ten minutes of thinking about system architectures.

Because natural selection started out so inefficient (as a completely accidental process), this tiny handful of meta-level improvements feeding back in from the replicators - nowhere near as complicated as the structure of a cat - structure the evolutionary epochs of life on Earth.

And after all that, natural selection is still a blind idiot of a god.  Gene pools can evolve to extinction, despite all cells and sex.

Now natural selection does feed on itself in the sense that each new adaptation opens up new avenues of further adaptation; but that takes place on the object level.  The gene pool feeds on its own complexity - but only thanks to the protected interpreter of natural selection that runs in the background, and is not itself rewritten or altered by the evolution of species.

Likewise, human beings invent sciences and technologies, but we have not yet begun to rewrite the protected structure of the human brain itself.  We have a prefrontal cortex and a temporal cortex and a cerebellum, just like the first inventors of agriculture.  We haven't started to genetically engineer ourselves.  On the object level, science feeds on science, and each new discovery paves the way for new discoveries - but all that takes place with a protected interpreter, the human brain, running untouched in the background.

We have meta-level inventions like science, that try to instruct humans in how to think.  But the first person to invent Bayes's Theorem, did not become a Bayesian; they could not rewrite themselves, lacking both that knowledge and that power.  Our significant innovations in the art of thinking, like writing and science, are so powerful that they structure the course of human history; but they do not rival the brain itself in complexity, and their effect upon the brain is comparatively shallow.

The present state of the art in rationality training is not sufficient to turn an arbitrarily selected mortal into Albert Einstein, which shows the power of a few minor genetic quirks of brain design compared to all the self-help books ever written in the 20th century.

Because the brain hums away invisibly in the background, people tend to overlook its contribution and take it for granted; and talk as if the simple instruction to "Test ideas by experiment" or the p<0.05 significance rule, were the same order of contribution as an entire human brain.  Try telling chimpanzees to test their ideas by experiment and see how far you get.

Now... some of us want to intelligently design an intelligence that would be capable of intelligently redesigning itself, right down to the level of machine code.

The machine code at first, and the laws of physics later, would be a protected level of a sort.  But that "protected level" would not contain the dynamic of optimization; the protected levels would not structure the work.  The human brain does quite a bit of optimization on its own, and screws up on its own, no matter what you try to tell it in school.  But this fully wraparound recursive optimizer would have no protected level that was optimizing.  All the structure of optimization would be subject to optimization itself.

And that is a sea change which breaks with the entire past since the first replicator, because it breaks the idiom of a protected meta-level.

The history of Earth up until now has been a history of optimizers spinning their wheels at a constant rate, generating a constant optimization pressure.  And creating optimized products, not at a constant rate, but at an accelerating rate, because of how object-level innovations open up the pathway to other object-level innovations.  But that acceleration is taking place with a protected meta-level doing the actual optimizing.  Like a search that leaps from island to island in the search space, and good islands tend to be adjacent to even better islands, but the jumper doesn't change its legs.  Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there.

Imagine an economy without investment, or a university without language, a technology without tools to make tools.  Once in a hundred million years, or once in a few centuries, someone invents a hammer.

That is what optimization has been like on Earth up until now.

When I look at the history of Earth, I don't see a history of optimization over time.  I see a history of optimization power in, and optimized products out.  Up until now, thanks to the existence of almost entirely protected meta-levels, it's been possible to split up the history of optimization into epochs, and, within each epoch, graph the cumulative object-level optimization over time, because the protected level is running in the background and is not itself changing within an epoch.

What happens when you build a fully wraparound, recursively self-improving AI?  Then you take the graph of "optimization in, optimized out", and fold the graph in on itself.  Metaphorically speaking.

If the AI is weak, it does nothing, because it is not powerful enough to significantly improve itself - like telling a chimpanzee to rewrite its own brain.

If the AI is powerful enough to rewrite itself in a way that increases its ability to make further improvements, and this reaches all the way down to the AI's full understanding of its own source code and its own design as an optimizer... then even if the graph of "optimization power in" and "optimized product out" looks essentially the same, the graph of optimization over time is going to look completely different from Earth's history so far.

People often say something like "But what if it requires exponentially greater amounts of self-rewriting for only a linear improvement?"  To this the obvious answer is, "Natural selection exerted roughly constant optimization power on the hominid line in the course of coughing up humans; and this doesn't seem to have required exponentially more time for each linear increment of improvement."

All of this is still mere analogic reasoning.  A full AGI thinking about the nature of optimization and doing its own AI research and rewriting its own source code, is not really like a graph of Earth's history folded in on itself.  It is a different sort of beast.  These analogies are at best good for qualitative predictions, and even then, I have a large amount of other beliefs not yet posted, which are telling me which analogies to make, etcetera.

But if you want to know why I might be reluctant to extend the graph of biological and economic growth over time, into the future and over the horizon of an AI that thinks at transistor speeds and invents self-replicating molecular nanofactories and improves its own source code, then there is my reason:  You are drawing the wrong graph, and it should be optimization power in versus optimized product out, not optimized product versus time.  Draw that graph, and the results - in what I would call common sense for the right values of "common sense" - are entirely compatible with the notion that a self-improving AI thinking millions of times faster and armed with molecular nanotechnology, would not be bound to one-month economic doubling times.  Nor bound to cooperation with large societies of equal-level entities with different goal systems, but that's a separate topic.

On the other hand, if the next Big Invention merely infringed slightly on the protected level - if, say, a series of intelligence-enhancing drugs, each good for 5 IQ points, began to be introduced into society - then I can well believe that the economic doubling time would go to something like 7 years; because the basic graphs are still in place, and the fundamental structure of optimization has not really changed all that much, and so you are not generalizing way outside the reasonable domain.

I really have a problem with saying, "Well, I don't know if the next innovation is going to be a recursively self-improving AI superintelligence or a series of neuropharmaceuticals, but whichever one is the actual case, I predict it will correspond to an economic doubling time of one month."  This seems like sheer Kurzweilian thinking to me, as if graphs of Moore's Law are the fundamental reality and all else a mere shadow.  One of these estimates is way too slow and one of them is way too fast - he said, eyeballing his mental graph of "optimization power in vs. optimized product out".  If we are going to draw graphs at all, I see no reason to privilege graphs against times.

I am juggling many balls right now, and am not able to prosecute this dispute properly.  Not to mention that I would prefer to have this whole conversation at a time when I had previously done more posts about, oh, say, the notion of an "optimization process"...  But let it at least not be said that I am dismissing ideas out of hand without justification, as though I thought them unworthy of engagement; for this I do not think, and I have my own complex views standing behind my Singularity beliefs, as one might well expect.

Off to pack, I've got a plane trip tomorrow.

New Comment
21 comments, sorted by Click to highlight new comments since:

Intelligence has been at work in evolution for millions of years.

The choices of intelligent agents guide the path which evolution takes - via mechanisms which include the Baldwin effect and sexual selection. Humans are very literally the product of intelligent agents - not blind, mindless forces.

Eliezer, if AGI (or something else) ends up being designed without Friendliness, and Robin turns out to be right that this has no major harmful consequences, and about the doubling time, would you admit that his argument might be better, or would you say that he had been lucky?

@Tim Tyler:

The Baldwin effect is cute, but probably nowhere near as significant as sex or science.



No, I would not say he had been lucky.

This is a sufficiently low-probability event on my model that it simply means I'm stupid.

I would - I hope! - humbly beg Robin's indulgence and pray that he might impart to me the secret of my own stupidity.

Then I would be lynched by my former supporters.

Why do you ask?

Carry on with your long winding road of reasoning.

Of particular interest, which I hope you will dwell on: What does "self-improving" in the context of an AI program mean precisely? If there is a utility function involved, exactly what is it?

I also hope you start introducing some formal notation, to make your speculations on these topics less like science fiction.

Evolution less complex than a creature?

Lets break it down Variation - Even including all the things you don't mention such as variable mutation rates across the length of the genome, transposons, hox genes and segmented body types, your picture is still probably right

Selection - Since what selects which creature dies or reproduces is the whole rest of the ecosystem and physical surroundings, I would respectfully disagree that this is simpler than the creature.

Simple things cannot drive the complexity of things very high. Have a look at the simple environments of Tierra and other Alife models and see how they tend to peter out without anything very much of interest being created. Despite using the "same" variation and selection of evolution.

My main problem with your view of the human brain, is that the brain does not remain untouched while it learns. It doesn't do what you want to do, but still it isn't unchanged. I think it modifies itself down to the equivalent of machine code, it is just invisible to the conscious brain it is all being done lower down. There is no reason the conscious monkey part of the brain should have permission to alter the low level bits, if it doesn't understand anything about it. Nothing else could explain to me how it learns echolocation in the blind, repurposes the visual cortex for braille and expands the hippocampus of cabbies. Tell me what exactly is the protected structure of the brain that doesn't or can't change, because I just don't see it.

There might be bits protected from the top level, but then the top conscious level has no clue on how to alter it, and possibly never could considering the complexity differential between the low levels and the top levels, and the difficulties at altering a moving target. This does not mean they are not altered.


talk as if the simple instruction to "Test ideas by experiment" or the p

I think you're missing something really big here. There is such a thing as an optimal algorithm (or process). The most naive implementation of a process in much worse than the optimal, but infinitely better than nothing. Every successive improvement to the process asymptotically brings us closer to the optimal algorithm, but they can't give you the same order of improvement as the preceding ones. Just because we've gone from O(n^2) to O(n log(n)) in sorting algorithms doesn't mean we'll eventually get to O(1).

Aha! You say. But human brains are so inefficient that actually we haven't even gone a smidgeon of the path to the optimal algorithm and there is a ton more space to go. But computers already overcome many of the inefficiencies of human brains. Our brains do a decent job of pruning the search space up to the near-optimal solution, and computers take care of the work intensive step of going from near-optimal to optimal. And as our software gets better, we have to prune the search space less and less before we give the problem to the computer.

Of course, maybe we still have many orders of magnitude of improvement to go. But you can't just assume that.

" . . .if you want to know why I might be reluctant to extend the graph of biological and economic growth over time, into the future and over the horizon of an AI that thinks at transistor speeds and invents self-replicating molecular nanofactories and improves its own source code . . ."

Machine intelligence has long been rated in raw speed of calculation. There is plenty of processing power available. If I handed AI researchers a computer an order of magnitude faster than what they were working on, their failures would certainly fail faster, which is an advantage, but there's no reason to think that they would necessarily be able to create an AGI immediately. If we knew how to code an AGI, we could do it today and run it on slower machines. Sure, it might take 10-100 times as long to think as machines that we will have in a few years, but that is irrelevant to displaying intelligent thought.

The main advantage of transistor technology is consistency, not speed -- Transistors don't forget. Absolute knowledge retention is the advantage that Deep Blue has over regular chess players. The speed element simply makes it possible for it to play chess at a speed that doesn't make humans bored.

Of course, it may be that human-like intelligence and creativity requires a sort of messiness. I worry that absolute precision and (human-style) creativity are somewhat incompatible, at least in a single entity. Undoubtedly, however, an AGI could at least be constructed that is much better at both than we are.

"If there is a utility function involved, exactly what is it?"

I think this is, um, an unsolved problem.

I also hope you start introducing some formal notation, to make your speculations on these topics less like science fiction.

Science as Attire

Z.M.: interesting discussion. weapons of math destruction is a wickedly clever phrase. Still, I can hope for more than "FAI must optimize something, we know not what. Before we can figure out what to optimize we have to understand Recursive Self Improvement. But we can't talk about that because it's too dangerous."

Nick: Yes, science is about models, as that post says. Formal models. It does not seem unreasonable to hope that some are forthcoming. Surely that is the goal. The post you reference is complaining about people making a distinction between the theoretical possibility of different levels of intelligence without any rational basis. That doesn't seem to be the same thing as merely asking for a little precision in the definitions of "intelligence", "self improvement", and "friendliness".

GavinBrown, you haven't got the hang of the "optimization power in vs. optimized product out" graph. Raw processing power isn't the crucial element. Sure, such a beast would/will require huge processing power by today's standards, but the crux is the ability to access to its own code and turn up its optimization ratio on the fly. Once that process has a foothold, it won't need any more humans to come and put more RAM in a server for it. It won't need humans at all, which is the concern.

If we knew how to code an AGI, we could do it today and run it on slower machines. Sure, it might take 10-100 times as long to think as machines that we will have in a few years...

This is where a Kurzweil-style graph is useful: that factor of 10-100 would be eaten up in very little time by a multi-level self-optimizing intelligence of the type Eliezer is proposing. Just how quickly, (the rate that the new graph will outstrip the original) I imagine Eliezer has thought very hard about, though I wonder whether he has a firm estimate or just thinks 'way, way too quickly.'


You say "raw processing power isn't the crucial element." I said that speed "is irrelevant to displaying intelligent thought." We're actually saying pretty much the same thing! All I was really trying to argue was that phrases like "the speed of transistors" need to be replaced with phrases like "the accuracy, retention, and flexibility of transistors." I was -not- trying to argue against the principle that being able to turn the product of a process back on improving that process will result in an exponential growth curve of both intelligence and productivity.

We get plenty of calculation power out of the meat in our brains, but it is unfocused, inaccurate, biased, and forgetful. Performing lots of "flops" is not our weakness. The reason that recursive self-improvement is possible in transistor-based entities has nothing to do with speed--that's the only point that I'm trying to make.

We should be wary not because the machine can think thoughts faster than we can, but because it can think thoughts -better- than we can.

asking for a little precision in the definitions of "intelligence", "self improvement", and "friendliness"

bambi, there is a good chance that Eliezer's excellent Knowability of FAI can give that to you.

You have been asking good questions, and it seems to me that if you put enough effort into it, you will become able to contribute positively to the singularity -- and the singularity sure could use more positive input from women. Do you have a blog or web site?

Richard: Thanks for the link; that looks like a bunch of O.B. posts glommed together; I don't find it any more precise or convincing than anything here so far. Don't get me wrong, though; like the suggestive material on O.B. it is very interesting. If it simply isn't possible to get more concrete because the ideas are not developed well enough, so be it.

For the record, my nickname is taken from a character in an old Disney animated film, a (male) deer.

That doesn't seem to be the same thing as merely asking for a little precision in the definitions of "intelligence", "self improvement", and "friendliness".

If it were possible for us to define those concepts precisely, we would already be well on our way to creating AI. A problem can only be resolved once we can define it, after all.

We have no such definition of 'intelligence', 'self-improvement' is only an intuitive concept at present, and 'friendliness' is entirely vapor. The reason no one is giving you what you're asking for is that no one can at present - and admitting that is more than some are willing to do.


Surely what I am about to write is obvious, and probably old. During World War II, when physicists began to realize the destructive potential of nuclear weapons, Albert Einstein was chosen by his peers to approach President Roosevelt. Einstein was perhaps not the best informed of the group, but he was the best known, and was thought to be able to get Roosevelt's ear, as he did. In response, Roosevelt was able to convene all the greatest Western minds in physics, mathematics, and engineering to work together for a rapid solution to the problem. Clearly, the importance of the development of recursively self-improving super-human intelligence has got to be, almost by definition, greater than all other current problems, since it is the one project that would allow for the speedy solution of all other problems. Is there no famous person or persons in the field, able to organize his peers, and with access to the government such that an effort similar to the Manhattan Project could be accomplished? The AI Institute has one research fellow, and are looking for one more. They have a couple of fund-raisers, but most of the world is unaware of AI altogether. This won't get it done in a reasonable time-frame. Your competitors may well be backed by their governments.

While the eventual use of the Manhattan Project's discoveries is about as far from Friendly AI as imaginable, the power of super-human recursive AI is such that no matter by whom or where it is developed it will become the eminent domain of a government, much like the most powerful Cray computers. You might as well have their money and all the manpower right from the start, and the ability to influence it's proper use.

Can/will this be done?

It's interesting how seldom meta-level innovation has happened in biological evolution. Is this because it is too hard to evolve a system whose basic underpinnings can be changed by evolution? Or perhaps, it is not such a big win after all, and most changes at this level just break things?

One of the biggest mysteries of evolution is how we "evolved to evolve", how we went from the simplest replicators to our complex system with DNA, chromosomes, jumping genes, sexuality, and other mechanisms that seem to be fine tuned to create just enough variation such that natural selection can make progress. People who have worked on genetic algorithms have found that it doesn't just happen, you have to put a lot of thought into creating a structure such that a selection mechanism leads to meaningful improvement. Somehow evolution did the job.

And even more amazing to me, this biochemical infrastructure, which has probably been unchanged for at least hundreds of millions of years, and which was selected for on the basis of creating very primitive organisms, nevertheless had within itself sufficient variability to lead to human intelligence. Yet our 20,000-odd genes are not all that different from those of yeast cells. It is almost unbelievable that we got so lucky as to be included within the envelope of what was possible, when such a goal could not possibly have been relevant at the time the biochemical system was fixed.

Hal, here I think the anthropic principle goes a long way. I think your feeling of amazement is rooted more in our brain's susceptibility to feel amazed than in the particular set of information we have regarding how we came to be. Here I kind of agree with Eliezer, not so much that anything in nature is inherently "normal", but that it could be arbitrary (as opposed to particularly "amazing").

Chip, I don't know what you mean by "The AI Institute", but such discussion would be more on-topic at the SL4 mailing list than in the comments section of a blog posting about optimization rates.