Interesting idea... though I still think you're wrong to step away from anthropomorphism, and 'necessary and sufficient' is a phrase that should probably be corralled into the domain of formal logic.

And I'm not sure this adds anything to Sternberg and Salter's definition: 'goal-directed adaptive behavior'.

I'm not an AGI researcher or developer (yet), but I think that the notion of a process steering the future into a constrained region is brilliant. It immediately feels much closer to implementation than any other definitions I've read before. Please continue posting on this topic. What I'm especially looking forward is anything on compression / abstraction over the search space.

Eliezer

One of your best ever posts IMHO and right on the nail. Of course this might be because I already agree with this definition but I left AI years ago, long before blogging and never wrote something like this up nor would I have said it so eloquently if I had.

Thom Blake 'goal-directed adaptive behaviour' is quite different and specifically it does not capture the notion of efficiency - of doing more with less.

Yep, definitely one of Eliezer's best posts.

Eliezer, out of curiosity, what was Lanier's response? Did he bite the bullet and say that wouldn't be an intelligence?

If you want to measure the intelligence of a system, I would suggest measuring its optimization power as before, but then dividing by the resources used. Or you might measure the degree of prior cognitive optimization required to achieve the same result using equal or fewer resources.

This really won't do. I think what you want is think in terms of a production function, which describe a system's output on a particular task as a function of its various inputs and features. Then we can talk about partial derivatives; rates at which output increases as a function of changes in inputs or features. The hard thing here is how to abstract well, how to collapse diverse tasks into similar task aggregates, and how to collapse diverse inputs and features into related input and feature aggregates. In particular, there is the challenge of how to identify which of those inputs or features count as its "intelligence."

IQ is a feature of human brains, and many have studied how the output of humans vary with IQ and task, both when other inputs are in some standard "reasonable" range, and when other inputs vary substantially. Even this is pretty hard; it is not at all clear to me how to broaden these abstractions even further, to talk about the "intelligence" of arbitrary systems on very wide ranges of tasks with wide ranges of other inputs and features.

I love this post, but there are several things in it that I have to take issue with.

You give the example of the asteroid and the tiger as things with different levels of intelligence, making the point that in both cases human beings are able to very rapidly respond with things that are outside the domain under which these entities are able to optimize. In the case of the asteroid, it can't optimize at all. In the case of the tiger, the human is more intelligent and so is better able to optimize over a wider domain.

However, the name of this blog is "overcoming bias," and one of its themes is the areas under which human intelligence breaks down. Just like you can toss poisoned meat to a tiger, there are things you can do to humans with similar effect. You can, for example, play on our strong confirmation bias. You can play tricks on our mind that exploit our poor intuition for statistics. Or, as in the case of Kasparov vs. Deep Blue, you can push us into a domain where we are slower than our opponent. Poisoned meat might do it for a tiger, but are economic bubbles and the "greater fool" pyramid that builds within them any different than this for humans? Aren't opaque mortgage-backed securities like the poisoned meat you toss to the tiger?

Secondly, I often object to the tendency of CS/AI folks to claim that evolution is incredibly slow. Simple iterative mutation and selection is slow and blind, but most of what goes on in evolution is not simple iterative mutation and selection. Evolution has evolved many strategies for evolution-- this is called the evolution of evolvability in the literature. These represent strategies for more efficiently finding local maxima in the fitness landscape under which these evolutionary processes operate. Examples include transposons, sexual reproduction, stress-mediated regulation of mutation rates, the interaction of epigenetics with the Baldwin effect, and many other evolvability strategies. Evolution doesn't just learn... it learns how to learn as well. Is this intelligence? It's obviously not human-like intelligence, but I think it qualifies as a form of very alien intelligence.

Finally, the paragraph about humans coming to parity with nature after 500 years of modern science is silly. Parity implies that there is some sort of conflict or contest going on. We are a part of the natural system. When we build roads and power plants, make vaccines, convert forests to farmland, etc. we are not fighting nature. We are part of nature, so these things are simply nature modifying itself just as it always has. Nuclear power plants are as "natural" as beehives and beaver dams. The "natural" vs. "artificial" dichotomy is actually a hidden form of anthropomorphism. It assumes that we are somehow metaphysically special.

g is a feature of human brains; IQ is a rough ranking of human brains with respect to g. I have not yet read of any means of actually measuring g , has anyone here got any references?

Eliezer, I have been trying to reread your series of ethics and morality posts in order, but am having trouble following the links backwards, I keep "finding" posts I missed. Any chance you could go and link them in the order you think they should be read?

Billswift, re rereading the series, check out Andrew Hay's list and associated graphs.

http://www.google.com/search?hl=en&q=tigers+climb+trees

On a more serious note, you may be interested in Marcus Hutter's 2007 paper "The Loss Rank Principle for Model Selection". It's about modeling, not about action selection, but there's a loss function involved, so there's a pragmatist viewpoint here, too.

Adam_Ierymenko: Evolution has evolved many strategies for evolution-- this is called the evolution of evolvability in the literature. These represent strategies for more efficiently finding local maxima in the fitness landscape under which these evolutionary processes operate. Examples include transposons, sexual reproduction,

Yes, Eliezer_Yudkowsky has discussed this before and calls that optimizaiton at the meta-level. Here is a representative post where he makes those distinctions.

Looking over the history of optimization on Earth up until now, the first step is to conceptually separate the meta level from the object level - separate the structure of optimization from that which is optimized. If you consider biology in the absence of hominids, then on the object level we have things like dinosaurs and butterflies and cats. On the meta level we have things like natural selection of asexual populations, and sexual recombination.

A quote from that post: "So animal brains - up until recently - were not major players in the planetary game of optimization; they were pieces but not players."

Again, no mention of sexual selection. Brains are players in the optimisation process. Animal brains get to perform selection directly. Females give male tyres a good kicking before choosing them - a bit like unit testing. Sexual selection is not even the only mechanism - natural selection can do this too.

Adam, you will find above that I contrasted human design to biology, not to nature.

Trying to toss a human a poisoned credit-default swap is more like trying to outrun a tiger or punch it in the nose - it's not an Outside Context Problem where the human simply doesn't understand what you're doing; rather, you're opposing the human on the same level and it can fight back using its own abilities, if it thinks of doing so.

Robin: This really won't do. I think what you want is think in terms of a production function, which describe a system's output on a particular task as a function of its various inputs and features. Then we can talk about partial derivatives; rates at which output increases as a function of changes in inputs or features. The hard thing here is how to abstract well, how to collapse diverse tasks into similar task aggregates, and how to collapse diverse inputs and features into related input and feature aggregates. In particular, there is the challenge of how to identify which of those inputs or features count as its "intelligence."

How do you measure output? As a raw quantity of material? As a narrow region of outcomes of equal or higher preference in an outcome space? Economists generally deal in quantities that are relatively fungible and liquid, but what about when the "output" is a hypothesis, a design for a new pharmaceutical, or an economic rescue plan? You can say "it's worth what people will pay for it" but this just palms off the valuation problem on hedge-funds or other financial actors, which need their own way of measuring the value somehow.

There's also a corresponding problem for complex inputs. As economists, you can to a large extent sit back and let the financial actors figure out how to value things, and you just measure the dollars. But the AI one tries to design is more in the position of actually being a hedge fund - the AI itself has to value resources and value outputs.

Economists tend to measure intermediate tasks that are taken for granted, but one of the key abilities of intelligence is to Jump Out Of The System and trace a different causal pathway to terminal values, eliminating intermediate tasks along the way. How do you measure fulfillment of terminal values if, for example, an AI or economy decides to eliminate money and replace it with something else? We haven't always had money. And if there's no assumption of money, how do you value inputs and outputs?

You run into problems with measuring the improbability of an outcome too, of course; I'm just saying that breaking up the system into subunits with an input-output diagram (which is what I think you're proposing?) is also subject to questions, especially since one of the key activities of creative intelligence is breaking obsolete production diagrams.

billswift, you might find this helpful: http://www.cs.auckland.ac.nz/~andwhay/graphsfiles/dependencygraphs.html#Something%20to%20protect.dot

With all this talk about poisoned meat and CDSes, I was inspired to draw this comic.

Eliezer's comment describes the importance of Jumping Out Of The System, which I attribute to the "cross-domain" aspect of intelligence, but I don't see this defined anywhere in the formula given for intelligence, which so far only covers "efficient" and "optimizer".

First, a quick-and-dirty description of the process: Find an optimization process in domain A (whether or not it help attain goals). Determine one or many mapping functions between domains A and B. Use a mapping to apply the optimization process to achieve a goal in domain B.

I think the heart of crossing domains is in the middle step - the construction of a mapping between domains. Plenty of these mappings will be incomplete, mere projections that lose countless dimensions, but they still occasionally allow for useful portings of optimization processes. This is the same skill as abstraction or generalization: turning data into simplified patterns, turning apples and oranges into numbers all the same. The measure of this power could then be the maximum distance from domain A to domain B that the agent can draw mappings across. Or maybe the maximum possible complexity of a mapping function (or is that the same thing)? Or the number of possible mappings between A and B? Or speed; it just would not do to run through every possible combination of projections between two domains. So here, then, is itself a domain that can be optimized in. Is the measure of being cross-domain just a measure of how efficiently one can optimize in the domain of "mapping between domains"?

Would Deep Blue no longer be intelligent at chess if it didn't have a human there to move the pieces on the board, or if it didn't signal the next move in a way that was readily intelligible to humans?

With no actuators at all, how would you distinguish the intelligence of Deep Blue from that of a heavy inert metal box?

Eliezer, even if you measure output as you propose in terms of a state space reduction factor, my main point was that simply "dividing by the resources used" makes little sense. Yes a production function formulation may abstract from some relevant details, but it is far closer to reality than dividing by "resources." Yes a market economy may help one to group and measure relevant inputs, and without that aid you'll have even more trouble grouping and measuring inputs.

With no actuators at all, how would you distinguish the intelligence of Deep Blue from that of a heavy inert metal box?

One tree near my home is an excellent chess player. If only it had some way to communicate...

I could teach any deciduous tree to play grandmaster chess if only I could communicate with it. (Well, not the really dumb ones.)

I have not yet read of any means of actually measuring g , has anyone here got any references?

There's no way to "actually measure g", because g has no operational definition beyond statistical analyses of IQ.

There have been some attempts to link calculated g with neural transmission speeds and how easily brains can cope with given problems, but there's been little success.

Re resources: if you look at how IQ tests work they usually attempt to factor out resources as best they can - no mechanical aids allowed, fixed time, etc. The only "wealth" you are permitted is what they can't strip off - education, genes, etc.

I imagine computer tests could be a bit like that - e.g. no net connection allowed, but you don't get penalised for having a faster CPU. For example, if you look at computer go contests, people get to supply their own hardware - and the idea is to win. You do not get many bonus points for using an Apple II.

Jeff Hawkins, in his book On Intelligence, says something similar to Eliezer. He says intelligence IS prediction. But Eliezer say intelligence is steering the future, not just predicting it. Steering is a behavior of agency, and if you cannot peer into the source code but only see the behaviors of an agent, then intelligence would necessarily be a measure of steering the future according to preference functions. This is behaviorism is it not? I thought behaviorism had been predicated as a useful field of inquiry in the cognitive sciences?

I can see where Eliezer is going with all this. The most moral/ethical/friendly AGI cannot take orders from any human, let alone be modeled on human agency to a large degree itself, and we also definitely do not want this agency to be a result of the same horrendous process of natural selection red in tooth and claw that created us.

That cancels out an anthropomorphic AI, cancels out evolution through natural selection, and it cancels out an unchecked oracle/genie type wish-granting intelligent system (though I personally feel that a controlled (friendly?) version of the oracle AI is the best option because I am skeptical with regard to Eliezer or anyone else coming up with a formal theory of friendliness imparted on an autonomous agent). ((Can an oracle type AI create a friendly AI agent? Is that a better path towards friendliness?))

Adam's comment above is misplace because I think Eliezer's recursively self-improving friendly intelligence optimization is a type of evolution, just not as blind as natural selection as has been played out through natural history on our earth.

Nice post.

And no, I don't think optimisation processes capture best what we generally mean by intelligence - but it captures best what we should mean by intelligence.

In practice it may have some weaknesses - depending on what exactly the valid domains are, there may be a more informative concept of intelligence in our universe - but it's good enough to do work with, while most definitions of intelligence aren't.

Would it do good to use something like sentience quotient, the quantity of bits per second per kg of matter a system can process, to assess the efficiency of a system ?

Of two systems having the same preferences, and the same sentience quotient, but whose optimization power isn't the same, one must then have a more efficient, smarter way of optimizing than the other ?

As for cross domain optimization, I don't see offhand how to mathematically charaterize different domains - and it is possible to define arbitrary domains anyway I think -

but if you have a nonrandom u, and are niverse, or environment, and are adapted to it, then if following your preferences you want to use all the information available locally in your environment, in your past light cone, you can only predict the course of your actions faster than will the universe given the implementation of physical laws upon matter if you can non destructively compress the information describing that environment; I guess, this works in any universe that has not reached maximal entropy, and the less entropy in that universe, the faster your speed for predicting future events will be compared to the speed of the universe implementing future events.

If you can't do that, then you have to use destructive compression to simplify your information about the environment into something you can manageably use to compute the future state of the universe following your actions, faster than the universe itself would implement them. There's a tradeoff between speed, simplicity, and precision, error rate in this case.

Just my immediate thoughts.

Jeff Hawkins observes that brains are constantly predicting the future.

That's quite consistent with the idea of brains acting as expected utility maximisers.

The brain predicts the future in order to detect if its model of the world needs updating when new sensory data arrives. If the data matches the model - no problem. If the data and the model conflict then the model needs updating.

In the expected utility framework, the brain has to predict the future anyway in order to judge the expected consequences of its actions. All it does is keep the results around for long enough to see if things go as it expected.

I often object to the tendency of CS/AI folks to claim that evolution is incredibly slow.

It is only nucleic evolution which is relatively slow. Cultural evolution has enormously magnified the rate of evolutionary change on the planet. In a century, skyscrapers have appeared out of nowhere, machines have probed the other planets, and the Earth has started to glow at night. Today, we can see evolutionary change taking place in real time - within an an individual's lifespan.

Kasparov competed against Deep Blue to steer the chessboard into a region where he won - knights and bishops were only his pawns

Were you trying to mix the literal and metaphorical here? Because I think that just his pawns were his pawns :)

I once heard a senior mainstream AI type suggest that we might try to quantify the intelligence of an AI system in terms of its RAM, processing power, and sensory input bandwidth.

Of course - this is correct. An AI system, like intelligence in general, is an algorithm and is thus governed by computational complexity theory and the physics of computation.

This at once reminded me of a quote from Dijkstra: "If we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger."

Code length is one fundemental quantitative measure (as in kolmogorv complexity), important in information theory, but it is not directly related nor to be confused with primary physical computational quantities such as space and time. Any pattern can be compressed - trading off time for space (expending energy to conserve mass).

If you want to measure the intelligence of a system, I would suggest measuring its optimization power as before, but then dividing by the resources used. Or you might measure the degree of prior cognitive optimization required to achieve the same result using equal or fewer resources. Intelligence, in other words, is efficient optimization.

The computational effeciency of one's intelligence algorithm is important, but effeciency is not the same as power, whether one is talking about heat engines or computational systems. Effeciency in computation is a measure of how much computation you get out for how much matter and energy you put in.

Intelligence, in typical english usage, does not connotate an effeciency measure - it connotates a power measure. If you have a super-computer AI that can think at a 3rd grade level, and you have a tiny cell phone AI that uses 1000 times less resources but thinks at a 2nd grade level, we'd still refer to the super-computer AI as being more intelligent, regardless of how effecient it is.

Intelligence is a computational power measure, but it is not a single scalar - it has temporal and spatial components (speed, depth, breadth, etc).

So if we say "efficient cross-domain optimization" - is that necessary to convey the wisest meaning of "intelligence", after making a proper effort to factor out anthropomorphism in ranking solutions?

I think "powerful generalized optimization" is closer to what you want, but one may also want to distinguish between static and dynamic intelligence (hard-coded vs adaptive). I'd also say that intelligence is a form of optimization, but optimization is a broader term. There are many computational optimization processes, most of which one would be hard pressed to call 'intelligent'.

Intelligence, in other words, is efficient optimization.

IMO, there's already plenty of space to include any efficiency criteria in the function being optimised.

So: there's no need to say "efficient" - but what you do, conventionally, need to say is something about the ability to solve a range of different types of problem. Include that and then consistently inefficient solutions get penalized automatically on problems where efficiency is specified in the utility function.

On the Concerning AI podcast, we addressed the question "What is Intelligence?" and settled on your definition as the one that would work for us. Thank you!

Is this definition circular? Let me explicit why that might be:

You define intelligent agents (or degree of intelligence) as "those physical systems that, using few resources, can optimize future states across many contexts (that is, when put in many different physical contexts, in which they need to face problems in different domains)" (consider also this more formal definition, which ignores the notion of resources).

But how are "resources" defined? There is a very natural notion of resources: energy, or negentropy. But I'm not sure this captures everything we care about here. There are many situations in which we care about the shape in which this energy is presented, since that is important to the useful usage of it (imagine finding a diamond in the desert, when you are dying of thirst). There are some ways in which energy can be stored (for example, in the nucleus of atoms?) which are very hard to make use of (without loads of other energy in other forms already available to take advantage of it). So we might naturally say something like "energy is that which, across many contexts, can be used to optimize for future states (across many preference orderings)". But what do we mean here by "can be used"? Probably something of the form "certain kinds of physical systems can take advantage of them, and transform them into optimization of future states". And I don't see how to define those physical systems other than as "intelligent agents".

A possible objection to this is that ultimately, when dealing with arbitrarily high "intelligence" and/or "variety of sources of energy available", we will have no problem with some energy sources being locked away from us: we have so many methods and machinery already available, that for any given source we can construct the necessary mechanisms to extract it. And so we don't need to resource to the definition of "resource" above, and can just use the natural and objective "energy" or "negentropy". But even if that's the case, I think for any finitely intelligent agent, with access to any finitely many energy sources, there will be some energy source it's locked away from. And even if that weren't the case for all such finite quantities, I do think in the intelligence regimes we care about many energy sources remain locked away.

You also seem to gesture at resources being equivalent to computation used (by the agent). Maybe we could understand this to mean something about the agent, and not its relationship to its surroundings (like the surrounding negentropy), such as "the agent is instantiated using at most such memory". But I'm not sure we can well-define this in any way other than "it fits in this small physical box". And then we get degenerations of the form "the most intelligent systems are black holes", or other expanding low-level physical processes.