Why an Intelligence Explosion might be a Low-Priority Global Risk

[-]prase14y140

It is true that a Harpy Eagle can lift more than three-quarters of its body weight while the Boeing 747 Large Cargo Freighter has a maximum take-off weight of almost double its operating empty weight (I suspect that insects can do better).

Which doesn't (automatically) mean that the 747 has worse design than the eagle. Smaller things (constructions, machines, animals) are relatively stronger than bigger things not because of their superior design, but because physics of materials is not scale invariant (unless you somehow managed to scale the size of atoms too). If you made a 1000:1 scaled copy of an ant, it wouldn't be able to lift objects twenty times heavier.

0Logos0114y

It wouldn't be able to survive the crush of gravity, in fact.

0wedrifid14y

It wouldn't even be able to breathe, for that matter. They don't have lungs and just absorb and release gases through the exoskeleton. That change in the surface area to volume ratio would kill them.

-2[anonymous]14y

Or breathe.

[-]TheWakalix7y110

Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

We have now, depending on how you interpret "teach itself". It wasn't given anything but the rules and how to play against itself.

[-]JoshuaZ14y100

One issue I've mentioned before and I think is worth addressing is how much of the ability to quickly self-improve might depend on strict computational limits from theoretical computer science (especially complexity theory). If P != NP in a strong sense then recursive self-improvement many be very difficult.

More explicitly, many problems that are relevant for recursive self-improvement (circuit design and memory management for example) explicitly involve graph coloring and traveling salesman variants which are NP-hard or NP-complete. In that context, it could well be that designing new hardware and software will quickly hit diminishing marginal returns. If P, NP. coNP, PSPACE, and EXP are all distinct in a strong sense, then this sort of result is plausible.

There are problems with this sort of issue. One major one is that the standard distinctions between complexity classes are all in terms of Big-Os. So it could well be that the various classes are distinct but that the constants are small enough such that for all practical purposes one can do whatever one wants. There are also possible loopholes. For example, Scott Aaronson has shown that if one has access to closed time-like cu... (read more)

6lessdazed14y

That sounds suspiciously like hard work rather than speculation from unfalsifiable assumptions in natural language that can't actually be cashed out even in theory.

5CarlShulman14y

Eric Horvitz at Microsoft Research has expressed interest in finding complexity results for self-improvement in an intelligence explosion context. I don't know if much has come of it.

2amcknight14y

I don't really see why solving these kinds of difficult problems is relevant. A system could still recursively self-improve to solve a vast number of easier problems. That being said, I'd probably still be interested in anything relating complexity classes to intelligence.

-1JoshuaZ14y

Well, the point here is that it looks like the problems involved with recursive self-improvement themselves fall into the difficult classes. For example, designing circuitboards involves a version of the traveling salesman problem which is NP-complete. Similarly, memory management and design involves graph coloring which is also NP complete.

-2timtyler14y

We have already seen quite a bit of software and hardware improvement. We already know that it goes pretty fast. Maybe. Speed limits for technological evolution seem far off to me. The paucity of results in this area so far may mean that bounding progress rates is not an easy problem.

-1JoshuaZ14y

Many practical problems we actually have pretty decent limits. For example, the best algorithm to find gcds in the integers are provably very close to best possible. Of course, the actual upper bounds for many problems may be far off or they may be near. That's why this is an argument that we need to do more research into this question not that it is a slamdunk against runaway self-improvement.

0timtyler14y

In practice, much is down to how fast scientific and technological progress can accelerate. If seems fairly clear that progress is autocatalytic - and that the rate of progress ramps up with the number of scientists, which does not have hard limits. Algorithm limits seem to apply more to the question of how smart a computer program can become in an isolated virtual world. Matt Mahoney has looked at that area - though his results so far do not seem terribly interesting to me. I think one math problem is much more important to progress than all the other ones: inductive inference. We can see a long history of progress in solving that problem - and I think we can see that the problem extends far above the human level. One possible issue is whether progress will slow down as we head towards human capabilities. It seems possible (though not very likely) that we are making progress simply by coding our own inductive inference skills into the machines.

-1JoshuaZ14y

It might ramp up with increasing the number of scientists, but there are clear diminishing marginal returns. There are more scientists today at some major research universities than there were at any point in the 19th century. Yet we don't have people constantly coming up with ideas as big as say evolution or Maxwell's equations. The low hanging fruit gets picked quickly. I agree that Mahoney's work isn't so far very impressive. The models used are simplistic and weak. Many forms of induction are NP-hard and some versions are NP-complete so these sorts of limits are clearly relevant. Some other forms are closely related where one models things in terms of recognizing pseudorandom number generators. But it seems to me to be incorrect to identify this as the only issue or even that it is necessarily more important.. If for example one could factor large numbers more efficiently, an AI could do a lot with that if it got minimal internet access.

0XiXiDu14y

You are far more knowledgeable than me and a lot better at expressing possible problems with an intelligence explosion. Since the very beginning I wondered why nobody has written down what speaks against that possibility. Which is one of the reasons for why I even bothered to start arguing against it myself -- the trigger has been a deletion of a certain post which made me realize that there is a lot more to it (socially and psychologically) than the average research project -- even though I knew very well that I don't have the necessary background, nor patience, to do so in a precise and elaborated manner. Do people think that a skeptical inquiry of, and counterarguments against an intelligence explosion are not valuable?

0JoshuaZ14y

I don't know about that. The primary issue I've talked about limiting an intelligence explosion is computational complexity issues. That's a necessarily technical area. Moreover, almost all the major boundaries are conjectural. If P=NP in a practical way, than an intelligence explosion may be quite easy. There's also a major danger that in thinking/arguing that this is relevant, I may be engaging in motivated cognition in that there's an obvious bias to thinking that things close to one's own field are somehow relevant.

-1timtyler14y

Perhaps eventually - but much depends on how you measure it. In dollar terms, scientists are doing fairly well - there are a lot of them and they command reasonable salaries. They may not be Newtons of Einsteins, but society still seems to be prepared to pay them in considerable numbers at the moment. I figure that means there is still important stuff that needs discovering. As Eray Özkural once said: "Every algorithm encodes a bit of intelligence". However, some algorithms do more so than others. A powerful inductive inference engine could be used to solve factoring problems - but also, a huge number of other problems.

[-]gwern14y100

Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

AIXI-MC can teach itself Pacman; Pacman by default is single-player so the game implementation has to be already done for AIXI-MC. I suppose you could set up a pair of AIXI-MCs like is sometimes done for training chess programs, and the two will gradually teach each other chess.

2magfrump14y

Does anyone here actually have the ability to implement this? Has this been done? Googling "AIXI Chess" doesn't turn up interesting results. This seems like a great experiment for a grad student or maybe even an undergrad.

[-]RobertLumley14y80

Evolution versus Intelligence

I didn't read all of this post, unfortunately, I didn't have time before class. But I wanted to mention my thoughts on this section. This seemed like a very unfortunate analogy. For one, the specific example of flying is wildly biased against humans, who are orders of magnitude handicapped by the square cube law. Secondly, you can think of any number of arbitrary counterexamples where the advantage is on the side of intelligence. For a similar counterexample, intelligence has invented cars, trains, and boats which allow huma... (read more)

0RobertLumley14y

Guess someone beat me to this point.

[-]timtyler14y60

Even if the AGI is not told to hold, e.g. compute as many digits of Pi as possible, I consider it an far-fetched assumption that any AGI intrinsically cares to take over the universe as fast as possible to compute as many digits of Pi as possible. Sure, if all of that are presuppositions then it will happen, but I don’t see that most of all AGI designs are like that. Most that have the potential for superhuman intelligence, but who are given simple goals, will in my opinion just bob up and down as slowly as possible.

It seems to be a kind-of irrelevant a... (read more)

[-]Grognor14y50

Before I dive into this material in depth, a few thoughts:

First, I want to sincerely congratulate you on being (it seems to me) the first in our tribe to dissent.

Second, it seems your problem isn't with an intelligence explosion as a risk all on its own, but rather as a risk among other risks, one that is farther from being solved (both in terms of work done and in resolvability), and so this post could use a better title, i.e., "Why an Intelligence Explosion is a Low-Priority Global Risk", which does not a priori exclude SIAI from potential dona... (read more)

[-]jhuffman14y40

I can't really accept innovation as random noise. That doesn't seem to account for the incredible growth in the rate of new technology development. I think a lot of developments are in fact based on sophisticated analysis of known physical laws - e.g. a lot of innovation is engineering versus discovery. Many foundational steps do seem to be products of luck; such as the acceptance of the scientific method.

2CarlShulman14y

Trivially, we observe in the world that more innovations happen where there are more scientists, scientists with higher IQs, scientists spending more time on research, densely connected scientists, subjective time for scientists to think, etc. These are inputs that could be greatly boosted with whole brain emulations or not-even-superhuman AGI.

1timtyler14y

This sounds like Campbell's: As I have argued here it is a rather misleading idea. There may be a random component. As Steven Johnson says: "Chance favours the connected mind".

[-]timtyler14y40

given all of the apparent disadvantages of the blind idiot God, evolution was able to come up with altruism, something that works two levels above the individual and one level above society.

Much human altruism is fictional - humans are nice to other humans mostly because being nice pays.

There are low-level selection explanations for most of the genuine forms of human altruism. IMHO, the most promising explanations are:

Altruism is the result of manipulation by other humans;
Altruism is the result of manipulation by memes.
Altruism is the result of over

... (read more)

[-]billswift14y30

The critical similarity is that both rely on dumb luck when it comes to genuine novelty.

Someone pointed out that a sufficiently powerful intelligence could search all of design space rather than relying on "luck".

I read it on the Web, but can't find it - search really sucks when you don't have a specific keyword or exact phrasing to match.

1Manfred14y

I don't think that really captures the idea of intelligence. A sufficiently patient calculator can churn out the 10^10th digit of pi by caluclating pi, but an intelligent calculator would figure out how to do it in about a minute on my desktop computer. The point being that the label "dumb luck," while vaguely accurate, views discovery too much as a black box. Which is sort of ironic from this article.

[-][anonymous]14y30

This should be in Main. I'd much rather have this than "how I broke up with my girlfriend" there.

(Otherwise, I don't have much to say because I basically agree with you. I find your arguments kinda weak and speculative, but much less so than arguments for the other side. So your skepticism is justified.)

3wedrifid14y

That's a thought. I have now reversed my vote, in case the article is moved. I want my vote to have the full effect.

-7RobertLumley14y

[-]lessdazed14y30

Many humans are not even capable of handling the complexity of the brain of a worm.

I don't think that's the right reference class. We're not asking is something is sufficient, but if something is likely.

Our “irrationality” and the patchwork-architecture of the human brain might constitute an actual feature. The noisiness and patchwork architecture of the human brain might play a significant role in the discovery of unknown unknowns because it allows us to become distracted, to leave the path of evidence based exploration...The noisiness of the human b

... (read more)

[-]Donald Hobson5y20

A lot of this post sounds like fake ignorance. If you just read over it, you might think the questions asked are genuinely unknown, but if you think for a bit, you can see we have quite a lot of evidence and can give a rough answer.

We simply don’t know if intelligence is instrumental or quickly hits diminishing returns.

Well humans are doing ok for themselves, it seems to have accelerating returns up to the level of a smart human. Whats more, intelligence gets more valuable with increasing scale, and with cheaper compute. When controlling a roomba, you are ... (read more)

[-]amcknight14y20

Just a reminder that risk from AI can occur without recursive self-improvement. Any AGI with a nice model of our world and some goals could potentially be extremely destructive. Even if intelligence has diminishing returns, there is a huge hardware base to be exploited and a huge number of processors working millions of times faster than brains to be harnessed. Maybe intelligence won't explode in terms of self-improvement, but it can nevertheless explode in terms of pervasiveness and power.

[-]Gedusa14y20

Due to the absence of any signs of intelligence out there, especially paper-clippers burning the cosmic commons, we might conclude that unfriendly AI could not be the most dangerous existential risk that we should worry about.

I view this as one of the single best arguments against risks from paperclippers. I'm a little concerned that it hasn't been dealt with properly by SIAI folks - aside from a few comments by Carl Shulman on Katja's blog.

I suspect the answer may be something to do with anthropics - but I'm not really certain of exactly what it is.

2Kaj_Sotala14y

The Fermi Paradox was considered a paradox even before anybody started talking about paperclippers. And even if we knew for certain that superintelligence was impossible, the Fermi Paradox would still remain a mystery - it's not paperclippers (one possible form of colonizer) in particular that are hard to reconcile with the Fermi Paradox, it's the idea of colonizers in general. Simply the fact that the paradox exists says little about the likelyhood of paperclippers, though it does somewhat suggest that we might run into some even worse x-risk before the paperclippers show up. (What value you attach to that "somewhat" depends on whether you think it's reasonable to presume that we've already passed the Great Filter.)

0JoshuaZ14y

One important thing to keep in mind is that although Katja emphasizes this argument in a context of anthropics, the argument goes through even if one hasn't ever heard of anthropic arguments at all simply in terms of the Great Filter.

2CarlShulman14y

Only very weakly. Various potential early filters are quite unconstrained by evidence, so that our uncertainty spans many orders of magnitude. Abiogenesis, the evolution of complex life and intelligence, creation of suitable solar systems for life, etc could easily cost many orders of magnitude. Late-filter doomsdays like extinction from nukes or bioweapons would have to be exceedingly convergent (across diverse civilizations) to make a difference of many orders of magnitude for the Filter (essentially certain doom). Unless you were already confident that the evolution of intelligent life is common, or consider convergent doom (99.9999% of civilizations nuke themselves into extinction, regardless of variation in history or geography or biology) pretty likely the non-anthropic Fermi update seems pretty small.

1JoshuaZ14y

I'm not sure which part of the argument you are referring to. Are you talking about estimates that most of the Great Filter is in front of us? If so, I'd be inclined to tentatively agree. (Although I've been updating more in the direction of more filtration in front for a variety of reasons.) I was talking about that the observation that we shouldn't expect AI to be a substantial fraction of the Great Filter. Katja's observation in that context is simply a comment about what our light cone looks like.

3CarlShulman14y

OK. Sure. I was saying that this alone (sans SIA) is much less powerful if we assign much weight to early filters. Say (assuming we're not in a simulation) you assigned 20% probability to intelligence being common and visible (this does invoke observation selection problems inevitably, since colonization could preempt human evolution), 5% to intelligence being common but invisible (environmentalist Von Neumann probes enforce low-visibility; or maybe the interstellar medium shreds even slow starships) 5% to intelligence arising often and self-destructing, and 70% to intelligence being rare. Then you look outside, rule out "common and visible," and update to 6.25% probability of invisible aliens, 6.25% probability of convergent self-destruction in a fertile universe, and 87.5% probability that intelligence is rare. With the SIA (assuming we're not in a simulation, even though the SIA would make us confident that we were) we would also chop off the "intelligence is rare" possibility, and wind up with 50% probability of invisible aliens and 50% probability of convergent self-destruction. And, as Katja agrees, SIA would make us very confident that AI or similar technologies will allow the production of vast numbers of simulations with our experiences, i.e. if we bought SIA we should think that we were simulations, and in the "outside world" AI was feasible, but not to have strong conclusions about late or early filters (within many orders of magnitude) about the outside world.

1JoshuaZ14y

I agree with most of this. The relevant point is about AI in particular. More specifically, if an AGI is likely to start expanding to control its light cone at a substantial fraction of the speed of light, and this is a major part of the Filter, then we'd expect to see it. In contrast to something like nanotech for example that if it destroys civilization on a planet will be hard for observers to notice. Anthropic approaches (both SIA and SSA) argue for large amounts of filtration in front. The point is that observation suggests that AGI isn't a major part of that filtration if that's correct. An example that might help illustrate the point better. Imagine that someone is worried that the filtration of civilizations generally occurs due to them running some sort of physics experiment that causes a false vacuum collapse that expands at less than the speed of light (say c/10,000). We can discount the likelyhood of such an event because we would see from basic astronomy the result of the civilizations that have wiped themselves out in how they impact the stars near them.

0timtyler14y

Katja's blog post on the topic is here. The claim that the argument there is significant depends strongly on this - where I made some critical comments.

[-]timtyler14y20

Intelligence is no solution in itself, it is merely an effective searchlight for unknown unknowns and who knows that the brightness of the light increases proportionally with the distance between unknown unknowns? To enable an intelligence explosion the light would have to reach out much farther with each increase in intelligence than the increase of the distance between unknown unknowns. I just don’t see that to be a reasonable assumption.

We do have some data on historical increases in intelligence due to organic and cultural evolution. There's the fo... (read more)

[-]Kaj_Sotala14y10

I agree with the above, yet given all of the apparent disadvantages of the blind idiot God, evolution was able to come up with altruism, something that works two levels above the individual and one level above society. So far we haven’t been able to show such ingenuity by incorporating successes that are not evident from an individual or even societal position.

I don't understand this paragraph. What does "something that works two levels above the individual and one level above society" mean? Or the follow-up sentence?

[-]Curiouskid14y00

It seems that if you increase intelligence you also increase the computational cost of its further improvement and the distance to the discovery of some unknown unknown that could enable another quantum leap. It seems that you need to apply a lot more energy to get a bit more complexity.

I don't necessarily think it's true that you need to know an unknown unknown to reach a "quantum leap". This is a very qualitative reasoning about intelligence. You could simply increase the speed. Also, evolution didn't make intelligence by knowing some unknown unknown, it was a result of trial and error. Further intelligence improvement could use the same method, just faster.

[This comment is no longer endorsed by its author]Reply

[-]Drahflow14y00

The AGI would have to acquire new resources slowly, as it couldn’t just self-improve to come up with faster and more efficient solutions. In other words, self-improvement would demand resources. The AGI could not profit from its ability to self-improve regarding the necessary acquisition of resources to be able to self-improve in the first place.

If the AGI creates a sufficiently convincing business plan / fake company front, it might well be able to command a significant share of the world's resources on credit and either repay after improving or grab power and leave it at that.

[-]magfrump14y00

The first several points you make seem very weak to me, however starting with the section on embodied cognition the post gets better.

Embodied cognition seems to me like a problem for programmers to overcome, not an argument against FOOM. However it serves as a good basis for your point about constrained resources; I suspect that with sufficient time and leeway and access to AIM an AGI could become an extremely effective social manipulator. However this seems like the only avenue which it would obviously have the ability to get and process responses easil... (read more)

[-]timtyler14y00

If you accept the premise that life is not unique and special then one other technological civilisation in the observable universe should be sufficient to leave potentially observable traces of technological tinkering.
Due to the absence of any signs of intelligence out there, especially paper-clippers burning the cosmic commons, we might conclude that unfriendly AI could not be the most dangerous existential risk that we should worry about.

It seems like an argument for DOOM - but what if getting this far is simply very difficult?

Then we could be locally... (read more)

1JoshuaZ14y

How does this imply that there's a lot behind us? It could be that the technology that creates most of the great filter is something that generally arises close to the tech level where one is about to get largescale space travel. (I think the great filter is likely behind us, but looking just at the fact that it has taken us millions of years to get here is not by itself a decent argument that much of the filter is in fact in the past.)

0timtyler14y

There's around four-billion years worth of hold-ups behind us - and there's probably not much longer to wait. That doesn't show that most of the risk isn't ahead - that is a complex speculation based on too many factors to fit into this blog comment. The point is that you can't argue from no aliens to major future risks - since we might well be past the bulk of the filter.

[-]Logos0114y00

We simply don’t know if intelligence is instrumental or quickly hits diminishing returns.

General intelligence -- defined as the ability to acquire, organize, and apply information -- is definitionally instrumental. Greater magnitudes of intelligence yield greater ability to acquire, organize, and apply said information.

Even if we postulate an increasing difficulty or threshold of "contemplative-productivity" per new "layer" of intelligence, the following remains true: Any AGI which is designed as more "intelligent" than th... (read more)

1JoshuaZ14y

Intelligence is instrumentally useful but it comes at cost. Note that only a few tens of species have developed intelligence. This suggests that intelligence is in general costly. Even if more intelligence helps an AI's goals more that doesn't mean that acquiring more intelligence is easy or worth the effort. Yes, but I don't think many people seriously doubt this. Humans will likely to do this in a few years even without any substantial AGI work simply by genetic engineering and/or implants. This does not follow. It could be that it gets more and more difficult to design a superior intelligence. There may be diminishing marginal returns. (See my comment elsewhere in this thread for one possible example of what could go wrong.)

0Logos0114y

We seem to be talking past one another. Why do you speak in terms of evolution, where I was discussing engineered intelligence?

0JoshuaZ14y

I'm only discussing evolved intelligences to make the point that intelligence seems to be costly from a resource perspective.

0Logos0114y

Certainly. But evolved intelligences do not optimize for intelligence. They optimize for perpetuation of the genome. Constructed intelligence allows for systems that are optimized for intelligence. This was what I was getting at with the mentions of the fact that evolution does not optimize for what we optimize for; that there is no evolutionary equivalent of the atom bomb nor the Saturn-V rocket. So mentioning "ways that can go wrong" and reinforcing that point with evolutionary precedent seems to be rather missing the point. It's apples-to-oranges. After all: even if there is diminishing return on investment in terms of invested energy to achieve a more-intelligent design, once that new design is achieved it can be replicated essentially indefinitely.

-1JoshuaZ14y

Taking energy to get there isn't what is relevant in that context. The relevant issue is that being intelligent takes a lot of resources up. This is an important distinction. And the fact that evolution doesn't optimize for intelligence but for other goals isn't really relevant, given that an AGI presumably won't optimize itself for intelligence (a paperclip maximizer for example will make itself just as intelligent enough as it estimates is optimal for making paperclips everywhere). The point is that based on the data from one very common optimization process, it seems that intelligence is so resource intensive generally that being highly intelligent is simply very rarely worth it. (This evidence is obviously weak. The substrate matters as do other issue. But the basic point is sound.) Note incidentally that most of the comment was not about evolved intelligences. This is not an argument occurring in isolation. See especially the other two remarks made.

0Logos0114y

Quite correct, but you're still making the fundamental error of extrapolating from evolution to non-evolved intelligence without first correcting for the "aims"/"goals" of evolution as compared to designed intelligences when it comes to how designers might approach intelligence.

0Logos0114y

By the very dint of the fact that it is designed for the purpose of being intelligent, any AGI conceivably constructed by men would be optimized for intelligence; this seems a rather routinely heritable phenomenon. While a paperclip optimizer itself might not seek to optimize itself for intelligence, if we postulate that it is in the business of making a 'smarter' paperclip optimizer it will optimize for intelligence. Of course we cannot know the agency of any given point in the sequence; whether it will "make the choice" to recurse upwards. That being said, there's a real "non-sequitor" here to your dialogue, insofar as I can see. "The relevant issue is that being intelligent takes a lot of resources up". -- Compared to what, exactly? Roughly 2/3 of our caloric intake goes to our brain. Our brain is not well-optimized for intelligence. "[...] given that an AGI presumably won't optimize itself for intelligence" -- but whatever designed that AGI would have. I strongly disagree. The basic point is deeply flawed. I've already tried to say this repeatedly: evolution does not optimize for intelligence. Pointing at evolution's history with intelligence and saying, "aha! Optimization finds intelligence expensive!" is missing the point altogether: evolution should find intelligence expensive. It doesn't match what evolution "does". Evolution 'seeks' stable local minima to perpetuate replication of the genome. That is all it does. Intelligence isn't integral to that process; humans didn't need to be any more intelligent than we are in order to reach our local minima of perpetuation, so we didn't evolve any more intelligence. To attempt to extrapolate from that to what intelligence-seeking designers would achieve is missing the point on a very deep level: to extrapolate correctly from the 'lessons' evolution would 'teach us', one would have to postulate a severe selection pressure favoring intelligence. I don't see how you're doing that. Quite correct, but you're still m

0JoshuaZ14y

I don't understand your remark. No one is going to make an AGI whose goal is to become as intelligent as possible. Evolution is thus in this context one type of optimizer. Whatever one is optimizing for, becoming as intelligent as possible won't generally be the optimal thing to do even if becoming more intelligent does help achieve its goals more.

-2Logos0114y

I would. To which intelligence is extraneous. Consider then an optimizer which focuses on agency rather than perpetuation. To agency, intelligence is instrumental. By dint of being artificial and designed to be intelligent whatever goalset would have to be integrated into it would value that intelligence. It is not possible to intentionally design a trait into a system without that trait being valuable to the system. Intelligence is definitionally instrumental to an artificial general intelligence. Given sufficient time, any AGI capable of constructing a superior AGI will do so.

-2JoshuaZ14y

Are you trying to make sure a bad Singularity happens? No, intelligence is one tool among many which the blind idiot god uses. It is a particularly useful tool for species that can have widely varied environments. Unfortunately, it is a resource intensive tool. That's why Azathoth doesn't use except in a few very bright species. You seem to be confusing two different notions of intelligence. One is the either/or "is it intelligent" and the other is how intelligent it is. I'm not sure what you mean here.

2wedrifid14y

If Logos is seeking it then I assume it is not something that he considers bad. Presumably because he thinks intelligence is just that cool. Pursuing the goal necessarily results in human extinction and tiling the universe with computronium and I call that Bad but he should still answer "No". (This applies even if I go all cognitive-realism on him and say he is objectively wrong and that he is, in fact, trying to make sure a Bad Singularity happens.)

0Logos0114y

As I've asked elsewhere to resoundingly negative results: Why is it automatically "bad" to create an AGI that causes human exinction? If I value ever-increasingly-capable sentience... why must I be anthropocentric about it? If I were to view recursively improving AGI that is sentient as a 'child' or 'inheritor' of "the human will" -- then why should it be so awful if humanity were to be rendered obsolete or even extinct by it? I do not, furthermore, define "humanity" in so strict terms as to require that it be flesh-and-blood to be "human". If our FOOMing AGI were a "human" one -- I personally would find it in the range of acceptable outcomes if it converted the available carbon and silicon of the earth into computronium. Sure, it would suck for me -- but those of us currently alive already die, and over a long enough timeline the survival rate for even the clinically immortal drops to zero. I ask this question because I feel that it is relevant. Why is "inheritor" non-Friendly AGI "bad"? Caveat: It is possible for me to discuss my own motivations using someone else's valuative framework. So context matters. The mere fact that I would say "No" does not mean that I could never say "Yes -- as you see it."

2wedrifid14y

It isn't automatically bad. I just don't want it. This is why I said your answer is legitimately "No".

2Logos0114y

Fair enough. Honest question: If our flesh were dissolved overnight and we instead were instantiated inside a simulated environment -- without our permission -- would you consider this a Friendly outcome?

2wedrifid14y

Potentially, depending on the simulated environment.

0Logos0114y

Assume Earth-like or video-game-like (in the latter-case including 'respawns').

0wedrifid14y

Video game upgrades! Sounds good.

0Logos0114y

I believe you mean, "I'm here to kick ass and chew bubblegum... and I'm all outta gum!"

0Logos0114y

Given the available routes to general intelligence available to "the blind idiot god" due to the characteristics it does optimize for. We have a language breakdown here. The reason I said intelligence is 'extraneous' to evolution was because evolution only 'seeks out' local minima for perpetuation of the genome. What specific configuration a given local minimum happens to be is extraneous to the algorithm. Intelligence is in the available solution space but it is extraneous to the process. Which is why generalists often lose out to specialists in limited environments. (Read: the pigmy people who "went back to the trees".) Intelligence is not a goal to evolution; it is extraneous to its criteria. Intelligence is not the metric by which evolution judges fitness. Successful perpetuation of the genome is. Nothing else. Not at all. Not even remotely. I'm stating that any agent -- a construct (biological or synthetic) that can actively select amongst variable results; a thing that makes choices -- inherently values intelligence; the capacity to 'make good choices'. A more-intelligent agent is a 'superior' agent, instrumentally speaking. Any time there is an intelligent designed agent, the presence of said intelligence is a hard indicator of the agent valuing intelligence. Designed intelligences are "designed to be intelligent". (This is tautological). This means that whoever designed that intelligence spent effort and time on making it intelligent. That, in turn, means that its designer valued that intelligence. Whatever goalset the designer imparted into the designed intelligence, thusly, is a goalset that requires intelligence to be effected. Which in turn means that intelligence is definitionally instrumentally useful to a designed intelligence. What gives you trouble with it? Try rephrasing it and I'll 'correct' said rephrasing towards my intended meaning as possible, perhaps? I want to be understood. :)

1Cthulhoo14y

There isn't the need to have infinite recursion. Even if there is some cap on intelligence (due to resources optimization or something else we have yet to discover), the risk is still there if the cap isn't exceptionally near the human level. If the AI is to us what we are to chimps it may very well be enough.

0Logos0114y

Or, frankly, recursion at all. Say we can't make anything smarter than humans... but we can make them reliably smart, and smaller than humans. AGI bots as smart as our average "brilliant" guy with no morals and the ability to accelerate as only solid-state equipment can... is frankly pretty damned scary all on its own. (You could also count, under some auspices, "intelligence explosion" as meaning "an explosion in the number of intelligences". Imagine if for every human being the AGIs had 10,000 minds. Exactly what impact would the average human's mental contributions have? What, then, of 'intellectual labor'? Or manual labor?)

3Cthulhoo14y

Good point. In addition, supposing the AI is slightly smarter than humans and can easily replicate itself, Black Team effects could possibly be relevant (just an hypothesis, really, but still interesting to consider).

0TimS14y

Could you expand this a little further. I'm not afraid of amoral, fast-thinking, miniature Isaac Newtons unless they are a substantial EDIT: number (>1000 at the very least) or are not known about by the relevant human policy-makers. ETA: what it used to say at the edit was "faction of the human population (>1% at the very least)" TheOtherDave corrected my mis-estimate.

2RomeoStevens14y

have you read that alien message? http://lesswrong.com/lw/qk/that_alien_message/

0TimS14y

TheOtherDave showed that I mis-estimated the critical number. That said, there are several differences between my hypo and the story. 1) Most importantly, the difference between average human and Newton is smaller than the difference portrayed between aliens and humans. 2) There is a huge population of humans in the story, and I expressly limited my non-concern to much smaller populations. 3) The super-intelligents in the story do not appear to be know about by the relevant policy-makers (i.e. senior military officials) Not that it would matter in the story, but it seems likely to matter if the population of supers was much smaller.

0RomeoStevens14y

I'm not sure I see the point of the details you mention. The main thrust is that humans within the normal range given a million fold speedup (as silicon does) and unlimited collaboration would be a de facto super intelligence.

0TimS14y

The humans were not within the current normal range. The average was explicitly higher. And I think that the aliens average intelligent was lower than the current human average, although the story is not explicit on that point. And there were billions of super-humans. Let me put it this way: Google is smarter, wealthier, and more knowledgeable than I. But even if everyone at Google thought millions of times faster than everyone else, I still wouldn't worry about them taking over the world. Unless nobody else important knew about this capacity. AI is a serious risk, but let's not underestimate how hard it is to be as capable as a Straumli Perversion.

0RomeoStevens14y

the higher average does not mean that they were not within the normal range. they are not individually super human.

2TheOtherDave14y

I don't have a clear sense of how dangerous a group of amoral fast-thinking miniature Isaac Newtons might be but it would surprise me if there were a particularly important risk-evaluation threshold crossed between 70 million amoral fast-thinking miniature Isaac Newtons and a mere, say, 700,000 of them. Admittedly, I may be being distracted by the image of hundreds of thousands of miniature Isaac Newtons descending on Washington DC or something. It's a far more entertaining idea than those interminable zombie stories.

0TimS14y

You are right that 1% of the world population is likely too large. I probably should have said "substantial numbers in existence." I've adjusted my estimate, so amoral Newtons don't worry me unless they are secret or exist (>1000). And the minimum number gets bigger unless there is reason to think amoral Newtons will cooperate amongst themselves to dominate humanity.

3Logos0114y

I don't think the numbers I was referencing quite came across to you. I was postulating humans:AGIs :: 1:10,000 So not 70,000 Newtons or 70 million Newtons -- 70,000 Billion Newtons.

[-]timtyler14y-30

There is a significant difference between intelligence and evolution if you apply intelligence to the improvement of evolutionary designs:

Intelligence is goal-oriented.

Intelligence can think ahead.

Intelligence can jump fitness gaps.

Intelligence can engage in direct experimentation.

Intelligence can observe and incorporate solutions of other optimizing agents.

Much of this seems pretty inaccurate. The first three points are true, but not realy the issue - and explaning why would go uncomfortably close to the topic I am forbidden from talking about ... (read more)

2JoshuaZ14y

The last two points are different though: Intelligence can engage in direct, systematic experimentation. Evolution in contrast only experiments in so far as it happens to stumble across things. Similarly, while you are correct about symbiosis to a limited extent (and there are also examples of horizontal gene transfer), intelligence can look at another solution and without happening to pick up genes or forming a symbiotic relationship can go simply steal a functional strategy.

0timtyler14y

So: direct experimentation by animals is part of Skinnerian learning - which is hundreds of millions of years old. The results of the experiments go on to influence the germ line of organisms via selection. Would you claim that that is not part of "evolution"? Right - so I agree that differences in evolution are developing a result of the development of engineers - but observing and incorporating the "solutions of other optimizing agents" doesn't really capture the difference in question. Imitating the strategies of others is hardly a new development either - animals have been imitating each other since around our LCA with songbirds - that too is an ordinary part of evolution.

5JoshuaZ14y

Very few animals engage in anything that can be described as experimentation in any systematic fashion nor do they actually learn from the results in any fashion other than simple operant conditioning. Thus for example, if cats are placed in a box with a trick latch where they are given food after they get out, they gradually learn to hit the latch to the point where they can do so routinely. But, if one graphs the time it takes them on average to do so, they have a slow decline rather than a steep decline as one would expect if they were actually learning. If by experiment you mean "try lots of random things and not even understand which one actually helped" then yes they experiment. If one means experiment in a more useful fashion then very few species (great apes, corvids, keas, possibly elephants, possibly dolphins) do so. This is a much more limited form of imitation, where evolution has simply given them an instinct to imitate sounds around them. In some cases this can be quite impressive (e.g. the lyrebird can imitate other birds, as well as camera clicks, car alarms and chainsaws) but they are essentially just imitating noises around them. The set of species that can look at another entity solve a problem and then learn from that to solve the problem is much tinier (humans, some other great apes, corvids, keas, African Grey Parrots, dolphins, and not much else). There's a real difference in the type of imitation going on here.

0timtyler14y

Not one of the replies I expected - denying that animal trial-and-error learning is methodical enough to count as being "experimentation". That is probably true of some trial-and-error learning, but - as you seem to agree - some animal learning can be more methodical. I am still inclined to count experimentation as a "millions-of-years-old" phenomenon and part of conventional evolution - and not something to do with more recent developments involving engineering. I'd argue that even "local enhancement" - a very undemanding form of social learning - can result in incorporating the "solutions of other optimizing agents". Anyhow, again you seem to agree that animals have been doing such things for millions of years. So, this is still the domain of pretty conventional evolution.

0dlthomas14y

This smells funny.

-4timtyler14y

Sure: it conflicts with the popular misunderstanding of evolution being "blind" - and without foresight.

2dlthomas14y

Pinning it down a little, the part I object to is: It is trivially true that any change will have an influence, chaos theory and all. It seems unlikely that this influence will in any way relate to the results of the experimentation, in terms of production of variance or difference in heritability. Yet this seems to be what you are hand-wavily suggesting, without backing it up with math.

0timtyler14y

We do know a fair bit about that. The idea is that it influences the germ line via selection. If you perform experiments and the result is that you die - or fail to mate - the selective mechanism of information-transfer into the germ line is obvious. Milder outcomes have less dramatic effects - but still result in information transfer. The phenomenon has been studied under these names: * Genetic_assimilation * The Baldwin effect * The assimilate-stretch principle If we go as far as humans there are also things like: Biocultural evolution. That learned information and genetic information were aspects of a more general underlying phenomenon was observed by Semon (1904) and later there were pioneering contributions by B.F.Skinner - for example in Selection by Consequences. Skinner explicitly based his theory of learning on Darwin's theory of evolution. The idea was refined further in Dennet's Tower of Generate-and-Test.

2jimrandomh14y

You won't explain because the explanation is a neighbor of a topic that you were once asked to shut up about, three years ago, because the conversation had gotten repetitive? I think that's being too deferential.

7wedrifid14y

You really interpreted the move as deferential? Or is that you being tactful?

1timtyler14y

Thanks for that - but for now I'll continue to go with what the site's moderator said. I am not refusing to explain - the explanation I would offer now is much the same as it was back then. Anyone interested can go and look at the topic that I was forbidden from further discussing.

LESSWRONG
LW

LESSWRONG
LW

2

Why an Intelligence Explosion might be a Low-Priority Global Risk

2

2

Intelligence, a cornucopia?

Intelligence amplification, is it worth it?

Humans and the importance of discovery

Evolution versus Intelligence

Humans are biased and irrational

Embodied cognition and the environment

Necessary resources for an intelligence explosion

Artificial general intelligence, a single break-through?

Paperclip maximizers

Fermi paradox

Summary

Further reading