Eliezer, I can't imagine you really think I disagree with anything important in the above description. I do think it more likely than not that life started before Earth, and so it may have been much less than nine billion years when nothing happened. But that detail hardly matters to the overall picture here.
"In the kingdom of blind chance, the myopic optimization process is king." I sometimes chime in with pithy ways of summarizing arguments, so let me say: this sentence is a winner. I cannot top it.
Eliezer: Or the first replicator to catch on, if there were failed alternatives lost to history - but this seems unlikely, given the Fermi Paradox; a replicator should be more improbable than that, or the stars would teem with life already.
Why should you assume that they do not? Humans may not have detected other life forms, but you cannot deduce from that that they do not exist. They might not want us to detect them. It might not be possible for a civilisation to extend beyond its solar system.
"But Robin and I both suspect, I think, that the fate of the universe, and all those distant stars that outweigh us, will end up shaped by Life"
And how do you suspect that is going to happen?
To add to Abigail's point: Is there significant evidence that the critically low term in the Drake Equation isn't f_i (i.e. P(intelligence|life))? If natural selection on earth hadn't happened to produce an intelligent species, I would assign a rather low probability of any locally evolved life surviving the local sun going nova. I don't see any reasonable way of even assigning a lower bound to f_i.
"Or the first replicator to catch on, if there were failed alternatives lost to history - but this seems unlikely, given the Fermi Paradox; a replicator should be more improbable than that, or the stars would teem with life already."
So do you thing that the vast majority of The Big Filter is concentrated on the creation of a first replicator? What's the justification for that?
I don't yet see why exactly Eliezer is dwelling on the origin of replicators. As Robin said, it would have been very surprising if Robin had disagreed with any of it.
I guess that Eliezer's main points were these: (1) The origin of life was an event where things changed abruptly in a way that wouldn't have been predicted by extrapolating from the previous 9 billion years. Moreover, (2) pretty much the entire mass of the universe, minus a small tidal pool, was basically irrelevant to how this abrupt change played out and continues to play out. That is, the rest of the universe only mattered in regards to its gross features. It was only in that tidal pool that the precise arrangement of molecules had and will have far-reaching causal implications for the fate of the universe.
Eliezer seems to want to argue that we should expect something like this when the singularity comes. His conclusion seems to be that it is futile to survey the universe as it is now to try to predict detailed features of the singularity. For, if the origin of life is any guide, practically all detailed features of the present universe will prove irrelevant. Their causal implications will be swept aside by the consequences of some localized event that is hidden in some obscure corner of the world, below our awareness. Since we know practically nothing about this event, our present models can't take it into account, so they are useless for predicting the details of its consequences. That, at any rate, is what I take his argument to be.
There seems to me to be a crucial problem with this line of attack on Robin's position. As Eliezer writes of the origin of life,
The first replicator was the first great break in History - the first Black Swan that would have been unimaginable by any surface analogy. No extrapolation of previous trends could have spotted it - you'd have had to dive down into causal modeling, in enough detail to visualize the unprecedented search. Not that I'm saying I would have guessed, without benefit of hindsight - if somehow I'd been there as a disembodied and unreflective spirit, knowing only the previous universe as my guide - having no highfalutin' concepts of "intelligence" or "natural selection" because those things didn't exist in my environment, and I had no mental mirror in which to see myself - and indeed, who should have guessed it with short of godlike intelligence? When all the previous history of the universe contained no break in History that sharp? The replicator was the first Black Swan.
The difference with Robin's current position, if I understand it, is that he doesn't see our present situation as one in which such a momentous development is inconceivable. On the contrary, he conceives of it as happening through brain-emulation.
Eliezer seems to me to establish this much. If our present models did not predict an abrupt change on the order of the singularity, and if such a change nonetheless happens, then it will probably spring out of some very local event that wipes out the causal implications of all but the grossest features of the rest of the universe. However, Robin believes that our current models already predict a singularity-type event. If he's right (a big if!), then a crucial hypothesis of Eliezer's argument fails to obtain. The analogy with the origin of life that Eliezer makes in this post breaks down.
So the root of the difference between Eliezer and Robin seems to be this: Do our current models already give some significant probability to the singularity arising out of processes that we already know something about, e.g., the development of brain emulation? If so, then the origin of life was a crucially different situation, and we can't draw the lessons from it that Eliezer wants to.
I don't yet see why exactly Eliezer is dwelling on the origin of replicators.
Check with the title: if you are considering the possibility of a world takeover, it obviously pays to examine the previous historical genetic takeovers.
Oh, and it's called the Great Filter.
Robin, I didn't imagine you would disagree with my history, but I thought you might disagree with my interpretation or emphasis.
Tyrrell: I think Eliezer is going to say that all previous trends were guided by processes that didn't involve self modifications of intelligence. That's the game-changer, whether it's hand coded or uploaded.
We know we are in for a dramatic finale when the history of the universe is recounted as prologue. Fortunate for us that the searchable neighborhood has always held superior possibilities. And one would expect the endpoint of intelligence to be when this ceases to be the case. Perhaps there will be a sign at the end of the universe that says 'You can't get there from here. Sorry.'.
Eliezer, as someone who as been married for 21 years, I know better than to try to pick fights about tone or emphasis when more direct and clear points of disagreement can be found. :)
to say nothing of a whole multicellular C. elegans earthworm
It would probably better if you had said nothing of it. It eats dead rotting vegetable matter, which you have just hypothetically removed. Plants are replicators too! So it would die in short order and time would be reversed.
But the significant thing was not how much material was recruited into the world of replication; the significant thing was the search, and the material just carried out that search.
Search is significant, but it was not the only significant thing. What was searched was also significant. If by chance a brain that searched the space of good chess strategies had spontaneously appeared, it would not be important. What was searched was "what patterns are good for survival to date", not "what patterns that are expected to be good for survival". This is important, it is real first hand information, we cannot exist because of some delusive part of natures mind that thinks we are good at surviving so far, we have to be!
Our existence is first order information about the world. Our mental models are only second order, they are one removed from reality. We try and update them by testing our reality against what the models predict, but there might always be black swans the models don't see. The machinery that creates them has to have been useful for surviving to date and carrying out many tasks that helps that survival, but the models themselves do not necessarily have to be useful, right, correct or true. I think there will always be a flow of information from the first order bodies to the second order models.
Even a very small step forward in evolution, taken as a 'short-cut', would result in failure. - life changed the chemistry around it - headline is the relative abundance and influence of free oxygen relative to CO2.
The point is that the search is ALWAYS for near neighbour variants, and even then, the huge majority of these are failures.
The (seemingly) vastly improbable success of variants that are not near neighbours has, I think, to do with complexity and the concomitant law of unintended (in this context, 'unwelcome' would be a better word, since no intention is involved) consequences. The larger the step, the exponentially larger probability of corollary catastrophic implications.
The issue, of course, is not whether AI is a game-changer. The issue is whether it will be a game-changer soon and suddenly. I have been looking forward to somebody explaining why this is likely, so I've got my popcorn popped and my box of wine in the fridge.
Tim Tyler,
I don't yet see why exactly Eliezer is dwelling on the origin of replicators.Check with the title: if you are considering the possibility of a world takeover, it obviously pays to examine the previous historical genetic takeovers.
Right. I get the surface analogy. But it seems to break down when I look at its deeper structure.
I still don't get how Eliezer and (according to Eliezer) Robin think that life (or Life, whatever, a capital letter won't do the trick) is going to shape the stars and (if I understand correctly) the whole Universe. For all we know, despite all the replication and optimization, living things don't seem to have done much of that. Living things have shaped (to some extent) the surface of one planet. Big deal. If we changed the orbit of that same planet, I would think it impressive, compared to what we've done in the past. But still, big deal. It would only be one planet. Shape the Universe? How? And, if you don't know how, why do you think life will do it?
(Let me guess, intelligence can do anything. All right. Intelligence is God. But, why?)
"I still don't get how Eliezer and (according to Eliezer) Robin think that life (or Life, whatever, a capital letter won't do the trick) is going to shape the stars and (if I understand correctly) the whole Universe."
I don't get, how you can't get it. Really. It's a matter of a degree, how far away we make some changes. I trust, you wouldn't even if you could. But I would turn Jupiter and everything beyond to some useware, only if I could.
So "I" would remain, you would not. How difficult is to understand this?
I get what you're saying, Thomas, you're just answering another question. If some living thing were powerful enough to use a whole planet the size of Jupiter to its advantage, I agree with you that that would be the living thing that remains, rather than others. And then, if some living thing were even more powerful and could turn all the stars in the Universe into something useful, then that living thing would be the winner. But notice the "ifs". There's no if in "But Robin and I both suspect, I think, that the fate of the universe, and all those distant stars that outweigh us, will end up shaped by Life".
Felix: the power of our tools has no relation to our physical size or power. Stronger tools make stronger tools which make stronger tools still. You don't need to be the size of Jupiter to push a button, you don't need to be the size of a molecule to manipulate genetic code.
Yes, that's my point. We have traveled quite a long way, and there is no sign we couldn't go further with some luck. Producing useware from jupiters and stars is something one would expect. At least from a minority of civilizations, had been any of them around. But apparently, there is only a big void.
OTOH, they have still a lot of time to evolve to at least our current level from nothing, before we get there. That holds for the majority of the Universe. The Galaxy is another story, it's too small for this scenario. But the Universe as a whole is mostly more than 5 billion light years away. Our tomorrow Singularity could still be one of many, far apart. The SAI should ponder this immediately. What to do, not to be in war with others, coming from every space direction? The Ethics' space will be narrowed by this fact or guess.
It was the Era of Accidents, before the dawn of optimization. You'd only expect to see something with 40 bits of optimization if you looked through a trillion samples.
There's no way that this is true. Inanimate processes optimize too. Lightning strikes minimize the shortest path. Drainage patterns in mountainous regions find short paths between raindrops and their oceans. Cracks propagate so as to find the weakest path. The idea that only living systems optimize is just a mistake.
This bit is nonsense too:
There was no search but blind search. Everything from scratch, not even looking at the neighbors of previously successful points. Not hill-climbing, not mutation and selection, not even discarding patterns already failed. Just a random sample from the same distribution, over and over again.
Perhaps look into the principle of least action and Universal Darwinism.
Before Robin and I move on to talking about the Future, it seems to me wise to check if we have disagreements in our view of the Past. Which might be much easier to discuss - and maybe even resolve... So...
In the beginning was the Bang. For nine billion years afterward, nothing much happened.
Stars formed, and burned for long periods or short periods depending on their structure; but "successful" stars that burned longer or brighter did not pass on their characteristics to other stars. The first replicators were yet to come.
It was the Day of the Stable Things, when your probability of seeing something was given by its probability of accidental formation times its duration. Stars last a long time; there are many helium atoms.
It was the Era of Accidents, before the dawn of optimization. You'd only expect to see something with 40 bits of optimization if you looked through a trillion samples. Something with 1000 bits' worth of functional complexity? You wouldn't expect to find that in the whole universe.
I would guess that, if you were going to be stuck on a desert island and you wanted to stay entertained as long as possible, then you should sooner choose to examine the complexity of the cells and biochemistry of a single Earthly butterfly, over all the stars and astrophysics in the visible universe beyond Earth.
It was the Age of Boredom.
The hallmark of the Age of Boredom was not lack of natural resources - it wasn't that the universe was low on hydrogen - but, rather, the lack of any cumulative search. If one star burned longer or brighter, that didn't affect the probability distribution of the next star to form. There was no search but blind search. Everything from scratch, not even looking at the neighbors of previously successful points. Not hill-climbing, not mutation and selection, not even discarding patterns already failed. Just a random sample from the same distribution, over and over again.
The Age of Boredom ended with the first replicator.
(Or the first replicator to catch on, if there were failed alternatives lost to history - but this seems unlikely, given the Fermi Paradox; a replicator should be more improbable than that, or the stars would teem with life already.)
Though it might be most dramatic to think of a single RNA strand a few dozen bases long, forming by pure accident after who-knows-how-many chances on who-knows-how-many planets, another class of hypotheses deals with catalytic hypercycles - chemicals whose presence makes it more likely for other chemicals to form, with the arrows happening to finally go around in a circle. If so, RNA would just be a crystallization of that hypercycle into a single chemical that can both take on enzymatic shapes, and store information in its sequence for easy replication.
The catalytic hypercycle is worth pondering, since it reminds us that the universe wasn't quite drawing its random patterns from the same distribution every time - the formation of a longlived star made it more likely for a planet to form (if not another star to form), the formation of a planet made it more likely for amino acids and RNA bases to form in a pool of muck somewhere (if not more likely for planets to form).
In this flow of probability, patterns in one attractor leading to other attractors becoming stronger, there was finally born a cycle - perhaps a single strand of RNA, perhaps a crystal in clay, perhaps a catalytic hypercycle - and that was the dawn.
What makes this cycle significant? Is it the large amount of material that the catalytic hypercycle or replicating RNA strand could absorb into its pattern?
Well, but any given mountain on Primordial Earth would probably weigh vastly more than the total mass devoted to copies of the first replicator. What effect does mere mass have on optimization?
Suppose the first replicator had a probability of formation of 10-30. If that first replicator managed to make 10,000,000,000 copies of itself (I don't know if this would be an overestimate or underestimate for a tidal pool) then this would increase your probability of encountering the replicator-pattern by a factor of 1010, the total probability going up to 10-20. (If you were observing "things" at random, that is, and not just on Earth but on all the planets with tidal pools.) So that was a kind of optimization-directed probability flow.
But vastly more important, in the scheme of things, was this - that the first replicator made copies of itself, and some of those copies were errors.
That is, it explored the neighboring regions of the search space - some of which contained better replicators - and then those replicators ended up with more probability flowing into them, which explored their neighborhoods.
Even in the Age of Boredom there were always regions of attractor-space that were the gateways to other regions of attractor-space. Stars begot planets, planets begot tidal pools. But that's not the same as a replicator begetting a replicator - it doesn't search a neighborhood, find something that better matches a criterion (in this case, the criterion of effective replication) and then search that neighborhood, over and over.
This did require a certain amount of raw material to act as replicator feedstock. But the significant thing was not how much material was recruited into the world of replication; the significant thing was the search, and the material just carried out that search. If, somehow, there'd been some way of doing the same search without all that raw material - if there'd just been a little beeping device that determined how well a pattern would replicate, and incremented a binary number representing "how much attention" to pay to that pattern, and then searched neighboring points in proportion to that number - well, that would have searched just the same. It's not something that evolution can do, but if it happened, it would generate the same information.
Human brains routinely outthink the evolution of whole species, species whose net weights of biological material outweigh a human brain a million times over - the gun against a lion's paws. It's not the amount of raw material, it's the search.
In the evolution of replicators, the raw material happens to carry out the search - but don't think that the key thing is how much gets produced, how much gets consumed. The raw material is just a way of keeping score. True, even in principle, you do need some negentropy and some matter to perform the computation. But the same search could theoretically be performed with much less material - examining fewer copies of a pattern, to draw the same conclusions, using more efficient updating on the evidence. Replicators happen to use the number of copies produced of themselves, as a way of keeping score.
But what really matters isn't the production, it's the search.
If, after the first primitive replicators had managed to produce a few tons of themselves, you deleted all those tons of biological material, and substituted a few dozen cells here and there from the future - a single algae, a single bacterium - to say nothing of a whole multicellular C. elegans earthworm with a 302-neuron brain - then Time would leap forward by billions of years, even if the total mass of Life had just apparently shrunk. The search would have leapt ahead, and production would recover from the apparent "setback" in a handful of easy doublings.
The first replicator was the first great break in History - the first Black Swan that would have been unimaginable by any surface analogy. No extrapolation of previous trends could have spotted it - you'd have had to dive down into causal modeling, in enough detail to visualize the unprecedented search.
Not that I'm saying I would have guessed, without benefit of hindsight - if somehow I'd been there as a disembodied and unreflective spirit, knowing only the previous universe as my guide - having no highfalutin' concepts of "intelligence" or "natural selection" because those things didn't exist in my environment, and I had no mental mirror in which to see myself - and indeed, who should have guessed it with short of godlike intelligence? When all the previous history of the universe contained no break in History that sharp? The replicator was the first Black Swan.
Maybe I, seeing the first replicator as a disembodied unreflective spirit, would have said, "Wow, what an amazing notion - some of the things I see won't form with high probability, or last for long times - they'll be things that are good at copying themselves, instead. It's the new, third reason for seeing a lot of something!" But would I have been imaginative enough to see the way to amoebas, to birds, to humans? Or would I have just expected it to hit the walls of the tidal pool and stop?
Try telling a disembodied spirit who had watched the whole history of the universe up to that point about the birds and the bees, and they would think you were absolutely and entirely out to lunch. For nothing remotely like that would have been found anywhere else in the universe - and it would obviously take an exponential and ridiculous amount of time to accidentally form a pattern like that, no matter how good it was at replicating itself once formed - and as for it happening many times over in a connected ecology, when the first replicator in the tidal pool took such a long time to happen - why, that would just be madness. The Absurdity Heuristic would come into play. Okay, it's neat that a little molecule can replicate itself - but this notion of a "squirrel" is insanity. So far beyond a Black Swan that you can't even call it a swan anymore.
That first replicator took over the world - in what sense? Earth's crust, Earth's magma, far outweighs its mass of Life. But Robin and I both suspect, I think, that the fate of the universe, and all those distant stars that outweigh us, will end up shaped by Life. So that the universe ends up hanging quite heavily on the existence of that first replicator, and not on the counterfactual states of any particular other molecules nearby... In that sense, a small handful of atoms once seized the reins of Destiny.
How? How did the first replicating pattern take over the world? Why didn't all those other molecules get an equal vote in the process?
Well, that initial replicating pattern was doing some kind of search - some kind of optimization - and nothing else in the Universe was even trying. Really it was evolution that took over the world, not the first replicating pattern per se - you don't see many copies of it around any more. But still, once upon a time the thread of Destiny was seized and concentrated and spun out from a small handful of atoms.
The first replicator did not set in motion a clever optimization process. Life didn't even have sex yet, or DNA to store information at very high fidelity. But the rest of the Universe had zip. In the kingdom of blind chance, the myopic optimization process is king.
Issues of "sharing improvements" or "trading improvements" wouldn't even arise - there were no partners from outside. All the agents, all the actors of our modern world, are descended from that first replicator, and none from the mountains and hills.
And that was the story of the First World Takeover, when a shift in the structure of optimization - namely, moving from no optimization whatsoever, to natural selection - produced a stark discontinuity with previous trends; and squeezed the flow of the whole universe's destiny through the needle's eye of a single place and time and pattern.
That's Life.