But I still feel like you're being a bit too charitable. I re-read the 'it's okay to use 'emerge"' parts several times, and as I understand it, he's not meaning to refer to a higher-level abstraction, he's using it in the general sense "whatever byproduct comes from this" in which case it would be just as meaningful to say "heat emerges from the body" which does not reflect any definition of emergence as a higher-level abstraction.

[,,,]

But it is not correct to say that acknowledging intelligence as emergent doesn't help us predict anything. If emergence can be described as a pattern that happens across different realms then it can help to predict things, through the use of analogy. 

I don't think Eliezer uses emergence that way. He is using it the way, if a person is asked "why do hands have X muscular movement?" , one may reply "it's an emergent phenomena" , that is the explanation which doesn't predict anything he's criticizing, unless a person clarifies what they mean by emergent phenomena. 

A proper explanation could be: (Depending on what is meant by the word "Why") ^[1]

1. Evolutionary: The reason why X muscular movement got selected for.
2. Biological/Shape/Adaptation; How it works or got implemented.

The common use of the word emergent is such that when a person is perplexed by the idea of free will, and finds a lack of contra-causal free will troubling to their preliminary intuitions , their encounter with the idea "free will is emergent" resolves the cognitive dissonance mistaking it for an explanation^[2] when it holds no predictive power, and doesn't actually work to resolve the initial confusion alongside physics.

What examples do I have in back of my mind that I think he's criticizing this particular usage? 

Eg-1: He uses the example of  "Intelligence is emergent". 

• In online spaces when asked, "Where is the 'you' in the brain if it's all just soft flesh?" , people often say , "I am emergent" . Which doesn't quite predict anything, like I learn nothing about when do I cease to be "I" , or why do I feel like "I" etc. 

Eg-2: He uses the example of "Free will is emergent" , where he mentions the phrasing "one level emerges from the other". 

To dissolve the puzzle of free will, you have to simultaneously imagine two levels of organization while keeping them conceptually distinct.  To get it on a gut level, you have to see the level transition—the way in which free will is how the human decision algorithm feels from inside.  (Being told flatly "one level emerges from the other" just relates them by a magical transition rule, "emergence".)

Eg-3; He uses "behavior of ant colony is emergent" in original post.

Eg-4; He also emphasizes that he's fine with the term that chemistry "arises from" interaction between atoms as per QED. Since chemistry—or parts of it, can be predicted in terms of QED. ^[2]

Chemistry arises from interactions between atoms, according to the specific model of quantum electrodynamics.

• Which he clarifies is fairly equivalent to "Chemistry emerges from interaction between atoms as per QED". 

None of these examples seem to argue against "emergence as a pattern that happens across different realms" . That seems like a different thing altogether and can be assigned a different word.

This particular usage he is criticizing, and this is the trap majority of people including my past self & a lot of people I know in real life fall into. Which is why, I think the disagreement here is mostly semantic, as highlighted by TAG. This can also be categorized as the trap of strong emergentism, or intuitions behind it such that it satisfies the human interrogation without adding anything to understanding. Moreover, the sequence in question is named "Mysterious answers" where he goes over certain concepts which are in common zeitgeist used as an explanations even when they're not.^[2]

From what I understand the way you're using it, in emergent cycle to import over to other places Eliezer would agree with your use case. He uses this as an argument as to why  maths is useful :- 

The apples are behaving like numbers? What do you mean? I thought numbers were this ethereal mathematical model that got pinpointed by axioms, not by looking at the real world.

"Whenever a part of reality behaves in a way that conforms to the number-axioms - for example, if putting apples into a bowl obeys rules, like no apple spontaneously appearing or vanishing, which yields the high-level behavior of numbers - then all the mathematical theorems we proved valid in the universe of numbers can be imported back into reality. The conclusion isn't absolutely certain, because it's not absolutely certain that nobody will sneak in and steal an apple and change the physical bowl's behavior so that it doesn't match the axioms any more. But so long as the premises are true, the conclusions are true; the conclusion can't fail unless a premise also failed. You get four apples in reality, because those apples behaving numerically isn't something you assume, it's something that's physically true. When two clouds collide and form a bigger cloud, on the other hand, they aren't behaving like integers, whether you assume they are or not."

But if the awesome hidden power of mathematical reasoning is to be imported into parts of reality that behave like math, why not reason about apples in the first place instead of these ethereal 'numbers'?

It seems like your emergent cycle is closer to this. He similarly to your emergent cycle for systems also asserts probability theory as laws underlying rational belief and similarly decision theory for rational actions of all agents:-

Probability theory is the set of laws underlying rational belief. The mathematics of probability applies equally to “figuring out where your bookcase is” and “estimating how many hairs were on Julius Caesars head,” even though our evidence for the claim “Julius Caesar was bald” is likely to be more complicated and indirect than our evidence for the claim “theres a bookcase in my room.” It’s all the same problem of how to process the evidence and observations to update one’s beliefs. Similarly, decision theory is the set of laws underlying rational action, and is equally applicable regardless of what one’s goals and available options are .

Decision theory works because it is sufficiently similar to the goal oriented systems in the universe. He also thinks intelligence is lawful, as in it's orderly such as following decision theory. Which seems similar to your defense in the sense of multiple leveled maps. 

To further flesh the point, he would agree with you on the eyes part :-

The notion of a "configuration space" is a way of translating object descriptions into object positions.  It may seem like blue is "closer" to blue-green than to red, but how much closer?  It's hard to answer that question by just staring at the colors.  But it helps to know that the (proportional) color coordinates in RGB are 0:0:5, 0:3:2 and 5:0:0.  It would be even clearer if plotted on a 3D graph.

In the same way, you can see a robin as a robin—brown tail, red breast, standard robin shape, maximum flying speed when unladen, its species-typical DNA and individual alleles.  Or you could see a robin as a single point in a configuration space whose dimensions described everything we knew, or could know, about the robin.

A robin is bigger than a virus, and smaller than an aircraft carrier—that might be the "volume" dimension.  Likewise a robin weighs more than a hydrogen atom, and less than a galaxy; that might be the "mass" dimension.  Different robins will have strong correlations between "volume" and "mass", so the robin-points will be lined up in a fairly linear string, in those two dimensions—but the correlation won't be exact, so we do need two separate dimensions.

[,,,]

We can even imagine a configuration space with one or more dimensions for every distinct characteristic of an object, so that the position of an object's point in this space corresponds to all the information in the real object itself.  Rather redundantly represented, too—dimensions would include the mass, the volume, and the density.

[,,,]

Suppose we mapped all the birds in the world into thingspace, using a distance metric that corresponds as well as possible to perceived similarity in humans:  A robin is more similar to another robin, than either is similar to a pigeon, but robins and pigeons are all more similar to each other than either is to a penguin, etcetera.

Then the center of all birdness would be densely populated by many neighboring tight clusters, robins and sparrows and canaries and pigeons and many other species.  Eagles and falcons and other large predatory birds would occupy a nearby cluster.  Penguins would be in a more distant cluster, and likewise chickens and ostriches.

He probably would assign the intensional/word "eye"  to an extensional similarity cluster but I think you and him might still disagree on nuances:  

The atoms of a screwdriver don't have tiny little XML tags inside describing their "objective" purpose. The designer had something in mind, yes, but that's not the same as what happens in the real world. If you forgot that the designer is a separate entity from the designed thing, you might think, "The purpose of the screwdriver is to drive screws"—as though this were an explicit property of the screwdriver itself, rather than a property of the designer's state of mind. You might be surprised that the screwdriver didn't reconfigure itself to the flat-head screw, since, after all, the screwdriver's purpose is to turn screws.

[,,,]

So the screwdriver's cause, and its shape, and its consequence, and its various meanings, are all different things; and only one of these things is found within the screwdriver itself.

Where do taste buds come from? Not from an intelligent designer visualizing their consequences, but from a frozen history of ancestry: Adam liked sugar and ate an apple and reproduced, Barbara liked sugar and ate an apple and reproduced, Charlie liked sugar and ate an apple and reproduced, and 2763 generations later, the allele became fixed in the population. For convenience of thought, we sometimes compress this giant history and say: "Evolution did it." But it's not a quick, local event like a human designer visualizing a screwdriver. This is the objective cause of a taste bud.

What is the objective shape of a taste bud? Technically, it's a molecular sensor connected to reinforcement circuitry. This adds another level of indirection, because the taste bud isn't directly acquiring food. It's influencing the organism's mind, making the organism want to eat foods that are similar to the food just eaten.

What is the objective consequence of a taste bud? In a modern First World human, it plays out in multiple chains of causality: from the desire to eat more chocolate, to the plan to eat more chocolate, to eating chocolate, to getting fat, to getting fewer dates, to reproducing less successfully. This consequence is directly opposite the key regularity in the long chain of ancestral successes which caused the taste bud's shape. But, since overeating has only recently become a problem, no significant evolution (compressed regularity of ancestry) has further influenced the taste bud's shape.

What is the meaning of eating chocolate? That's between you and your moral philosophy. Personally, I think chocolate tastes good, but I wish it were less harmful; acceptable solutions would include redesigning the chocolate or redesigning my biochemistry.

Which is to say he would disagree with the blanket categorization of "purpose", which can be a problematic term and leaves space for misunderstanding regarding normativity and would likely advocate for more clearer thinking and wordings along the lines highlighted above. 

Although I think he would be fine with concepts like 'agency' in decision theories to degree to which axioms happen to coincide with that of reality—just like apples behaving like numbers listed above— since it can be bounded to physical systems amongst other considerations such as well biological agent sustaining itself etc. 

Which brings us to his second potential source of disagreement, regarding analogies :^[3]

A medieval alchemist puts lemon glazing onto a lump of lead.  The lemon glazing is yellow, and gold is yellow.  It seems like it ought to work... but the lead obstinately refuses to turn into gold.  Reality just comes back and says, "So what?  Things can be similar in some aspects without being similar in other aspects."

[,,,]

 

The general form of failing-by-analogy runs something like this:

• You want property P.
• X has property P.
• You build Y, which has one or two surface similarities S to X.
• You argue that Y resembles X and should also P.
• Yet there is no reasoning which you can do on Y as a thing-in-itself to show that it will have property P, regardless of whether or not X had ever existed.

[,,,]

If two processes have forms that are nearly identical, including internal structure that is similar to as many decimal places as you care to reason about, then you may be able to almost-prove results from one to the other.  But if there is even one difference in the internal structure, then any number of other similarities may be rendered void.  Two deterministic computations with identical data and identical rules will yield identical outputs.  But if a single input bit is flipped from zero to one, the outputs are no longer required to have anything in common.  The strength of analogical reasoning can be destroyed by a single perturbation.

Yes, sometimes analogy works.  But the more complex and dissimilar the objects are, the less likely it is to work.  The narrower the conditions required for success, the less likely it is to work.  The more complex the machinery doing the job, the less likely it is to work.  The more shallow your understanding of the object of the analogy, the more you are looking at its surface characteristics rather than its deep mechanisms, the less likely analogy is to work.

[,,,]

Admittedly, analogy often works in mathematics - much better than it does in science, in fact.  In mathematics you can go back and prove the idea which analogy originally suggested.  In mathematics, you get quick feedback about which analogies worked and which analogies didn't, and soon you pick up the pattern.  And in mathematics you can always see the entire insides of things; you are not stuck examining the surface of an opaque mystery.  Mathematical proposition A may be analogous to mathematical proposition B, which suggests the method; but afterward you can go back and prove A in its own right, regardless of whether or not B is true.  In some cases you may need proposition B as a lemma, but certainly not all cases.

Which is to say: despite the misleading surface similarity, the "analogies" which mathematicians use are not analogous to the "analogies" of alchemists, and you cannot reason from the success of one to the success of the other.

Which basically goes over the necessity for having a reasoning as to why, someone suspects this analogy to work aside from being merely similar in certain aspects, because most of them don't. Take the example of more sophisticated analogy of biological evolution extended to corporations: 

Do corporations evolve? They certainly compete. They occasionally spin off children. Their resources are limited. They sometimes die.

But how much does the child of a corporation resemble its parents? Much of the personality of a corporation derives from key officers, and CEOs cannot divide themselves by fission. Price's Equation only operates to the extent that characteristics are heritable across generations. If great-great-grandchildren don't much resemble their great-great-grandparents, you won't get more than four generations' worth of cumulative selection pressure—anything that happened more than four generations ago will blur itself out. Yes, the personality of a corporation can influence its spinoff—but that's nothing like the heritability of DNA, which is digital rather than analog, and can transmit itself with 10^-8 errors per base per generation.

With DNA you have heritability lasting for millions of generations. That's how complex adaptations can arise by pure evolution—the digital DNA lasts long enough for a gene conveying 3% advantage to spread itself over 768 generations, and then another gene dependent on it can arise. Even if corporations replicated with digital fidelity, they would currently be at most ten generations into the RNA World.

Now, corporations are certainly selected, in the sense that incompetent corporations go bust. This should logically make you more likely to observe corporations with features contributing to competence. And in the same sense, any star that goes nova shortly after it forms, is less likely to be visible when you look up at the night sky. But if an accident of stellar dynamics makes one star burn longer than another star, that doesn't make it more likely that future stars will also burn longer—the feature will not be copied onto other stars. We should not expect future astrophysicists to discover complex internal features of stars which seem designed to help them burn longer. That kind of mechanical adaptation requires much larger cumulative selection pressures than a once-off winnowing.

Think of the principle introduced in Einstein's Arrogance—that the vast majority of the evidence required to think of General Relativity had to go into raising that one particular equation to the level of Einstein's personal attention; the amount of evidence required to raise it from a deliberately considered possibility to 99.9% certainty was trivial by comparison. In the same sense, complex features of corporations which require hundreds of bits to specify, are produced primarily by human intelligence, not a handful of generations of low-fidelity evolution. In biology, the mutations are purely random and evolution supplies thousands of bits of cumulative selection pressure. In corporations, humans offer up thousand-bit intelligently designed complex "mutations", and then the further selection pressure of "Did it go bankrupt or not?" accounts for a handful of additional bits in explaining what you see.

Since this analogy can mislead one to think corporations can evolve on their own and someone may conclude that hence anyone can become CEO of a corporation, but it's the case that majority of changes are due to human intelligence so that would be an example of failure by analogy.^[4]

As per my amateur analysis you seem to have taken caution in your emergent cycle analogy, as to only apply it when entropy has inverse relationship, which is still quite a broad application and but on surface seems to isolate the generalization to certain systems which have sufficiently enough internal structure for the analogy to carry over under those constraints.

1. ^{^}

   A thing to note here is Eliezer is a predictivist, both of these type of explanation would narrow down experience to hand's muscle moving a certain way.

2. ^{^}

   For Eliezer— a predictivist—an explanation narrows down anticipated experience.

3. ^{^}

    In this post he criticized neural networks, and as we know that particular prediction of his aged poorly, but those are for different reasons, but the general point regarding analogy still stands.

4. ^{^}

   Although I can see someone making the case of memetic evolution.