"it" isn't a single theory.

The argument that Everettian MW is favoured by Solomonoff induction, is flawed.

If the program running the SWE outputs information about all worlds on a single output tape, they are going to have to be concatenated or interleaved somehow. Which means that to make use of the information, you gave to identify the subset of bits relating to your world. That's extra complexity which isn't accounted for because it's being done by hand, as it were..

By far the best definition I’ve ever heard of the supernatural is Richard Carrier’s: A “supernatural” explanation appeals to ontologically basic mental things, mental entities that cannot be reduced to nonmental entities.

Physicalism, materialism, empiricism, and reductionism are clearly similar ideas, but not identical. Carrier's criterion captures something about a supernatural ontology, but nothing about supernatural epistemology. Surely the central claim of natural epistemology is that you have to look...you can't rely on faith , or clear ideas implanted in our minds by God.

it seems that we have very good grounds for excluding supernatural explanations a priori

But making reductionism aprioristic arguably makes it less scientific...at least, what you gain in scientific ontology, you lose in scientific epistemology.

I mean, what would the universe look like if reductionism were false

We wouldn't have reductive explanations of some apparently high level phenomena ... Which we don't.

I previously defined the reductionist thesis as follows: human minds create multi-level models of reality in which high-level patterns and low-level patterns are separately and explicitly represented. A physicist knows Newton’s equation for gravity, Einstein’s equation for gravity, and the derivation of the former as a low-speed approximation of the latter. But these three separate mental representations, are only a convenience of human cognition. It is not that reality itself has an Einstein equation that governs at high speeds, a Newton equation that governs at low speeds, and a “bridging law” that smooths the interface. Reality itself has only a single level, Einsteinian gravity. It is only the Mind Projection Fallacy that makes some people talk as if the higher levels could have a separate existence—different levels of organization can have separate representations in human maps, but the territory itself is a single unified low-level mathematical object. Suppose this were wrong.

Suppose that the Mind Projection Fallacy was not a fallacy, but simply true.

Note that there are four possibilities here...

1. I assume a one level universe, all further details are correct.

2. I assume a one level universe, some details may be incorrect

3. I assume a multi level universe, all further details are correct.

4. I assume a multi level universe, some details may be incorrect.

How do we know that the MPF is actually fallacious, and what does it mean anyway?

If all forms of mind projection projection are wrong, then reductive physicalism is wrong, because quarks, or whatever is ultimately real, should not be mind projected, either.

If no higher level concept should be mind projected, then reducible higher level concepts shouldn't be ...which is not EY's intention.

Well, maybe irreducible high level concepts are the ones that shouldn't be mind projected.

That certainly amounts to disbelieving in non reductionism...but it doesn't have much to do with mind projection. If some examples of mind projection are acceptable , and the unacceptable ones coincide with the ones forbidden by reductivism, then MPF is being used as a Trojan horse for reductionism.

And if reductionism is an obvious truth , it could have stood on its own as apriori truth.

Suppose that a 747 had a fundamental physical existence apart from the quarks making up the 747. What experimental observations would you expect to make, if you found yourself in such a universe?

Science isn't 100% observation,it's a mixture of observation and explanation.

A reductionist ontology is a one level universe: the evidence for it is the success of reductive explanation , the ability to explain higher level phenomena entirely in terms of lower level behaviour. And the existence of explanations is aposteriori, without being observational data, in the usual sense. Explanations are abductive,not inductive or deductive.

As before, you should expect to be able to make reductive explanations of all high level phenomena in a one level universe....if you are sufficiently intelligent. It's like the Laplace's Demon illustration of determinism,only "vertical". If you find yourself unable to make reductive explanations of all phenomena, that might be because you lack the intelligence , or because you are in a non reductive multi level universe or because you haven't had enough time...

Either way, it's doubtful and aposteriori, not certain and apriori.

If you can’t come up with a good answer to that, it’s not observation that’s ruling out “non-reductionist” beliefs, but a priori logical incoherence"

I think I have answered that. I don't need observations to rule it out. Observations-rule it-in, and incoherence-rules-it-out aren't the only options.

People who live in reductionist universes cannot concretely envision non-reductionist universes.

Which is a funny thing to say, since science was non-reductionist till about 100 years ago.

One of the clinching arguments for reductionism.was the Schrödinger equation, which showed that in principle, the whole of chemistry is reducible to physics, while the rise of milecular biology showeds th rreducxibility of Before that, educators would point to the de facto hierarchy of the sciences -- physics, chemistry, biology, psychology, sociology -- as evidence of a multi-layer reality.

Unless the point is about "concretely". What does it mean to concretely envision a reductionist universe? Pehaps it means you imagine all the prima facie layers, and also reductive explanations linking them. But then the non-reductionist universe would require less envisioning, because byit's the same thing without the bridging explanations! Or maybe it means just envisioing huge arrays of quarks. Which you can't do. The reductionist world view , in combination with the limitations of the brain, implies that you pretty much have to use higher level, summarised concepts...and that they are not necessarily wrong.

But now we get to the dilemma: if the staid conventional normal boring understanding of physics and the brain is correct, there’s no way in principle that a human being can concretely envision, and derive testable experimental predictions about, an alternate universe in which things are irreducibly mental. Because, if the boring old normal model is correct, your brain is made of quarks, and so your brain will only be able to envision and concretely predict things that can predicted by quarks.

1. "Your brain is made of quarks" is aposteriori, not apriori.

2. Your brain being made of quarks doesn't imply anything about computability. In fact, the computatbolity of the ultimately correct version of quantum physics is an open question.

3. Incomputability isn't the only thing that implies irreducibility, as @ChronoDas points out.

4. Non reductionism is conceivable, or there would be no need to argue for reductionism.

The Deutsch-Yudkowsky argument for the Many Worlds Interpretation states that you can take the core of Quantum Mechanics -- the Schrödinger wave equation, and the projection postulate -- remove the projection postulate (also known as collapse and reduction ), and end with a simpler theory that is still adequate to explain observation. The idea is that entanglement can replace collapse: a scientist observing a superposed state becomes becomes entangled with it, an effectively splits into two, each having made a definite observation.

Moreover Yudkowsky, following David Deutsch, holds the many worlds interpretation to be obviously correct, in contrast to the majority of philosophers and physicists, who regard the problem of interpreting QM as difficult and unsolved.

This has some problems.

(Which are to do with the specific argument, and the level of certainty ascribed to it. To say that you cannot be certain about a claim is not to say it is false. To point out that one argument for a claim does not work is likewise not to say that the claim itself is false. There could be better arguments for these versions of many worlds, or better many worlds theories, for that matter).

The Problems.

The first thing to note is that there is more than one quantum mechanical many worlds theory. What splittng is...how complete and irrevocable it is ... varies between particular theories. So does the rate of splitting, so does the mechanism of splitting.

The second thing to note is that many worlders are pointing at something implied the physical formalism and saying "that's a world"....but whether it qualifies as a world is a separate question from whether it's in the formalism , and a separate kind of question, from whether it is really there in the formalism. One would expect a world, or universe, to be large, stable, non-interacting, and so on . It's possible to have a theory that has collapse , without having worlds. A successful MWI needs to jump three hurdles: empirical correctness, mathematical correctness and conceptual correctness -- actually having worlds

The third problem to note is that all outstanding issues with MWI are connected in some way with quantum mechanical basis....a subject about which Deutsch and Yudkowsky have little to say.

Coherence versus Decoherence

There is an approach to MWI based on coherent superpositions, and a version based on decoherence. These are (for all practical purposes) incompatible opposites, but are treated as interchangeable in Yudkowsky's writings.

Quantum superposition is a fundamental principle of quantum mechanics that states that linear combinations of solutions to the Schrödinger equation are also solutions of the Schrödinger equation. This follows from the fact that the Schrödinger equation is a linear differential equation in time and position. (WP)

Coherent superpositions are straightforwardly implied by the core mathematics of Quantum mechanics. They are small scale in two senses: they can go down to the single particle level, and it is difficult to.maintain large coherent superpositions even if you want to. They are also possibly observer dependent, reversible, and continue to interact (strictly speaking , interfere) after "splitting". The last point is particularly problematical. because if large scale coherent superposition exist , that would create naked eye, macrocsopic evidence:, e.g. ghostly traces of a world where the Nazis won. All in all, a coherent superposition isn't a world you could live in.

I said complex coherent superpositions are difficult to maintain. What destroys them? Environmental induced decoherence!

Interference phenomena are a well-known and crucial aspect of quantum mechanics, famously exemplified by the two-slit experiment. There are many situations, however, in which interference effects are artificially or spontaneously suppressed. The theory of decoherence is precisely the study of such situations. (SEP)

Decoherence tries to explain why we don't notice "quantum weirdness" in everyday life -- why the world of our experience is a more-or-less classical world. From the standpoint of decoherence, sure there might not be any objective fact about which slit an electron went through, but there is an objective fact about what you ate for breakfast this morning: the two situations are not the same!

The basic idea is that, as soon as the information encoded in a quantum state "leaks out" into the external world, that state will look locally like a classical state. In other words, as far as a local observer is concerned, there's no difference between a classical bit and a qubit that's become hopelessly entangled with the rest of the universe.

(http://scottaaronson.com/democritus)

Decoherence is the study of interactions between a quantum system (generally a very small number of microscopic particles like electrons, photons, atoms, molecules, etc. - often just a single particle) and the larger macroscopic environment, which is normally treated "classically," that is, by ignoring quantum effects, but which decoherence theorists study quantum mechanically. Decoherence theorists attribute the absence of macroscopic quantum effects like interference (which is a coherent process) to interactions between a quantum system and the larger macroscopic environment.(www.informationphilosopher.com)

Decoherent branches are necessarily large, since decoherence is a high level phenomenon. They are also stable, non interacting and irreversible...everything that would be intuitively expected of a "world". But there is no empirical evidence for them (in the plural) , nor are they obviously supported by the core mathematics of quantum mechanics, the Schrödinger equation.

We have evidence of small scale coherent superposition, since a number of observed quantum effects depend on it, and we have evidence of decoherence, since complex superposition are difficult to maintain. What we don't have evidence of is decoherence into multiple branches. From the theoretical perspective, decoherence is a complex , entropy like process which occurs when a complex system interacts with its environment. But without decoherence, MW doesn't match observation. So there is no theory of MW that is both simple and empirically adequate, contra Yudkowsky and Deutsch.

The original, Everettian, approach is based on coherence. (Yudkowsky says "Macroscopic decoherence, a.k.a. many-worlds, was first proposed in a 1957 paper by Hugh Everett III" ... but the paper doesn't mention decoherence^[1]) As such, it fails to predict classical observations -- at all -- it fails to predict the appearance of a broadly classical universe. If everything is coherently superposed, so are observers...but the naturally expected experience an observer in coherent superposition with themselves, is that they function as a single observer making ambiguous, superposed observations ... not two observers each making an unambiguous , classical observation, and each unaware of the other. Such observers would only ever see superpositions of dead and living cats, etc.

(A popular but mistaken idea is that full splitting happens microscopically, at every elementary interaction But that would make complex superpositions non-existent, whereas a number of instruments and technologies depend on them -- so it's empirically false).

Later, post 1970s, many world theorists started to include decoherence to make the theory more empirically adequate, but inasmuch as it is additional structure, it places the simplicity of MWI in doubt. In the worst case, the complexity is SWE+decoherence+preferred basis, whereas in the best case, it's SWE alone, because decoherence is implicit in SWE, and preferred basis is implicit in decoherence. Decoherentists hope to show that the theory can be reduced to core QM, such as the Schrödinger equation, but it currently uses more complex math, the "reduced density matrix". The fact that this research is ongoing is strong evidence that the whole problem was not resolved by Everetts's 1957 paper. In any case, without a single definitive mechanism of decoherence, there is no definitive answer to "how complex is MWI".

And single-universe decoherence is quite feasible. Decoherence adds something to many worlds, but many worlds doesn't add anything to decoherence.

So, coherent superpositions exist, but their components aren't worlds in any intuitive sense; and decoherent branches would be worlds in the intuitive sense, but decoherence isn't simple. Also, theoretically and observationally, decoherence could be a single world phenomenon. Those facts -- the fact that it doesn't necessarily involve multi way branching, and the fact that it is hard to evaluate its complexity because there is not a single satisfactory theory for it -- means it is not a "slam dunk" in Yudkowsky's sense.

The Yudkowsky-Deutsch claim is that there is a single MW theory, which explains everything that needed explaining, and is obviously simpler than its rivals. But coherence doesn't save appearances , and decoherence, while more workable, is not known to be simple. So neither theory has both virtues

---

---

1. Which makes the term *Everett branch" rather confusing. The writer possibly means a decohered branch, under the mistaken assumption that Everett was talking about them. Everett's dissertation can be found here ↩︎

If [normality] means something else …what?

We are in agreement here. And if you read to the end of the post you’ll see the answer to your question:

I have, and I still haven't seen a definition of normality.

I think you meant Euclidean.

That’s too, though I was mostly hinting to relativity.

It's general.relativity that implies non euclidean space.

Curious how our understanding of things that some people may assume are beyond observations are then happen to be changed by scientific discovery, isn’t it?

Not once you realise scientific discovery isn't pure empiricism.

Almost all contemporary epistemologists will say that they are fallibilists

That’s nice. Though the key words here are “contemporary” and “epistemologists”.

So where are the philosophers obsessed with certainty? In the past, or in other departments?

On one hand, skepticism is a legitimate philosophical view with no good arguments against.

Which version of scepticism? Even the article you link to concedes there is,at least one form of scepticism that's refutable through self defeat.

If extreme scepticism is self defeating, and certainty is unobtainable, then what you are left with is moderate scepticism -- AKA fallibilism. What you need to try is the right amount if the right kind of scepticism.

We can even see how this kind of reasoning makes some sense with the ultimate goal of adding philosophy up to normality

If the world is weird, I wish to believe that the world is weird.

The goal is to add up to truth.

If normality means supporting all your intuitions, then you are going to have to disbelieve much science and maths.

If it means something else ...what?

If the brain has an amazing “a priori truth factory” that works to produce accurate beliefs, it makes you wonder why a thirsty hunter-gatherer can’t use the “a priori truth factory” to locate drinkable water. It makes you wonder why eyes evolved in the first place, if there are ways to produce accurate beliefs without looking at things.

Something can work in some contexts, but not in in others.

Empiricism doesn't work for things you can't see, eg:-

Modality, Counterfactuals, Possible versus Actual.

Normativity, Ought versus is.

Essence versus Existence, hidden explanatory mechanisms.

A priori truth could have a naturalistic basis. Many organisms can instinctively recognise food, predators, rivals and mates. But even the broadest evolutionary knowledge must that operate within the limits of empiricism .. not the "invisibles" I mentioned. And of course it is a rather different kind of apriori knowledge than analytical kind, based on language and tautologies.

Emanuel Kant came to the conclusion that there is no way to justify the existence of space and time with observations, as space and time are prerequisites for any observations in the first place. Therefore, they have to be justifiable “à priori” in a matter, suspiciously resembling cyclical reasoning:

Unless “à priori” justifications are true, space and time are not justifiable. But space and time has to be justifiable[1]. Therefore “à priori” justifications has to be

Space and time don't need to be justifiable, because they are not propositions. they are aoriori, but not apriori propositions. They are, for Kant, preconditions for having any experience, and therefore apriori. Geometric truths are apriori propositions. They need justification, which for Kant, is the apriori nature of space.and time -- for humans.

There is an extra level of irony here that, among other things, Kant has “à priori” figured out that space and time are absolute, which we now know not to be the case

I think you meant Euclidean.

Note that no claim is made that space is Euclidean in itself. The tower of.justification is only rooted in the operation of human perception.The

Kantian issues about how much of what we perceive is out there, and how much how.much our form of representation are still live, and crop up within science. Just looking doesn't solve them, because you can only look from within the mind-world relationship ... you can't look at it from the outside.

And so most philosophers are really into certainty.

What does that mean?

" Almost all contemporary epistemologists will say that they are fallibilists"

https://iep.utm.edu/fallibil/#H1

It’s deeply entangled with the view that philosophy (or at the least some part of it) is a separate magisterium that lies beyond the empiricism of sciences

Half of science lies beyond the empiricism of science.

ETA

Engineers building rockets do not sweat about Cartesian Demon and yet the rockets seem to works fine. If something is good enough for building rockets maybe it’s good enough for our reasoning in general?I

On the other hand, maybe it isnt. There's no guarantee. Maybe forms of reasoning that depend on visible feedback about whether something happens or works can't extend to areas where there is no such feedback -- areas like "what is the ultimate nature of reality". There is no direct test for correspondence-to-reality. So truth is different to usefulness.

The usefulness cluster of concepts includes the ability to make predictions, as well as create technology. The truth cluster of concepts involves identification of the causes of perceptions, and offering explanations, not just predictions. Usefulness corresponds to engineering, truth to philosophy, and science straddles both. Usefulness corresponds to phenomenal, truth to noumena.

The usefulness cluster corresponds to scientilfic instrumentalism , the truth cluster to scientific instrumentalism. The truth cluster corresponds to epistemological rationalism, the usefulness cluster to instrumental rationalim. Usefulness corresponds to practice, truth to theory.

Truth is correspondence to reality , which is not identical to the ability to make predictions. One can predict that the sun will rise without knowing what the Sun really is. "Curve fitting" science is adequate to make predictions. Trial and error is adequate to come up with useful technologies. But other means are needed to find the underlying reality. One can't achieve convergence by "just using evidence" because the questions of what evidence is, and how to interpret depend on one's episteme. One also can't because ruling out empirically reconfirmed theories generally doesn't leave you with a single candidate.

Science straddles both. Science is not just a matter of making predictions, it's a matter of of answering "how" and "why" questions, of finding explanations. That can't be done by stamp -collecting observations ... It requires the conscious and creative conjecture of explanatory hypotheses, something beyond empiricism.

Also:-

CEV seems much much more difficult than strawberry alignment and I have written it off as a potential option for a baby’s first try at constructing superintelligence.

To be clear, I also expect that strawberry alignment is too hard for these babies and we’ll just die. But things can always be even more difficult, and with targeting CEV on a first try, it sure would be.

There’s zero room, there is negative room, to give away to luxury targets like CEV. They’re not even going to be able to do strawberry alignment, and ify some miracle we were able to do strawberry alignment and so humanity survived, that miracle would not suffice to get CEV right on the first try.

Because there is no fire in the ontology of modern physics and there are no laws of physics that say that some arrangement of atoms are fire.

Fire is reducible. If you want a physical explanation, you can have one.

Binding comes from a human desire to describe things in an approximate, useful way

No it comes from the observationthat our sensorium is not a picture of our brains?

It factors into localised parts.

Approximately localized

If Qualia are identified with microphysical properties, those properties need to be localised to solve the binding problem.

(by which I think most of them do mean “shared human value” even if they don’t all bother to specify), and that I’m suggesting pointing Value Learning at.

I'm suggesting they should bother to specify.

(along with more basic things, like us being around flowers, parks, seashores, and temperature around 75°F) what I’m suggesting as a candidate definition for the “human values”

But are they relevant to ethics or alignment? a lot of tuem are aesthetic preferences that can be satisfied without public policy.

I also didn’t specify above what I think should be done, if it turns out that say, about 96–98% of humans genetically have those shared values, and 2–4% have different alleles.

Shared genetics can lead to different blood and tissue types, so it can lead to different ethical types.

Politics indicates it's more like 50-50, when you are talking about the kind of values that cannot be satisfied individually.

And “my tribe”. What you want is Universalism, but universalism is a late and strange development. It seems obvious to twenty first century Californians, by they are The weirdest of the WEIRD. Reading values out of evopsych is likely to push you in the direction of tribalism, so I don’t see how it helps.

On the Savannah, yes of course it does. In a world-spanning culture of eight billion people, quite a few of whom are part of nuclear-armed alliances, intelligence and the fact that extinction is forever suggests defining “tribe” ~= “species + our comensal pets”. And also noting and reflecting upon that the human default tendency to assume that tribes are around our Dunbar Number in size is now maladaptive, and has been for millennia.

There are technologically advanced tribalists destroying each other right now. It 's not that simple.

It’s not the case that science boils down to Bayes alone,

Are you saying that there’s more to the Scientific Method that applied approximate Bayesiasm?

Yes. I learnt physics without ever learning Bayes. Science=Bayes is the extraordinary claim that needs justification.

or that science is the only alternative to philosophy. Alignment/control is more like engineering.

Engineering is applied Science, Science is applied Mathematics; from Philosophy’s point of view it’s all Naturalism. In the above, it kept turning out that Engineering methodology is exactly what Evolutionary Psychology says is the adaptive way for a social species to treat their extended phenotype.

Again, I would suggest using the word engineering, if engineering is what you mean.

So, In philosophy of science terminology, pholosophers have plenty of hypothesis generation, but very little falsifiability (beyond, as Gettier did, demonstarting an internal logical inconsistency), so the tendency it to increase the number of credible candidate answers, rather than decreasing them.

That's still useful If you have some way of judging their correctness -- it doesn't have to be empiricism. To find the one true hypothesis, you need to consider all.of them,.and to approximate that , you need to do consider a lot of them.

The same thing occurs within science , because science isn't pure empiricism. The panoply of interpretations of QM is an example.

I agree. The way to define the phrase “human values” for AI Value Learning that I’m suggesting above is that it’s the portion of it that we generally tend to agree on,

That could have been expressed as shared human value.

to the extent that the great majority of us share similar moral intuitions since we (generally) share the same genetic behavioral adaptations. On the parts we disagree on, it’s silent. That’s a portion of why it only induces a partial ordering. However if, for example, a particular choice of weighting on the balance between individual freedom and social cohesion turned out to rather consistently depends on some feature of your upbringing, say, the level of danger the society then appeared to be in, then that tendency might be an adaptation — and if it were then my definition of “human values” would then include a conditional such as, say “if society is was in danger during your formative years, then weight social cohesion more highly”.

So “human value” [not otherwise specified] can’t be equated with the good or the right.

By “the good” or “the right”, do you mean the sorts of things that Moral Realism posits the reality of?

Minimally, it's something that allows you to make a decision.

Not all of human behavior/values is the problem here. For example, on the survival instinct, most humans say something like “I fully endorse the right not to be killed for all humans (especially in my own case)”.

And "my tribe". What you want is Universalism, but universalism is a late and strange development. It seems obvious to twenty first century Californians, by they are The weirdest of the WEIRD. Reading values out of evopsych is likely to push you in the direction of tribalism, so I don't see how it helps.

I think if I were The Sorting Hat, I would tend to assign Rationalists to Ravenclaw :-)

What would that make the E/Accs?

---

I suggest avoiding a dependency on Philosophy entirely, and using Science instead. Which has a means for telling people their ideas are bad, called Bayesianism (a.k.a. the Scientific Method). For ethics, the relevant science is Evolutionary Moral Psychology.

It's not the case that science boils down to Bayes alone, or that science is the only alternative to philosophy. Alignment/control is more like engineering.

There are a significant number of subjects that philosophers used to discuss, before science was able to study them, then later science started to cast light on them, eventually identified the correct hypotheses, and the philosophers gradually lost interest in them. So there is and alternative to the above: identify all the alignment-relevant questions that we currently have philosophical hypotheses about but not scientific answers, and develop scientific answers to them.

The fact that some philosophical problems can be answered by science isn't a guarantee that that they all can. There is an in-principal.argument against a scienxe, which is basically objective, being able to understand Hard Problem consciousness ,which is essentially subjective.

Human value isn't a set preferences that can be pursued individually without conflict. Evolutionary psychology doesn't predict that , and we see the conflicts played out very day. there is more evidence for the incoherence of human value than there is for just about anything.

So "human value" can't be equated with the good or the right. It's a problem not a solution.

It also can't be equated with the safe. Copying human values into an AI won't give you an AI that won't kill you. You've mentioned self preservation as something that is part of Human value (even if instrumentally) and dangerous in an AI; , but the Will to Power, the cluster of values connected with ambition, hierarchy and dominance is more so.

It's odd that this point is so often missed by rationalists. Perhaps that's because they tend to have Hufflepuff values.

A handwavy argument that “training is a bit like evolution, so maybe the same social dynamics should apply to its products” is inaccurate: you can train in aligned behavior, so you should (in both the evolutionary and engineering senses of the word) — but you can’t evolve it, evolution just doesn’t do that.

Inasmuch as it is not like natural selection , it is like artificial selection. That's good news, because artificial selection doesn't copy human values blindly: if you want AI s that are helpful and self sacrificing, you can select them to be, much more so than a human.

It's generally hard to see what useful work is being done by "human value*. Value is only relevant to.alignment as opposed to control, for one thing...some human values are.down right unsafe for another. Taking "human value" out of the loop allows you to get to the conclusions that "if you doing want to be killed, build AI's that don't want to kill you" more quickly.

However, Evolutionary Psychology does make it very clear that (while morally anthropomorphizing aligned AIs is cognitively-natural for current humans), doing this is also maladaptive. This is because AIs aren’t in the right category – things whose behavior is predicted by evolutionary theory – for the mechanisms of Evolutionary Moral Psychology to apply to them.

Their behaviour is not predicted by evolutionary theory , but is predicted by something wider.

Those mechanisms make this behavior optimal when interacting with co-evolved intelligences that you can ally with (and thus instinctive to us) — whereas, for something you constructed, this behavior is suboptimal.

Things do what they do. It doesn't have to be an optimization.