"it" isn't a single theory.

The argument that Everettian MW is favoured by Solomonoff induction, is flawed.

If the program running the SWE outputs information about all worlds on a single output tape, they are going to have to be concatenated or interleaved somehow. Which means that to make use of the information, you gave to identify the subset of bits relating to your world. That's extra complexity which isn't accounted for because it's being done by hand, as it were..

By far the best definition I’ve ever heard of the supernatural is Richard Carrier’s: A “supernatural” explanation appeals to ontologically basic mental things, mental entities that cannot be reduced to nonmental entities.

Physicalism, materialism, empiricism, and reductionism are clearly similar ideas, but not identical. Carrier's criterion captures something about a supernatural ontology, but nothing about supernatural epistemology. Surely the central claim of natural epistemology is that you have to look...you can't rely on faith , or clear ideas implanted in our minds by God.

it seems that we have very good grounds for excluding supernatural explanations a priori

But making reductionism aprioristic arguably makes it less scientific...at least, what you gain in scientific ontology, you lose in scientific epistemology.

I mean, what would the universe look like if reductionism were false

We wouldn't have reductive explanations of some apparently high level phenomena ... Which we don't.

I previously defined the reductionist thesis as follows: human minds create multi-level models of reality in which high-level patterns and low-level patterns are separately and explicitly represented. A physicist knows Newton’s equation for gravity, Einstein’s equation for gravity, and the derivation of the former as a low-speed approximation of the latter. But these three separate mental representations, are only a convenience of human cognition. It is not that reality itself has an Einstein equation that governs at high speeds, a Newton equation that governs at low speeds, and a “bridging law” that smooths the interface. Reality itself has only a single level, Einsteinian gravity. It is only the Mind Projection Fallacy that makes some people talk as if the higher levels could have a separate existence—different levels of organization can have separate representations in human maps, but the territory itself is a single unified low-level mathematical object. Suppose this were wrong.

Suppose that the Mind Projection Fallacy was not a fallacy, but simply true.

Note that there are four possibilities here...

1. I assume a one level universe, all further details are correct.

2. I assume a one level universe, some details may be incorrect

3. I assume a multi level universe, all further details are correct.

4. I assume a multi level universe, some details may be incorrect.

How do we know that the MPF is actually fallacious, and what does it mean anyway?

If all forms of mind projection projection are wrong, then reductive physicalism is wrong, because quarks, or whatever is ultimately real, should not be mind projected, either.

If no higher level concept should be mind projected, then reducible higher level concepts shouldn't be ...which is not EY's intention.

Well, maybe irreducible high level concepts are the ones that shouldn't be mind projected.

That certainly amounts to disbelieving in non reductionism...but it doesn't have much to do with mind projection. If some examples of mind projection are acceptable , and the unacceptable ones coincide with the ones forbidden by reductivism, then MPF is being used as a Trojan horse for reductionism.

And if reductionism is an obvious truth , it could have stood on its own as apriori truth.

Suppose that a 747 had a fundamental physical existence apart from the quarks making up the 747. What experimental observations would you expect to make, if you found yourself in such a universe?

Science isn't 100% observation,it's a mixture of observation and explanation.

A reductionist ontology is a one level universe: the evidence for it is the success of reductive explanation , the ability to explain higher level phenomena entirely in terms of lower level behaviour. And the existence of explanations is aposteriori, without being observational data, in the usual sense. Explanations are abductive,not inductive or deductive.

As before, you should expect to be able to make reductive explanations of all high level phenomena in a one level universe....if you are sufficiently intelligent. It's like the Laplace's Demon illustration of determinism,only "vertical". If you find yourself unable to make reductive explanations of all phenomena, that might be because you lack the intelligence , or because you are in a non reductive multi level universe or because you haven't had enough time...

Either way, it's doubtful and aposteriori, not certain and apriori.

If you can’t come up with a good answer to that, it’s not observation that’s ruling out “non-reductionist” beliefs, but a priori logical incoherence"

I think I have answered that. I don't need observations to rule it out. Observations-rule it-in, and incoherence-rules-it-out aren't the only options.

People who live in reductionist universes cannot concretely envision non-reductionist universes.

Which is a funny thing to say, since science was non-reductionist till about 100 years ago.

One of the clinching arguments for reductionism.was the Schrödinger equation, which showed that in principle, the whole of chemistry is reducible to physics, while the rise of milecular biology showeds th rreducxibility of Before that, educators would point to the de facto hierarchy of the sciences -- physics, chemistry, biology, psychology, sociology -- as evidence of a multi-layer reality.

Unless the point is about "concretely". What does it mean to concretely envision a reductionist universe? Pehaps it means you imagine all the prima facie layers, and also reductive explanations linking them. But then the non-reductionist universe would require less envisioning, because byit's the same thing without the bridging explanations! Or maybe it means just envisioing huge arrays of quarks. Which you can't do. The reductionist world view , in combination with the limitations of the brain, implies that you pretty much have to use higher level, summarised concepts...and that they are not necessarily wrong.

But now we get to the dilemma: if the staid conventional normal boring understanding of physics and the brain is correct, there’s no way in principle that a human being can concretely envision, and derive testable experimental predictions about, an alternate universe in which things are irreducibly mental. Because, if the boring old normal model is correct, your brain is made of quarks, and so your brain will only be able to envision and concretely predict things that can predicted by quarks.

1. "Your brain is made of quarks" is aposteriori, not apriori.

2. Your brain being made of quarks doesn't imply anything about computability. In fact, the computatbolity of the ultimately correct version of quantum physics is an open question.

3. Incomputability isn't the only thing that implies irreducibility, as @ChronoDas points out.

4. Non reductionism is conceivable, or there would be no need to argue for reductionism.

The Deutsch-Yudkowsky argument for the Many Worlds Interpretation states that you can take the core of Quantum Mechanics -- the Schrödinger wave equation, and the projection postulate -- remove the projection postulate (also known as collapse and reduction ), and end with a simpler theory that is still adequate to explain observation. The idea is that entanglement can replace collapse: a scientist observing a superposed state becomes becomes entangled with it, an effectively splits into two, each having made a definite observation.

Moreover Yudkowsky, following David Deutsch, holds the many worlds interpretation to be obviously correct, in contrast to the majority of philosophers and physicists, who regard the problem of interpreting QM as difficult and unsolved.

This has some problems.

(Which are to do with the specific argument, and the level of certainty ascribed to it. To say that you cannot be certain about a claim is not to say it is false. To point out that one argument for a claim does not work is likewise not to say that the claim itself is false. There could be better arguments for these versions of many worlds, or better many worlds theories, for that matter).

The Problems.

The first thing to note is that there is more than one quantum mechanical many worlds theory. What splittng is...how complete and irrevocable it is ... varies between particular theories. So does the rate of splitting, so does the mechanism of splitting.

The second thing to note is that many worlders are pointing at something implied the physical formalism and saying "that's a world"....but whether it qualifies as a world is a separate question from whether it's in the formalism , and a separate kind of question, from whether it is really there in the formalism. One would expect a world, or universe, to be large, stable, non-interacting, and so on . It's possible to have a theory that has collapse , without having worlds. A successful MWI needs to jump three hurdles: empirical correctness, mathematical correctness and conceptual correctness -- actually having worlds

The third problem to note is that all outstanding issues with MWI are connected in some way with quantum mechanical basis....a subject about which Deutsch and Yudkowsky have little to say.

Coherence versus Decoherence

There is an approach to MWI based on coherent superpositions, and a version based on decoherence. These are (for all practical purposes) incompatible opposites, but are treated as interchangeable in Yudkowsky's writings.

Quantum superposition is a fundamental principle of quantum mechanics that states that linear combinations of solutions to the Schrödinger equation are also solutions of the Schrödinger equation. This follows from the fact that the Schrödinger equation is a linear differential equation in time and position. (WP)

Coherent superpositions are straightforwardly implied by the core mathematics of Quantum mechanics. They are small scale in two senses: they can go down to the single particle level, and it is difficult to.maintain large coherent superpositions even if you want to. They are also possibly observer dependent, reversible, and continue to interact (strictly speaking , interfere) after "splitting". The last point is particularly problematical. because if large scale coherent superposition exist , that would create naked eye, macrocsopic evidence:, e.g. ghostly traces of a world where the Nazis won. All in all, a coherent superposition isn't a world you could live in.

I said complex coherent superpositions are difficult to maintain. What destroys them? Environmental induced decoherence!

Interference phenomena are a well-known and crucial aspect of quantum mechanics, famously exemplified by the two-slit experiment. There are many situations, however, in which interference effects are artificially or spontaneously suppressed. The theory of decoherence is precisely the study of such situations. (SEP)

Decoherence tries to explain why we don't notice "quantum weirdness" in everyday life -- why the world of our experience is a more-or-less classical world. From the standpoint of decoherence, sure there might not be any objective fact about which slit an electron went through, but there is an objective fact about what you ate for breakfast this morning: the two situations are not the same!

The basic idea is that, as soon as the information encoded in a quantum state "leaks out" into the external world, that state will look locally like a classical state. In other words, as far as a local observer is concerned, there's no difference between a classical bit and a qubit that's become hopelessly entangled with the rest of the universe.

(http://scottaaronson.com/democritus)

Decoherence is the study of interactions between a quantum system (generally a very small number of microscopic particles like electrons, photons, atoms, molecules, etc. - often just a single particle) and the larger macroscopic environment, which is normally treated "classically," that is, by ignoring quantum effects, but which decoherence theorists study quantum mechanically. Decoherence theorists attribute the absence of macroscopic quantum effects like interference (which is a coherent process) to interactions between a quantum system and the larger macroscopic environment.(www.informationphilosopher.com)

Decoherent branches are necessarily large, since decoherence is a high level phenomenon. They are also stable, non interacting and irreversible...everything that would be intuitively expected of a "world". But there is no empirical evidence for them (in the plural) , nor are they obviously supported by the core mathematics of quantum mechanics, the Schrödinger equation.

We have evidence of small scale coherent superposition, since a number of observed quantum effects depend on it, and we have evidence of decoherence, since complex superposition are difficult to maintain. What we don't have evidence of is decoherence into multiple branches. From the theoretical perspective, decoherence is a complex , entropy like process which occurs when a complex system interacts with its environment. But without decoherence, MW doesn't match observation. So there is no theory of MW that is both simple and empirically adequate, contra Yudkowsky and Deutsch.

The original, Everettian, approach is based on coherence. (Yudkowsky says "Macroscopic decoherence, a.k.a. many-worlds, was first proposed in a 1957 paper by Hugh Everett III" ... but the paper doesn't mention decoherence^[1]) As such, it fails to predict classical observations -- at all -- it fails to predict the appearance of a broadly classical universe. If everything is coherently superposed, so are observers...but the naturally expected experience an observer in coherent superposition with themselves, is that they function as a single observer making ambiguous, superposed observations ... not two observers each making an unambiguous , classical observation, and each unaware of the other. Such observers would only ever see superpositions of dead and living cats, etc.

(A popular but mistaken idea is that full splitting happens microscopically, at every elementary interaction But that would make complex superpositions non-existent, whereas a number of instruments and technologies depend on them -- so it's empirically false).

Later, post 1970s, many world theorists started to include decoherence to make the theory more empirically adequate, but inasmuch as it is additional structure, it places the simplicity of MWI in doubt. In the worst case, the complexity is SWE+decoherence+preferred basis, whereas in the best case, it's SWE alone, because decoherence is implicit in SWE, and preferred basis is implicit in decoherence. Decoherentists hope to show that the theory can be reduced to core QM, such as the Schrödinger equation, but it currently uses more complex math, the "reduced density matrix". The fact that this research is ongoing is strong evidence that the whole problem was not resolved by Everetts's 1957 paper. In any case, without a single definitive mechanism of decoherence, there is no definitive answer to "how complex is MWI".

And single-universe decoherence is quite feasible. Decoherence adds something to many worlds, but many worlds doesn't add anything to decoherence.

So, coherent superpositions exist, but their components aren't worlds in any intuitive sense; and decoherent branches would be worlds in the intuitive sense, but decoherence isn't simple. Also, theoretically and observationally, decoherence could be a single world phenomenon. Those facts -- the fact that it doesn't necessarily involve multi way branching, and the fact that it is hard to evaluate its complexity because there is not a single satisfactory theory for it -- means it is not a "slam dunk" in Yudkowsky's sense.

The Yudkowsky-Deutsch claim is that there is a single MW theory, which explains everything that needed explaining, and is obviously simpler than its rivals. But coherence doesn't save appearances , and decoherence, while more workable, is not known to be simple. So neither theory has both virtues

I don’t see why camp #1 requires dualism

Camp 1 is unable to account for self evident experience qua experience, it creates an outside to physical reality in saying “this isn’t worth addressing”, there’s what’s real and, perhaps uncharitably, what’s fiction. To create a truly material monist ontology you must create a material account of this fiction or belief or whatever experience.

Camp 1 is everyone who thinks that qualia are explicable within a materialist ontology. .. so it includes all attempted matetialist explanations, successful or not ..so it includes yours.

You cannot simultaneously assert that the agent is “sufficiently advanced” such that it can always think at or above human-level higher order reasoning, and also that it will not be able to re-prioritise its terminal goals (survivability, paperclips, etc).

Almost. A given agent is not necessary capable of changing its own goals ... but it's not necessarily goal stable either:-

Goal stability is an assumption behind recursive self improvement, which is an argument behind rapid takeoff, which is an argument behind getting goals/value right on the first attempt

It is plausible that an agent would desire to preserve its goals, but the desire to preserve goals does not imply the ability to preserve goals.

An argument for the desire to preserve goals is the "Ghandi Pill" argument.

"See the “Ghandi argument”—if you offer Ghandi a pill that makes him love murdering people, he won’t take the pill because right now he doesn’t want to murder people. A self-modifying AI that wants things will tend to avoid changing what it wants, because it can predict that would lead to things it doesn’t want"

Humans do not preserve the same goals across their lifetimes, and do things which are known to change values, such as getting educated, travelling, and getting married. So the Ghandi Pill argument only seems to to apply to extreme cases, in humans. And there a no reason to suppose that any given AI will b different to a human -- the human mind is a possibility in mindspace.

The Orthogonality Thesis is also sometimes mistakenly called on to support goal stability. It does imply that a lot of combinations of goals and intelligence levels are possible, but doesn't imply that all possible minds have goals, or that all goal driven agents have fixed, incorrigible goals. There are goalless and corrigible agents in mindspace, too.

No goal stable system of any complexity exists on this planet, and goal instability cannot be assumed as a default or given. So the orthogonality thesis is true of momentary combinations of goal and intelligence, given the provisos above, but not necessarily true of stable combinations.

There is, however, a puzzle about how an agent would rationally update its goals. AI doomers like to think in terms of von Neumann rationality.

According to the framework of von Neumann rationality, such an update must be guided by a set of terminal goals ... but how can it be when the agents goals themselves are subject to change?

But the puzzle is rather artificial and inconsequential. For one thing, vNM is a highly artificial framework, with the advantage of functionally well defined, and the disadvantage of being inapplicable to almost any real world system, including most AI s. For another, rational people -- in the informal sense -- are capable of reflecting on , and changing, their goals and values. (https://www.lesswrong.com/posts/8A6wXarDpr6ckMmTn/another-argument-against-utility-centric-alignment-paradigms).?

Irrational goal-drift is possible, too. One thing the Orthogonality Thesis does show is that most possible minds aren't artificial philosophers that will converge onto the One True Morality.

A sign has meaning by comparison to things that it’s not. Saussure in his “Course in General Linguistics” shows effectively how the differentiation of a given spoken word from other words is what allows us to assign a meaning to it, if a word begins to be pronounced in two different ways in different contexts then this difference in pronunciation can take on a conceptual difference.[2]

Which amounts to the claim that difference is necessary for meaning. The more contentious claim is that it is sufficient.

tiated from each other in order for them to be connected back to signifiers. Philosophers obsessed with qualia often ask things like what gives the red of a rose its redness. From a semiotic perspective the answer is ready at hand: the redness of a rose comes from its differentiation from green and black and blue and all the other colors of all the other objects and their associated textures, smells, etc

You seem to be hinting at a theory of pure difference, that Qualia are not intrinsic properties with difference between , but constituted by the differences alone. That's a very difficult theory to make work. If Red is constitutes by being differentiate blue and green , blue is constituted by being different to red and and green, etc, then each of them is an (unspecified) X that is different to an (also unspecified) Y and Z....so they are all the same! The situation is even worse dealing with normatively laden qualia like pleasure and pain. Does pain really have n o characteristic s other than being different from pleasure?

Having said that, I can't be certain that you are putting forward a theory of pure difference. If instead you are just saying that qualia are [something unknown] that happens to have a structure of differences, then you are making a statement anyone could agree with and not solving the problem.

People might produce a more or less fuzzy and three dimensional projection of a rose in their mind, but no matter how sharp it is, the qualities the mental construct has are present on a spectrum where the other positions of that spectrum are other possible characteristics of objects. Green vs woody stems, red petals vs white ones, thorns vs fuzz, and so on.

Yes, qualia space has a structure. It doesn't follow from that a quale is characterised entirely by its location in qualia space. Qualia space would still have a structure,.If Qualia had intrinsic properties instead of being pure differences.That's the necessity/sufficiency distinction again.

Now that we’ve established that ideas and qualia are signs

That has been assumed, not established.

Any and all systems of signs that process an input according to a preexisting code necessarily have qualia, and share the sort of negativity which philosophers have so long restricted to the human subject. Your dog has qualia, your phone has qualia, your infection has qualia.

Obviously not, since humans have conditions in which qualia can go missing, or become hypertrophic, eg. colour blindness and synaesthesia.

I’m purposefully trying to avoid either camp as both camps require a type of dualism

I don't see why camp #1 requires dualism.

But brainwashing was never really a thing.

On the other hand, social conditioning does work. You can have societies where 98% of people believe in the same religion, and multiple societies who believe they are objectively the best, and so on. Social conditioning is the the thing that's implemented by anthem-singing, flag waving, public prayer, rallies, marches and parades, and a host of other than things that are seen as perfectly normal ... unlike the weird stuff cults get up to.

Brainwashing is a special or intensified form of conditioning ... so why wouldn't it work, when social conditioning generally does? One of the pieces of evidence against brainwashing is that US soldiers who had been "brainwashed" after being captured by communists reverted when they returned to the US. That could. be seen as brainwashing lacking a particular feature, the ability to lock-in permanently. It could also be seen as a success of the kind of social conditioning that's unnoticed and in the water. Attempted cult brainwashing into minority beliefs has the Achille's heal of attempting to instill minority beliefs, despite the fact that people generally want to fit in with majority beliefs. Cults try to get round this by separating their subjects from wider society, which doesn't entirely work , because they need to proselytize. On the other hand, small religions are large cults, and they work just fine.

Depending on how you define it, brainwashing is ubiquitous, non existent , or underwhelming effective. ,

There's is a reason I mentioned conceivability without contradiction. I don't particularly have it but into the standard framework about conceivability, metaphysical possibility, etc, and I dont. If you ground conceivability in non contradiction, then it has an objective basis, otherwise it's just an opinion. P-zombies don't contradict physics , but do contradict physicalism, which is why physicalists don't believe in them.

In 2014, we proposed that researchers try to find ways to make highly capable AIs “corrigible,” or “able to be corrected.” The idea would be to build AIs in such a way that they reliably want to *help *and cooperate with their programmers, rather than hinder them — even as they become smarter and more powerful, and even though they aren’t yet perfectly aligned.

Minimally, corrigibility just means present goals can be changed. There could be crude, brute force ways of doing that, like Pavlovian conditioning ,or overwriting an explicit coded UF.

The whole point of corrigibility is to scale to novel contexts and new capability regimes.

That can be done gradually, if everything is gradual.

@Roko

The reason that MIRI wasn’t able to make corrigibility work is that corrigibility is basically a silly thing to want, I can’t really think of any system in the (large) human world which needs perfectly corrigible parts, i.e. humans whose motivations can be arbitrarily reprogrammed.

That's normal much an argument against corrigibility, as an argument against perfect corrigibillity.

here he argues directly against “To be able to determine the future in a way that is itself determined falls short of being able to steer to a future of your choosing.”

Not really, because he is only saying , in typical compatiblist fashion, that you can bring about future in accordance with your desires ...which are determined. There is still no unpredictable turning point when you make a choice ... and therefore no steering to one of a number of physically possible futures...just a bunch of clockwork that's all inevitable.