## LESSWRONGLW

[anonymous]12y0

sigh

Not even close, any of you ;)

The question of whether there are any moral givens or not is analagous to the question of whether there are any mathematical givens (which was covered by E.Yudkowsky in an earlier series). Unfortunately, he doesn't seem to have learned the lesson.

Firstly, in the series on mathematics it was (correctly I think) put forward that even mathematics is not in fact, a priori or axiomatic. 2+2=4 for instance, can be empirically based on the observation that when you have two apples, and you add another two apples, you end up with a total of four apples. So this empirical fact can be taken as empirical evidence that 2+2=4. We don't directly perceieve the fact that 2+2=4, this mathematical fact is indirectly inferred from the empirical evidence.

Similarly, if there are any moral givens, I would agree that they have to result in real empirical differences. Again, we could never percieve moral givens directly (since they are abstract) - they would have to be indirectly inferred from empricial data.

The question is: what empirical differences would enable us to infer the moral givens? Ah, now that would be giving the game away wouldn't it? ;)

But again, clues can be obtained by looking at the arguments over whether math is objective or not, and reasoning by analogy for morality.

pure math is concerned with the objective logical properties of physical systems. These properties do exist... as demonstrated by the example of taking 2 apples, adding another 2 and always getting a total of 4. This is the empirical evidence for the postulated objective logical/mathematics properties.

But... applied math (for example probability theory) is not about objective logical properties, instead, it is about cognitive systems, or the process of making inferences about the objective logical/mathematical properties. The key point to note here, is that probablity theory only works because there do exist objective logical/mathematical properties of systems independently of observers. So the existence of these logical/mathematical properties is what ensures the coherence of probability theory. If there were no objective logical/mathematical properties independent of observers (ie for example adding 2 apples to another 2 apples did not always result in 4 apples in a consistent way) then probability theory would not work.

Just as math was concerned with logical properties of physical systems, so, if moral givens exist, they would be concerned with teleological properties of physical systems. And what science deal with this? Decision theory of course. It is the moral givens that provide the explanatory justification for decision theory.

Just as postulating real mathematical entities provided the explanatory justification for probablity theory, so too, does postulating the existence of real moral givens provide the explanatory justification for decision theory.

Why does decision theory work? What are these mysterious 'utilities' that keep being referred to? Are they preferences? No (in some cases at least), they're moral givens ;)

# 23

Continuation ofIs Morality Preference?

(Disclaimer:  Neither Subhan nor Obert represent my own position on morality; rather they represent different sides of the questions I hope to answer.)

Subhan:  "What is this 'morality' stuff, if it is not a preference within you?"

Obert:  "I know that my mere wants, don't change what is right; but I don't claim to have absolute knowledge of what is right—"

Subhan:  "You're not escaping that easily!  How does a universe in which murder is wrong, differ from a universe in which murder is right?  How can you detect the difference experimentally?  If the answer to that is 'No', then how does any human being come to know that murder is wrong?"

Obert:  "Am I allowed to say 'I don't know'?"

Subhan:  "No.  You believe now that murder is wrong.  You must believe you already have evidence and you should be able to present it now."

Obert:  "That's too strict!  It's like saying to a hunter-gatherer, 'Why is the sky blue?' and expecting an immediate answer."

Subhan:  "No, it's like saying to a hunter-gatherer:  Why do you believe the sky is blue?"

Obert:  "Because it seems blue, just as murder seems wrong.  Just don't ask me what the sky is, or how I can see it."

Subhan:  "But—aren't we discussing the nature of morality?"

Obert:  "That, I confess, is not one of my strong points.  I specialize in plain old morality.  And as a matter of morality, I know that I can't make murder right just by wanting to kill someone."

Subhan:  "But if you wanted to kill someone, you would say, 'I know murdering this guy is right, and I couldn't make it wrong just by not wanting to do it.'"

Obert:  "Then, if I said that, I would be wrong.  That's common moral sense, right?"

Subhan:  "Argh!  It's difficult to even argue with you, since you won't tell me exactly what you think morality is made of, or where you're getting all these amazing moral truths—"

Obert:  "Well, I do regret having to frustrate you.  But it's more important that I act morally, than that I come up with amazing new theories of the nature of morality.  I don't claim that my strong point is in explaining the fundamental nature of morality.  Rather, my strong point is coming up with theories of morality that give normal moral answers to questions like, 'If you feel like killing someone, does that make it right to do so?'  The common-sense answer is 'No' and I really see no reason to adopt a theory that makes the answer 'Yes'.  Adding up to moral normality—that is my theory's strong point."

Subhan:  "Okay... look.  You say that, if you believed it was right to murder someone, you would be wrong."

Obert:  "Yes, of course!  And just to cut off any quibbles, we'll specify that we're not talking about going back in time and shooting Stalin, but rather, stalking some innocent bystander through a dark alley and slitting their throat for no other reason but my own enjoyment.  That's wrong."

Subhan:  "And anyone who says murder is right, is mistaken."

Obert:  "Yes."

Subhan:  "Suppose there's an alien species somewhere in the vastness of the multiverse, who evolved from carnivores.  In fact, through most of their evolutionary history, they were cannibals.  They've evolved different emotions from us, and they have no concept that murder is wrong—"

Obert:  "Why doesn't their society fall apart in an orgy of mutual killing?"

Subhan:  "That doesn't matter for our purposes of theoretical metaethical investigation.  But since you ask, we'll suppose that the Space Cannibals have a strong sense of honor—they won't kill someone they promise not to kill; they have a very strong idea that violating an oath is wrong.  Their society holds together on that basis, and on the basis of vengeance contracts with private assassination companies.  But so far as the actual killing is concerned, the aliens just think it's fun.  When someone gets executed for, say, driving through a traffic light, there's a bidding war for the rights to personally tear out the offender's throat."

Obert:  "Okay... where is this going?"

Subhan:  "I'm proposing that the Space Cannibals not only have no sense that murder is wrong—indeed, they have a positive sense that killing is an important part of life—but moreover, there's no path of arguments you could use to persuade a Space Cannibal of your view that murder is wrong.  There's no fact the aliens can learn, and no chain of reasoning they can discover, which will ever cause them to conclude that murder is a moral wrong.  Nor is there any way to persuade them that they should modify themselves to perceive things differently."

Obert:  "I'm not sure I believe that's possible—"

Subhan:  "Then you believe in universally compelling arguments processed by a ghost in the machine.  For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value—1.  A mind is a physical device and you can't have a little blue woman pop out of nowhere and make it say 1 when the physics calls for it to say 0."

Obert:  "Suppose I were to concede this.  Then?"

Subhan:  "Then it's possible to have an alien species that believes murder is not wrong, and moreover, will continue to believe this given knowledge of every possible fact and every possible argument.  Can you say these aliens are mistaken?"

Obert:  "Maybe it's the right thing to do in their very different, alien world—"

Subhan:  "And then they land on Earth and start slitting human throats, laughing all the while, because they don't believe it's wrong.  Are they mistaken?"

Obert:  "Yes."

Subhan:  "Where exactly is the mistake?  In which step of reasoning?"

Obert:  "I don't know exactly.  My guess is that they've got a bad axiom."

Subhan:  "Dammit!  Okay, look.  Is it possible that—by analogy with the Space Cannibals—there are true moral facts of which the human species is not only presently unaware, but incapable of perceiving in principle?  Could we have been born defective—incapable even of being compelled by the arguments that would lead us to the light?  Moreover, born without any desire to modify ourselves to be capable of understanding such arguments?  Could we be irrevocably mistaken about morality—just like you say the Space Cannibals are?"

Obert:  "I... guess so..."

Subhan:  "You guess so?  Surely this is an inevitable consequence of believing that morality is a given, independent of anyone's preferences!  Now, is it possible that we, not the Space Cannibals, are the ones who are irrevocably mistaken in believing that murder is wrong?"

Obert:  "That doesn't seem likely."

Subhan:  "I'm not asking you if it's likely, I'm asking you if it's logically possible!  If it's not possible, then you have just confessed that human morality is ultimately determined by our human constitutions.  And if it is possible, then what distinguishes this scenario of 'humanity is irrevocably mistaken about morality', from finding a stone tablet on which is written the phrase 'Thou Shalt Murder' without any known justification attached?  How is a given morality any different from an unjustified stone tablet?"

Obert:  "Slow down.  Why does this argument show that morality is determined by our own constitutions?"

Subhan:  "Once upon a time, theologians tried to say that God was the foundation of morality.  And even since the time of the ancient Greeks, philosophers were sophisticated enough to go on and ask the next question—'Why follow God's commands?'  Does God have knowledge of morality, so that we should follow Its orders as good advice?  But then what is this morality, outside God, of which God has knowledge?  Do God's commands determine morality?  But then why, morally, should one follow God's orders?"

Obert:  "Yes, this demolishes attempts to answer questions about the nature of morality just by saying 'God!', unless you answer the obvious further questions.  But so what?"

Subhan:  "And furthermore, let us castigate those who made the argument originally, for the sin of trying to cast off responsibility—trying to wave a scripture and say, 'I'm just following God's orders!'  Even if God had told them to do a thing, it would still have been their own decision to follow God's orders."

Obert:  "I agree—as a matter of morality, there is no evading of moral responsibility.  Even if your parents, or your government, or some kind of hypothetical superintelligence, tells you to do something, you are responsible for your decision in doing it."

Subhan:  "But you see, this also demolishes the idea of any morality that is outside, beyond, or above human preference.  Just substitute 'morality' for 'God' in the argument!"

Obert:  "What?"

Subhan:  "John McCarthy said:  'You say you couldn't live if you thought the world had no purpose. You're saying that you can't form purposes of your own-that you need someone to tell you what to do. The average child has more gumption than that.'  For every kind of stone tablet that you might imagine anywhere, in the trends of the universe or in the structure of logic, you are still left with the question:  'And why obey this morality?'  It would be your decision to follow this trend of the universe, or obey this structure of logic.  Your decision—and your preference."

Obert:  "That doesn't follow!  Just because it is my decision to be moral—and even because there are drives in me that lead me to make that decision—it doesn't follow that the morality I follow consists merely of my preferences.  If someone gives me a pill that makes me prefer to not be moral, to commit murder, then this just alters my preference—but not the morality; murder is still wrong.  That's common moral sense—"

Subhan:  "I beat my head against my keyboard!  What about scientific common sense?  If morality is this mysterious given thing, from beyond space and time—and I don't even see why we should follow it, in that case—but in any case, if morality exists independently of human nature, then isn't it a remarkable coincidence that, say, love is good?"

Obert:  "Coincidence?  How so?"

Subhan:  "Just where on Earth do you think the emotion of love comes from?  If the ancient Greeks had ever thought of the theory of natural selection, they could have looked at the human institution of sexual romance, or parental love for that matter, and deduced in one flash that human beings had evolved—or at least derived tremendous Bayesian evidence for human evolution.  Parental bonds and sexual romance clearly display the signature of evolutionary psychology—they're archetypal cases, in fact, so obvious we usually don't even see it."

Obert:  "But love isn't just about reproduction—"

Subhan:  "Of course not; individual organisms are adaptation-executers, not fitness-maximizers.  But for something independent of humans, morality looks remarkably like godshatter of natural selection.  Indeed, it is far too much coincidence for me to credit.  Is happiness morally preferable to pain?  What a coincidence!  And if you claim that there is any emotion, any instinctive preference, any complex brain circuitry in humanity which was created by some external morality thingy and not natural selection, then you are infringing upon science and you will surely be torn to shreds—science has never needed to postulate anything but evolution to explain any feature of human psychology—"

Obert:  "I'm not saying that humans got here by anything except evolution."

Subhan:  "Then why does morality look so amazingly like a product of an evolved psychology?"

Subhan:  "Say that—forsake love and life and happiness, and follow some useless damn trend of the universe or whatever—and you will lose every scrap of the moral normality that you once touted as your strong point.  And I will be right here, asking, 'Why even bother?'  It would be a pitiful mind indeed that demanded authoritative answers so strongly, that it would forsake all good things to have some authority beyond itself to follow."

Obert:  "All right... then maybe the reason morality seems to bear certain similarities to our human constitutions, is that we could only perceive morality at all, if we happened, by luck, to evolve in consonance with it."

Subhan:  "Horsemanure."

Obert:  "Fine... you're right, that wasn't very plausible.  Look, I admit you've driven me into quite a corner here.  But even if there were nothing more to morality than preference, I would still prefer to act as morality were real.  I mean, if it's all just preference, that way is as good as anything else—"

Subhan:  "Now you're just trying to avoid facing reality!  Like someone who says, 'If there is no Heaven or Hell, then I may as well still act as if God's going to punish me for sinning.'"

Obert:  "That may be a good metaphor, in fact.  Consider two theists, in the process of becoming atheists.  One says, 'There is no Heaven or Hell, so I may as well cheat and steal, if I can get away without being caught, since there's no God to watch me.'  And the other says, 'Even though there's no God, I intend to pretend that God is watching me, so that I can go on being a moral person.'  Now they are both mistaken, but the first is straying much further from the path."

Subhan:  "And what is the second one's flaw?  Failure to accept personal responsibility!"

Subhan:  "So you take refuge in arguing from definitions?"

Obert:  "You know, when I reflect on this whole argument, it seems to me that your position has the definite advantage when it comes to arguments about ontology and reality and all that stuff—"

Subhan:  "'All that stuff'?  What else is there, besides reality?"

Obert:  "Okay, the morality-as-preference viewpoint is a lot easier to shoehorn into a universe of quarks.  But I still think the morality-as-given viewpoint has the advantage when it comes to, you know, the actual morality part of it—giving answers that are good in the sense of being morally good, not in the sense of being a good reductionist.  Because, you know, there are such things as moral errors, there is moral progress, and you really shouldn't go around thinking that murder would be right if you wanted it to be right."

Subhan:  "That sounds to me like the logical fallacy of appealing to consequences."

Obert:  "Oh?  Well, it sounds to me like an incomplete reduction—one that doesn't quite add up to normality."

Part of The Metaethics Sequence

Next post: "Where Recursive Justification Hits Bottom"

Previous post: "Is Morality Preference?"