What Would You Do Without Morality?

Followup toNo Universally Compelling Arguments

To those who say "Nothing is real," I once replied, "That's great, but how does the nothing work?"

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.

Devastating news, to be sure—and no, I am not telling you this in real life.  But suppose I did tell it to you.  Suppose that, whatever you think is the basis of your moral philosophy, I convincingly tore it apart, and moreover showed you that nothing could fill its place.  Suppose I proved that all utilities equaled zero.

I know that Your-Moral-Philosophy is as true and undisprovable as 2 + 2 = 4. But still, I ask that you do your best to perform the thought experiment, and concretely envision the possibilities even if they seem painful, or pointless, or logically incapable of any good reply.

Would you still tip cabdrivers?  Would you cheat on your Significant Other?  If a child lay fainted on the train tracks, would you still drag them off?

Would you still eat the same kinds of foods—or would you only eat the cheapest food, since there's no reason you should have fun—or would you eat very expensive food, since there's no reason you should save money for tomorrow?

Would you wear black and write gloomy poetry and denounce all altruists as fools?  But there's no reason you should do that—it's just a cached thought.

Would you stay in bed because there was no reason to get up?  What about when you finally got hungry and stumbled into the kitchen—what would you do after you were done eating?

Would you go on reading Overcoming Bias, and if not, what would you read instead?  Would you still try to be rational, and if not, what would you think instead?

Close your eyes, take as long as necessary to answer:

What would you do, if nothing were right?

 

Part of The Metaethics Sequence

Next post: "The Moral Void"

Previous post: "2-Place and 1-Place Words"

174 comments, sorted by
magical algorithm
Highlighting new comments since Today at 7:16 PM
Select new highlight date
Moderation Guidelines: Reign of Terror - I delete anything I judge to be annoying or counterproductiveexpand_more

Well I've argued that shoulds are overrated, that wants are enough. I really can't imagine you convincing me that I don't want anything more than anything else.

Did you convinve me that nothing is morally right, or that all utilities are 0.

If you convinced me that there is no moral rightness, I would be less inclined to take action to promote the things I currently consider abstract goods, but would still be moved by my desires and reactions to my immediate circumstances.

If you did persuade me that nothing has any value, I suspect that, over time, my desires would slowly convince me that things had value again.

If, 'convincing' includes an effect on my basic desires (as opposed to my inferrentially derived) then I would would not be moved to act in any cognitively mediated way (though I may still exhibit behaviors with non-cognitive causes).

Why the assumption that morality is analysable with utilities?

...it has been shown in countless experiments that people do not behave in accordance with this theorem. So what conclusions do you want to draw from this?

...you do realise there are many problems with rational choice theory right? See chapter 3 and 4 from 'Philosophy of Economics: A Contemporary Introduction' by Julian Reiss for a brief introduction to the theory's problems. If you can't get your hands on that, see lectures 4-6 from Philosophy of Economics: Theory, Methods, and Values http://jreiss.org/jreiss.org/Teaching.html for an even briefer introduction.

...what has this got to do with morality?

I'm going to take a look at the lectures you linked later.

For now:

...what has this got to do with morality?

Your morals are your preferences; if you say that doing A is more moral than doing B, you prefer doing A to B (barring cognitive dissonance). So if preferences can be reduced to utilities, morality can be too.

In fact, you'd have to argue that the axioms don't apply to morality, and justify that position.

I highly doubt that morals are preferences, with or without what you (assumedly loosely) term cognitive dissonance. One can have morals that aren't preferences:

If one is a Christian deontologist, one thinks everyone ought to follow a certain set of rules, but one needn't prefer that - one might be rather pleased that only oneself will get into heaven by the following the rules. One might believe things, events or people are morally "good" or "bad" without preferring or preferring not that thing, event or person. For instance, one might think that a person is bad without preferring that person didn't exist. One can believe one ought to do something, without wanting to do it. This is seen very often in most people.

And one can obviously have preferences which aren't morals. For instance, I can prefer to eat a chocolate now without thinking I ought to do so.

We should also be wary of equivocating on what we mean by "preferences". Revealed preference theory is very popular in economics, and it equates preferences with actions, which evidently stops us having preferences about anything we don't do, and thus means most of the usages of the word "preference" above are illegitimate. I think we normally mean some psychological state when we refer to a preference. For instance, I see the word used as "concious desire" pretty often.

If one is a Christian deontologist, one thinks everyone ought to follow a certain set of rules, but one needn't prefer that - one might be rather pleased that only oneself will get into heaven by the following the rules. One might believe things, events or people are morally "good" or "bad" without preferring or preferring not that thing, event or person. For instance, one might think that a person is bad without preferring that person didn't exist. One can believe one ought to do something, without wanting to do it. This is seen very often in most people.

I'm talking about personal morals here, i.e. "what should I do", which are the only ones that matter for my own decision making. For my own actions, the theorem shows that there must be some utility function that captures my decision-making, or I am irrational in some way.

Even if preferences are distinct from morals, each will still be expressible by a utility function or fail some axiom.

And one can obviously have preferences which aren't morals. For instance, I can prefer to eat a chocolate now without thinking I ought to do so.

That example is one where the errors are so low that it doesn't make sense to spend time thinking about it. If you value your happiness and consider it good, then you ought to eat the chocolate, but it may represent so little utility that it uses more just to figure that out.

We should also be wary of equivocating on what we mean by "preferences". Revealed preference theory is very popular in economics, and it equates preferences with actions, which evidently stops us having preferences about anything we don't do, and thus means most of the usages of the word "preference" above are illegitimate. I think we normally mean some psychological state when we refer to a preference. For instance, I see the word used as "concious desire" pretty often.

When I say preference I mean "what state do you want the world to be in". The problem of akrasia is well known, and it means that our actions don't always express our preferences.

Preferences should be over outcomes, while actions are not. An imbalance can be akrasia, or the result of a misprediction.

Regardless of how you define preference, if it meets the axioms then it can be expressed as a utility function. So every form of preference corresponds to different utility functions, whether it's revealed, actual, or some other thing.

Oh, so now you're just talking about personal morals. One of my examples already covered that: 'One can believe one ought to do something, without wanting to do it'. Why the presumption that utility functions capture decision-making? You acknowledge that preferences and hence utilities don't always lead to decisions. And why the assumption that not meeting the axioms of rational choice theory makes you irrational? Morality might not even be appropriately described by the axioms of rational choice theory; how can you express everyone's moral beliefs as real numbers? On the chocolate example, I can think I ought not eat the chocolate, but nevertheless prefer to eat it, and even actually eat; so your counterargument does not work. Given that you are not claiming all preferences meet the axioms - only "rational" preferences do (where's your support?) - you cannot say 'every form of preference corresponds to different utility functions, whether it's revealed, actual, or some other thing'. And again, we ought to ask ourselves whether preferences or rational preferences are actually the right sort of thing to be expressed by the axioms; can they really be expressed as real numbers?

Which axiom do you think shouldn't apply? If you can't give me an argument why not to agree with any given axiom, then why shouldn't I use them?

Given that you are not claiming all preferences meet the axioms - only "rational" preferences do (where's your support?) - you cannot say 'every form of preference corresponds to different utility functions, whether it's revealed, actual, or some other thing'.

Obviously, if I prefer X to Y, and also prefer Y to X, then I'm being incoherent and that can't be captured by a utility function. I expressly outlaw those kind of preferences.

Argue for a specific form of preference that violates the axioms.

If you can't give me an argument as to why all your axioms apply, then why should I accept any of your claims?

A specific form of preference that violates the axioms? Any preference which is "irrational" under those axioms, and you already acknowledged preferences of that sort existed.

If you can't give me an argument as to why all your axioms apply, then why should I accept any of your claims?

I see no counterexamples to any of the axioms. If they're so wrong, you should be able to come up with a set of preferences that someone could actually support.

A specific form of preference that violates the axioms? Any preference which is "irrational" under those axioms, and you already acknowledged preferences of that sort existed.

You need to argue that those are useful in some sense. Preferring A over B and B over A doesn't follow the axioms, but I see no reason to use such systems. Is that really your position, that coherence and consistency don't matter?

Any preference which is "irrational" under those axioms, and you already acknowledged preferences of that sort existed.

As an extremely basic example: I could prefer chocolate ice cream over vanilla ice cream, and prefer vanilla ice cream over pistachio ice cream. Under the Von Neumann-Morgenstein axioms, however, I cannot then prefer pistachio to chocolate because that would violate the transitivity axiom. You are correct that there is probably someone out there who holds all three preferences simultaneously. I would call such a person "irrational". Wouldn't you?

"Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden."

First Existential Crisis: Age 15

"Would you wear black and write gloomy poetry and denounce all altruists as fools?"

Been there, done that.

"But there's no reason you should do that - it's just a cached thought."

Realized this.

"Would you stay in bed because there was no reason to get up?"

Tried that.

"What about when you finally got hungry and stumbled into the kitchen - what would you do after you were done eating?"

Stare at the wall.

"Would you go on reading Overcoming Bias, and if not, what would you read instead?"

Shakespeare, Nitzsche

"Would you still try to be rational, and if not, what would you think instead"

No-- Came up with entire philosophy of "It doesn't matter if anything I say, do, or think is consistent with itself or each other... everything in my head has been set up by the universe- my parents ideas of right and wrong- television- paternalistic hopes of approving/forgiving/nonexistent god and his ability to grant immortality, so why should I worry about trying to put it together in any kind of sensible fashion? Let it all sort itself out...

"What would you do, if nothing were right?" What felt best.

First, can you clarify what you mean by "everything is permissible and nothing is forbidden"?

In my familiar world, "permissible" and "forbidden" refer to certain expected consequences. I can still choose to murder, or cheat, blaspheme, neglect to earn a living, etc; they're only forbidden in the sense of not wanting to experience the consequences.

Are you suggesting I imagine that the consequences would be different or nonexistent? Or that I would no longer have a preference about consequences? Or something else?

To be perfectly honest, if I had my morality stripped away, and I thought could get away with it, I'd rape as many women as possible.

Not joking; my tastes already run towards domination and BDSM and the like, and without morality, there'd be no reason to hold back for fear of traumatizing my partners, other than the fear of the government punishing me for doing so.

Your honesty is appreciated.

Personally, I would aim to change things so that the attainment of any goal whatsoever is possible for me to achieve. Essentially, to modify myself into a universe conquering, unfriendly super-intelligence.

But why rape? I mean, it just seems so arbitrary and trivial...

The way I frame this question is "what if I executed my personal volition extrapolating FAI, it ran, created a pretty light show, and then did nothing, and I checked over the code many times with many people who also knew the theory and we all agreed that it should have worked, then tried again with completely different code many (maybe 100 or 1000 or millions) times, sometimes extrapolating somewhat different volitions with somewhat different dynamics and each time it produced the same pretty light show and then did nothing. Lets say I have spend a few thousand years on this while running as an upload. Now what?"

In this scenario there's no optimization reason I shouldn't just execute cached thoughts. In fact, that's pretty much what anything I do in this scenario amounts to doing. Executing cached thoughts does, of course, happen lawfully, so there is a reason to dress in black etc in that sense. I used to be pretty good at writing some sad but mostly non-gloomy poetry and denouncing people as fools. Might be even more fun to do that with other modified upload copies of myself. When that got old, maybe use my knowledge of FAI theory to build myself a philosophy of math oracle neural module. Hard to guess how my actions would differ once it was brought on-line. It seems to me that it might add up to normality because there might be an irreducible difference between utility for me and utility for an external AGI even if it was an extrapolation of my volition, but for now I'm a blind man speculating on the relative merits of Picasso and Van Gogh.

Honestly I'm much less concerned about this scenario than I once was. Pretty convinced that there are ways to extrapolate me that do something even if they discover infinite computing power.

Dynamically linked: No-one but nerds and children care what moral philosophies say anyway, at least, not in a way that effects their actions. You, TGGP and Unknown are very atypical. Poke is much closer to correct. If anything, when the dust settled the world would be more peaceful if most people understood the proof.

Eric Mesoy: If utilities = 0 then dying from malnourishment isn't horrible.

Andy M: Your answer sounds more appropriate for someone fairly shallow and 20 years old who discovers that the world or his life will end in 6 months than for someone for whom utilities are set to zero or morality is lost.

Constant Pablo and especially Sebastian: Clearly thought! I should probably start reading your comments more carefully in the future.

Laura: Why unsympathetic? My guess is that you still confuse my and Eliezer's aspirations with some puerile Nietzschean ambition. I like who I am now too thank you very much, and if my extrapolated volition does want to replace who I am it is for reasons that I would approve of if I knew them, e.g. what it will replace me with is not "completely different, incomprehensible, and unsympathetic". That's the difference between a positive and a negative singularity. Death isn't abhorrent, life/experience/growth/joy/flourishing/fulfillment, rather, is good, and a universe more full of them more good than one less full, whether viewed from inside or from outside. Math is full of both death and flourishing and is not lessened by the former.

Phil: Very entertaining and thoughtful post.

I don't believe in objective morality in the first place.

My moral system has only one axiom:

Maximise your utility.

If nothing were right, I'd still go on maximising my utility. I don't try to maximise my utility because I believe utility maxismisation is some apriori "right" thing to—I try to maximise my utility because I want to. Unless your proof changed my desires (in which case I don't know what I would do), I expect I would go on trying to maximise my utility.

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.

Suppose I proved that all utilities equaled zero.

If I still feel hunger then food has an utility > 0. If I don't feel anything anymore, then I wouldn't care about anything.

So our morality is defined by our emotions. The decisions I make are a tradeoff. Do I tip the waiter? Depends on my financial situation and if I'm willing to endure the awkwardness of breaking a social convention. Yes, I've often eaten without tipping.

Do I save the human in need? Yes, I have the tendency to do so, although this also depends on a series of factors. And I'm aware that this is also hardwired empathy. Abstract moral principles are just rationalizations from our emotionally hardwired brain.

I cannot imagine myself without morality because that wouldn't be me, but another brain.

Does your laptop care if the battery is running out? Yes, it will start beeping, because it is hardwired to do so. If you removed this hardwired beeping you have removed the laptop's morality.

Morality is not a ghost in the machine, but it is defined by the machine itself.

Eliezer you can prove to me that all utilities are 0 but since that wouldn't change my emotional wiring, for me some utilities would still be != 0.

"Morality" generally refers to guidelines on one of two things:

(1). Doing good to other sentients. (2). Ensuring that the future is nice.

If you wanted to make me stop caring about (1), you could convince me that all other sentients were computer simulations who were different in kind than I was, and that there emotions were simulated according to sophisticated computer models. In that case, I would probably continue to treat sentients as peers, because things would be a lot more boring if I started thinking of them as mere NPCs.

If you wanted to make me stop caring about (2), you could tell me that I was living in computer simulation that would grant my every request (similar to the plot of this novel). If that were the case, I would set up sophisticated games for myself. Just taking the path of least resistance and maximizing momentary dopamine release would get boring quickly. (There's a reason why you see more kids eating candy than adults.) I would think carefully before I even experimented with maximizing dopamine release, since it would make everything else seem petty by comparison.

Either way, you would be ruining the secret to happiness:

"The secret of happiness is to find something more important than you are and dedicate your life to it." - Dan Dennet

I would be depressed and do nothing at all, as empirically verified.

Gotta have _some_ answer to "what is good".

How did I reconcile this? What is the right morality when everyone's morality differs?

Well, mine, of course. What else?

If heaven and Earth, despoiled of its august stamp could ever cease to manifest it, if Morality didn't exist, it would be necessary to invent it. Let the wise proclaim it, and kings fear it.

I'm still trying to understand what Eliezer really means by this question. Here is a list of a few reasons why I don't kill the annoying kid across the street. Which of these reasons might disappear upon my being shown this proof?

1. The kid and his friends and family would suffer, and since I don't enjoy suffering myself, my ability to empathise stops me wanting to.

2. I would probably be arrested and jailed, which doesn't fit in with my plans.

3. I have an emotional reaction to the idea of killing a kid (in such circumstances -- though I'm not actually sure that this disclaimer is necessary): it fills me with such revulsion that I doubt I would actually be able to carry out the task. My emotions would prevent my body working properly.

4. I recognise that the kid is not causing very much harm to me. It seems fair to cause little harm to him in return.

5. My family and friends might suffer because they might imagine they could have prevented my doing this and failed to (guilt, I suppose is the word); see 1, also this reaction is even stronger because I have vested interests in my friends and family not suffering.

6. I myself would suffer guilt as a result of 1, 3 and 4, and I don't enjoy suffering.

I suppose 2 wouldn't change, because "it all adds up to normality" (although, as I said in my last comment, I don't think this could add up to normality; hence my trying to understand the question better), so other people's actions would not be altered. It would be something in me that changed: a new understanding that affected my value judgements. What would it affect? The fact that I don't like suffering, which would take out 1 and 6? My ability to empathise, taking out 1 and 5? My emotional reactions, taking out 3 and possibly 6? My ability to judge what is fair and what is unfair -- or the fact that I care about acting fairly -- taking out 4?

Perhaps all I've done here is attempt to Taboo the concept of morality for one particular case. Saying "it's immoral to kill the kid" suggests that the concept of morality not really existing makes sense. My list reveals that I, at least, can't make sense of it. I'm still confused as to what the question really means.

Michael Vassar, I read that and laughed and said, "Oh, great, now I've got to play the thought experiment again in this new version."

Albeit I would postulate that on every occasion, the FAI underwent the water-flowing-downhill automatic shutdown that was automatically enginereed into it, with the stop code "desirability differentials vanished".

The responses that occurred to me - and yes, I had to think about it for a while - would be as follows:

*) Peek at the code. Figure out what happened. Go on from there.

Assuming we don't allow that (and it's not in the spirit of the thought experiment), then:

*) Try running the FAI at simpler extrapolations until it preserved desirability; stop worrying about anything that was in the desirability-killing extrapolations. So if being "more the people we wished we were" was the desirability-killer, then I would stop worrying about that, and update my morality accordingly.

*) Transform myself to something with a coherent morality.

*) Proceed as before, but with a shorter-term focus on when my life's goals are to be achieved, thinking less about the far future - as if you told me that, no matter what, I had to die before a thousand years were up.

Dynamically Linked said:

Seriously, most moral philosophies are against cheating, stealing, murdering, etc. I think it's safe to guess that there would be more cheating, stealing, and murdering in the world if everyone became absolutely convinced that none of these moral philosophies are valid.

That's not a safe guess at all. And in fact, is likely wrong.

You observe that (most?) moral philosophies suggest your list of sins are "wrong". But then you guess that people tend not to do these things because the moral philosophies say they are wrong.

There's another alternative. It could be that human behavior is generally constrained by something else (e.g. utility maximization), and it is this far more fundamental force which prevents much "immoral" sinning, and that explicit "moral philosophies" are actual constrained by observed human behavior.

In other words, you've reversed cause and effect.

(Thus: the moral philosophies are not valid, but the behavior constraints are still rational nonetheless.)

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.

There are different ways of understanding that. To clarify, let's transplant the thought experiment. Suppose you learned that there are no elephants. This could mean various things. Two things it might mean:

1) That there are no big mammals with trunks. If you see what you once thought was an elephant stampeding in your direction, if you stay still nothing will happen to you because it is not really there. If you offer a seeming elephant peanuts, the peanuts will pass through the trunk which is not there and will fall to the ground.

2) That big mammals with trunks are not elephants. If you see what you once thought was an elephant stampeding in your direction, if you stay still you will be trampled. If you offer a seeming elephant peanuts, the animal will accept and enjoy the peanuts.

Among those who would be persuaded that there is no morality, those who interpret the 'no morality' claim as analogous to (1) will change their behavior. Those who interpret the 'no morality' claim as analogous to (2) will not change their behavior.

(1) is a substantial claim about the world. (2) is a claim about language, about what how things should be labeled.

Those who claim that they would change nothing in their activity are treating the no-morality hypothetical as if it were merely a claim about how things should be labeled. Those who claim that they would change their behavior are treating the no-morality hypothetical as if it were a substantial claim about the world.

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.
I'd do precisely the same thing I would do upon being informed that an irresistible force has just met an immovable object:

Inform the other person that they didn't know what they were talking about.

Nothing is right, you say? What a very curious position to take.

I just had another idea: maybe I would begin to design an Unfriendly AI. After all, being an evil genius would at least be fun, and besides, it would be a way to get revenge on Eliezer for proving that morality doesn't exist.

It seems people are interpreting the question in two different ways, one that we don't have any desires any more, and therefore no actions, and the other in the more natural way, namely that "moral philosophy" and "moral claims" have no meaning or are all false. The first way of interpreting the question is useless, and I guess Eliezer intended the second.

Most commenters are saying that it would make no difference to them. My suspicion is that this is true, but mainly because they already believe that moral claims are meaningless or false.

Possibly (I am not sure of this) Eliezer hopes that everyone will answer in this way, so that he can say that morality is unnecessary.

Personally, I agree with Dynamically Linked. I would start out by stealing wallets and purses, and it would just go downhill from there. In other words, if I didn't believe that such things were wrong, the bad feeling that results from doing them, and the idea that it hurts people, wouldn't be strong enough to stop me, and once I got started, the feeling would go away too-- this much I know from the experience of doing wrong. And once I had changed the way I feel about these things, the way I feel about other things (too horrible to mention at the moment) would begin to change too. So I can't really tell where it would end, but it would be bad (according to my present judgment).

There are others who would follow or have followed the same course. TGGP says that over time his life did change after he ceased to believe in morality, and at one point he said that he would torture a stranger to avoid stubbing his toe, which presumably he would not have done when he believed in morality.

So if it is the case that Eliezer hoped that morality is unnecessary to prevent such things, his hope is in vain.

Isn't this the movie Groundhog Day, but with certain knowledge that the world will reset daily forever? No happy ending.

I'd just get really, really bored. Studying something (learning the piano, as he does in the movie) would be the only open-ended thing you could do. Otherwise, you'd be living forever with the same set of people, and the same more-or-less limited set of possibilities.

The existence of objective moral values seems to have been a topic in the discussion below. I would like to state my view on the matter, since it connects to the original article. I define objective moral values as moral values that exist independently of the existence of life.

I do not believe that any objective moral values exist and I usually argue as follows: I ask three questions: When did objective moral values come into existence? Have we ever observed them or how can we observe them? Do we need objective moral values to explain anything that we cannot otherwise explain?

First question: A reasonable answer is that objective moral values exist in the same manner that mathematics or logic exists, and how they came into existence or in what manner they exist is a topic in itself and I will not address the issue further here. I will just state that this seems to be a reasonable answer to the question, but I am already having doubts.

Second question: I will go out on a limb here and suggest that no one has ever observed an objective moral value, but it is interesting how it could be observed. Probably not by suddenly observing divine letters in the sky. For me it is actually a problem just imagining how to observe an objective moral value. But let us assume that someone has a better imagination than I, thou increasing my doubts.

Third question: The third question is what I really consider to be the nail in the coffin, since I can not think of anything that we actually need objective moral values to explain in this universe. Every phenomenon I can think of is better explained by something else, so by Occam's razor I choose to not include objective moral values into my world view.

So what I instead believe exist are subjective moral values, and then I mean subjective in the sense that for example preference of art is subjective. For example if I state that a particular piece of art is beautiful then I do not state that it is beautiful in a higher objective sense, but instead that it is beautiful to me.

The answer to the above questions are very different for subjective moral values. First question: I believe subjective moral values came into existence when life came into existence, since subjective moral values depend of life itself and they exist perhaps in the same sense that thoughts exists.

Second question: Subjective moral values are observed every day, at least in an indirect way. When I decide to buy ecological groceries I express a subjective moral value. In the same way I believe every living thing express subjective moral values through behaviour, for example an antelope running away from a lion, then the antelope expresses that it would not like to be eaten and as such being eaten is bad according to the antelope, while the lion has a subjective moral value that the antelope being eaten by the lion is good.

Third question: Subjective moral values also has some explanatory power, if we know the subjective moral values of an individual we can pretty much explain that individual's behaviour.

So to conclude I do believe all living organisms express and have subjective moral values, which are dependent on the organism itself and that there is no objective moral values which can ever be observed. And to connect with the original post I would not be very alarmed in that situation since it goes well with my current view of reality.

Sorry for the long comment. I tried shortening it down a bit, but now I instead I feel like I have excluded a lot of important points and that my arguments are a bit brief. I hope you get the overall idea.

I once asked a friend a similar question. His answer was, "Everything."

I don't understand this post. Asking me to imagine that all utilities equal zero is like asking to imagine being a philosophical zombie. I'd do exactly the same as before of course.

That's what I'd do too. If all utilities equal 0, then there's no reason not to act as though utilities are non-zero. There's also no reason to privilege any set of utilities over any other set. Firstly this means that if there's any probability that utilities don't really all equal zero (maybe EY's proof is flawed, maybe my brain made an error in hearing the proof and it really proves something else entirely...) then the p-mass on "all utilities are 0" should have no effect on my decisions. If it actually is true, with probability 1 (which EY says doesn't exist, but I'm not sure whether that's true[*]), then I have no reason to behave differently, nor any reason to behave the same, so in some sense I "may as well" behave the same - but I can't formalise this, because of course there's no negative utility attached to "changing one's behaviour". I wonder if it can be got out of a limit - whether my behaviour in the limit as P(all utilities are 0) goes to 1 ought to define my behaviour when it equals 1 - but defining behaviour of limit to equal limit of behaviour is precisely what makes unbounded utility functions Dutch-bookable (as EY showed in Trust in Bayes).

So... I'd behave exactly as I do now, believing in utility functions, but I can't justify that if I know for certain that all utilities are 0. Given that I haven't thus far accepted the argument that '0 and 1 are not probabilities', this is disturbing and confusing, hence maybe I should accept that argument; at least, updating on this has caused me to raise my probability estimate that 0 and 1 are not probabilities.

[*] If I were sure that ¬\exist X : P(X) = 1, then P(¬\exist X : P(X) = 1) = 1, in which case things break. A formal system can't talk about itself coherently. (That 'coherently' is necessary, because Gödel numberings do allow PA to do something that looks to us like "talk about itself", but you can't conclude PA is talking about itself unless you have some metatheory outside PA, which ends up recursing to a skyhook.)

What would I do if you destroyed my moral philosophy?

Well, empathy for others is built into me (and all other non-psychopaths) whether I like it or not. It isn't really affected by propositions. So not much would really change. Proving that moral truths didn't exist would free us all up to act "however we like," but I can still pigheadedly "like" to be nice.

What did you mean by "all utilities are 0"?

What did you mean by "all utilities are 0"?

Utility Functions are a way to represent preferences, such that states of the universe that map to larger numbers are more desirable. If every state of the universe mapped to the same utility, for example 0, that represents having no preference about anything at all.

Well, empathy for others is built into me...

It looks like you got the core point of this article.

Yeah, I'm somewhat familiar with the concept of utility... I suppose what I wanted clarified was "utility for whom," but I guess it's obvious Eliezer was being tongue-in-cheek about this.

Still, it's surprising how often you find people saying "nothing matters, because the universe is heading toward heat death/there is no afterlife/we're just chemicals." What can you do but laugh and remember the opening of Annie Hall? :)

What would I do?

I'd make a like a typical nihilistic postmodernist and adopt the leftist modus operandi of decrying the truth and moral content of everyone's arguments except my own.

michael vassar: I meant "horrible" from my current perspective, much like I would view that future me as psychopathic and immoral. (It wouldn't, or if it did, it would consider them meaningless labels.)

Dynamically Linked: I'm using my real name and I think I'd do things that I (and most of the people I know) currently consider immoral. I'm not sure about using "admit" to describe it, thought, as I don't consider it a dark secret. I have a certain utility function which has a negative valuation of a hypothetical future self without the same utility function. While my current utility function has an entry for "truth", that entry isn't valued above all the others that Eliezer suggests disproving the way I understand it. But then, I'm still a bit confused on how the question should be read.

This is a spectacularly ill-posed question. For one thing, it seems to blur the distinction between morality and values in general, by asking such questions like "Would you stay in bed because there was no reason to get up?" What does that have to do with morality?

When you get rid of a sense of values, the result is clinical depression (and generally, a non-functional person). When you get rid of a sense of morality, the result is a psychopath. Psychopaths, unlike the depressed, are quite functional.

So the question reduces to, what would you do if you were a psychopath? This is perhaps interesting to think about, but hard to answer, since most of us are not psychopaths and find it extremely difficult to imagine what it would be like to be one. And if you were one, you wouldn't be you, since the fundamental structure of your personality would be vastly different.

I wonder if Eliezer is planning to say that morality is just an extrapolation of our own desires? If so, then my morality would be an extrapolation of my desires, and your morality would be an extrapolation of yours. This is disturbing, because if our extrapolated desires don't turn out to be EXACTLY the same, something might be immoral for me to do which is moral for you to do, or moral for me and immoral for you.

If this is so, then if I programmed an AI, I would be morally obligated to program it to extrapolate my personal desires-- i.e. my personal desires, not the desires of the human race. So Eliezer would be deceiving us about FAI: his intention is to extrapolate his personal desires, since he is morally obligated to do so. Maybe someone should stop him before it's too late?

All: I'm really disappointed that no-one else seems to have found my "after the FAI does nothing" frame useful for making sense of this post. Is anyone interested in responding to that version? It seems so much more interesting and complete than the three versions E.C. Hopkins gave.

Dynamically: My "moral philosophy" if you insist on using that term (model of a recipe for generating a utility function considered desirable by certain optimizers in my brain would be a better term) is the main thing that HAS told me to steal, cheat, and murder. Simpler optimization patterns based on herd behavior, operant conditioning, moderately strong typical male primate aversions to violence, projections of parental authority through internalized neural agents etc have told me not to do those things and have won enough attention from the more complex optimizers to convince them (since the complex optimizers can reflect and be convinced of things) not to do so after all, and upon examination those simpler patterns have mostly turned out to be right judged by the standards of the moral philosophy. On a few occasions that I am aware of my conditioned etc morality was very wrong (judged reflectively), and possibly on a few other occasions, but they were much much less wrong than the occasions on which they were right and casual examination of my reflective self was in doubt.

Dynamically: It appears that you have a fixed preconception of what behavior "human nature" requires, and you will not accept answers that don't adhere to that preconception.

Wow- far too much self-realization going on here... Just to provide a data point, when I was in high school, I convinced an awkward, naive, young catholic boy who had a crush on me of just this point... He attempted suicide that day.

....

For follow up, he has been in a very happy exclusive homosexual relationship for the past three years.

Maybe I didn't do such a bad thing...

Roland wrote:

.I cannot imagine myself without morality because that wouldn't be me, but another brain.

Does your laptop care if the battery is running out? Yes, it will start beeping, because it is hardwired to do so. If you removed this hardwired beeping you have removed the laptop's morality.

Morality is not a ghost in the machine, but it is defined by the machine itself.

Well put.

I'd stop being a vegetarian. Wait; I'm not a vegetarian. (Are there no vegetarians on OvBias?) But I'd stop feeling guilty about it.

I'd stop doing volunteer work and donating money to charities. Wait; I stopped doing that a few years ago. But I'd stop having to rationalize it.

I'd stop writing open-source software. Wait; I already stopped doing that.

Maybe I'm not a very good person anymore.

People do some things that are a lot of work, with little profit, mostly for the benefit of others, that have no moral dimension. For instance, running a website for fans of Harry Potter. Writing open-source software. Organizing non-professional conventions.

(Other people.)

Some people on this blog have said that they would do something different. Some people on this blog have said that they actually came to that conclusion, and actually did something different. Despite these facts, we have commenters projecting themselves onto other people, saying that NO ONE would do anything different under this scenario.

Of course, people who don't think that anything is right or wrong also don't think it's wrong to accuse other people of lying, without any evidence.

Once again, I most certainly would act differently if I thought that nothing was right or wrong, because there are many things that I restrain myself from doing precisely because I think they are wrong, and for no other reason-- or at least for no other reason strong enough to stop me from doing them.

Another perspective on the meaning of morality:

On one had there is morality as "those things which I want." I would join a lot of people here in saying that I think that what I want is arbitrary in that it was caused by some combination of my nature and nurture, rather than being in any fundamental way a product of my rationality. At the same time I can't deny that my morality is real, or that it governs my behavior. This is why I would call myself a moral skeptic, along the lines of Hume, rather than a nihilist. I also couldn't become an egoist without giving up my moral skepticism.

So what would it mean, and what would I do if I was stripped of this sort of morality? I don't think I can properly imagine it since I don't believe I can even imagine person-hood without this kind of morality.

On the other hand there is the morality that is the set of rules I use to bring my various wants and desires into harmony with each other. I can imagine this being removed from me while I still remain me, and I think this would result in a lot of incoherent and possibly hedonistic behavior before I recreated something like it.

The point is: even in a moralless meaningless nihilistic universe, it all adds up to normality.

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.
I'm a physical system optimizing my environment in certain ways. I prefer some hypothetical futures to others; that's a result of my physical structure. I don't really know the algorithm I use for assigning utility, but that's because my design is pretty messed up. Nevertheless, there is an algorithm, and it's what I talk about when I use the words "right" and "wrong".
Moral rightness is fundamentally a two-place function: it takes both an optimization process and a hypothetical future as arguments. In practice, people frequently use the curried form, with themselves as the implied first argument.

Suppose I proved that all utilities equaled zero.
That result is obviously false for my present self. If the proof pertains to that entity, it's either incorrect or the formal system it is phrased in is inappropriate for modeling this aspect of reality.
It's also false for all of my possible future selves. I refuse to recognize something which doesn't have preferences over hypothetical futures as a future-self of me; whatever it is, it's lost too many important functions for that.

I would play a bunch of video games -- not necessarily Second Life, but just anything to keep my mind occupied during the day. I would try to join some sort of recreational sports league, and I would find a job that paid me just enough money to solicit a regular supply of prostitutes.

If I were actually convinced that there is no right or wrong (very unlikely), I would probably do everything I could to keep the secret from getting out.

Even if there is no morality, my continued existence relies on everyone else believing that there is one, so that they continue to behave altruistically towards me.

A brief note to the (surprisingly numerous) egoists/moral nihilists who commented so far. Can't you folks see that virtually all the reasons to be skeptical about morality are also reasons to be skeptical about practical rationality? Don't you folks realize that the argument that begins questioning whether one should care about others naturally leads to the question of whether one should care about oneself? Whenever I read commenters here proudly voicing that they are concerned with nothing but their own "persistence odds", or that they would willingly torture others to avoid a minor discomfort to themselves, I am reminded of Kieran Healy's remarks about Mensa, "the organization for highly intelligent people who are nevertheless not quite intelligent enough not to belong to it." If you are so smart that you can see through the illusion that is morality, don't be so stupid to take for granted the validity of practical rationality. Others may not matter, but if so you probably don't either.

I'd do everything I do now. You can't escape your own psychology and I've already expressed my skepticism about the efficacy of moral deliberation. I'll go further and say that nobody would act any differently. Sure, after you shout in from the rooftops, maybe there will be an upsurge in crime and the demand for black nail polish for a month or so but when the dust settled nothing would have changed. People would still cringe at the sight of blood and still react to the pain of others just as they react to their own pain. People would still experience guilt. People would still find it hard to lie to loved ones. People would still eat when they got hungry and drink when they got thirsty. We vastly overestimate our ability to alter our own behavior.

The post says "when you finally got hungry [...] what would you do after you were done eating?", which I take to understand that I still have desire and reason to eat. But it also asks me to imagine a proof that all utilities are zero, which confuses me because when I'm hungry, I expect a form of utility (not being hungry, which is better than being hungry) from eating. I'm probably confused on this point in some manner, though, so I'll try to answer the question the way I understand it, which is that the more abstracted/cultural/etc utilities are removed. (Feel free to enlighten/flame me on this point.)

I expect that I'd probably do a number of things that I currently avoid, most of which would probably be clustered under "psychopathy". I think there's something wrong with them now, but I wouldn't think that there was something wrong with them post-proof. Most of my behavior would probably stay the same due to enlightened self-interest, and I'm not sure what would change. For example, the child on the train tracks. My current moral system says I should pull them off, no argument. If you ripped that system away, I'd weigh off the possible benefit the child might bring me in the future (since it's in my vicinity, it's probably a First World kid with a better than average chance of a good education and a productive life) against considerations like overpopulation. I'd cheat on my Significant Other if I thought it would increase my expected happiness (roughly: "if I can get away with it"). I'd go on reading Overcoming Bias and being rational because rationality seems like a better tool for deciding what to eat when hungry, such as at the basic level of bread vs. candles, and generalise from there. (If that goes away, I probably die horribly from misnourishment.)

Eliezer, I've got a whole set of plans ready to roll, just waiting on your word that the final Proof is ready. It's going to be bloody wicked... and just plain bloody, hehe.

Since my current moral system is pretty selfish and involves me doing altruistic things to make me happy, I wouldn't change a thing. At first glance it might appear that my actions should be more shortsighted since my long-term goals wouldn't matter, but my short-term goals and happiness wouldn't matter just as much. Is this thought exercise another thing that just all adds up to normality?

Like many others here, I don't believe that there is anything like a moral truth that exists independently of thinking beings (or even dependently on thinking beings in anything like an objective sense), so I already live in something like that hypothetical. Thus my behavior would not be altered in the slightest.

I am already fairly well convinced of this; I am hoping against hope you have something up your sleeve to change my mind.

I had this revelation sometime back. I tried living without meaning for a week, and it turn out that not a whole lot changed. Oops?

I'm already convinced that nothing is right or wrong in the absolute sense most people (and religions) imply.

So what do I do? Whatever I want. Right now, I'm posting a comment to a blog. Why? Not because it's right. Right or wrong has nothing to do with it. I just want to.

Eliezer: I'm finding this one hard, because I'm not sure what it would mean for you to convince me that nothing was right. Since my current ethics system goes something like, "All morality is arbitrary, there's nothing that's right-in-the-abstract or wrong-in-the-abstract, so I might as well try to make myself as happy as possible," I'm not sure what you're convincing me of--that there's no particular reason to believe that I should make myself happy? But I already believe that. I've chosen to try to be happy, but I don't think there's a good 'reason' for it.

On the other hand, maybe I right now am the end result you're looking for. In which case, yes, I do tip cabdrivers; no, I don't cheat; and usually I'd pull the kid off, if there weren't much risk to me.

There are several things wrong with this post. Firstly, I'm sure different people would react to being convinced their moral philosophy was wrong in different ways. Some might wail and scream and commit suicide. Some might question search further and try to find a more convincing moral philosophy. Some would just carry go on living there lives and not caring.

Furthermore, the outcome would be different if you could simultaneously convince everyone in a society, and give everyone the knowledge that everyone had been convinced. Perhaps the society would break down as the police and institutions upholding the law abandoned their tasks due to both apathy and a desire to capitalise on the new state of affairs, with no guilt. Who knows.

The fundamental flaw of this article is that it asks us to consult our intuitions about what would happen if so and so. Consulting our intuitions is something I believe this site shuns, so it is quite hypocritical that the author has requested we place so much emphasis on them in this instance. Furthermore, anyone answering this question who believes in moral eliminativism has a confirmation bias to say 'nothing would change' as this is seen by them to support their beliefs.

Consulting your intuition in a matter of descriptive questions should be done with caution. (But even then, it's not forbidden or even really discouraged, since intuition can offer valuable--if non-rigorous--insights.) Using your intuition when confronting normative or prescriptive problems, on the other hand, is perfectly fine, because there's no "should" without an intuition about what "should" be. (Unless, of course, you think that normative problems are also descriptive, in which case you believe in objective morality, which has its own problems.)

The fundamental flaw of this article is that it asks us to consult our intuitions about what would happen if so and so.

This seems a bizarre claim. If you think the conclusion that EY is intuition-pumping to advocate for is false (which you seem to, given your first two paragraphs), surely that's a more fundamental flaw than the fact that he's intuition-pumping to advocate for it.

That said, I'll admit I don't really understand on what grounds you oppose the conclusion. (In fact, it's not even clear to me what you think the advocated-for conclusion is.)

I mean, your point seems to be that not everyone would respond to discovering that "nothing is moral and nothing is right; that everything is permissible and nothing is forbidden" in the same way, either as individuals or as collectives. And I agree with that, but I don't see how it relates to any claims made by the post you reply to.

Taking another stab at clarifying your objections might be worthwhile, if only to get clearer in your own mind about what you believe and what you expect.

I have no idea what the conclusion of this article is. I suspect the author wants to argue for moral eliminativism, and hopes to support moral eliminativism by claiming that nothing would change if someone (or is it everyone?) was convinced their moral beliefs were wrong. I'm not sure how exactly the author intends that to work out.

But in any case, my comment only intended to criticise the methodology of the article, and was not aimed at discussing moral eliminativism. I simply pointed out that the question asked - what would happen is someone (or everyone?) was convinced their moral beliefs were wrong - was vague in several important aspects. And any results from intuition would be suspect, especially if the person holding those intuitions was a moral eliminativist. I was not "objecting" to anything, as the article didn't actually make any positive claims.

I might as well clarify and support myself by listing all the variations on the question possible.

(1) What would you personally do if you had no moral beliefs? (2) What would you personally do if you believed in (some form of) moral eliminativism - e.g. that nothing is right or wrong? (3) What would you personally do if you were convinced your moral beliefs were wrong? What would a randomly selected person from the populace of the Earth do if (1), (2) or (3) happened to them? What would happen if everyone in a society/ the world simultaneously had (1), (2) or (3) happen to them?

I simply pointed out that the question asked - what would happen is someone (or everyone?) was convinced their moral beliefs were wrong - was vague in several important aspects.

It's vague in an additional way: you interpreted it to mean "what would you do if you were convinced that your moral beliefs were wrong". But I think Eliezer was asking "what would you do if your moral beliefs actually were wrong and you were aware of that."

That has its own problem. It's like asking "if someone could prove that creationism was true and evolution isn't, would you agree that scientists are closed-minded in rejecting it?" A hypothetical world in which creationism was true wouldn't be exactly like our own except that it contains a piece of paper with a proof of creationism written down on it. In a world where creationism really was true, scientists would either have figured it out, or would have not figured it out but would be a lot more clueless than actual-world scientists. Likewise, a world where moral beliefs were all wrong would be very unlike our world, if indeed it's a coherent concept at all--it would not be a world that is exactly like this one with the exception that I am now in possession of a proof.

It's like asking "if someone could prove that creationism was true and evolution isn't, would you agree that scientists are closed-minded in rejecting it?"

For my own part, I don't have a problem with that question either, though how I answer it depends a lot on whether (and to what extent) I think we're engaged in idea-exploration vs. tribal boundary-defending. If the former, my answer is "sure" and I wait to see what follows. If the latter, I challenge the question (not unlike your answer) or otherwise push back on the boundary violation.

Very true. I didn't get that from reading the article at first, but now I'm getting that vibe. I guess the more charitable reading is 'what would you do if you were convinced that your moral beliefs were wrong' or one of my variations, because you rightly point out that 'what would you do if your moral beliefs actually were wrong and you were aware of that' is an exceedingly presumptuous question.

Consulting our intuitions is something I believe this site shuns,

That's not true. Our relationship to intuition is just more complex.

Huh. And there you had me thinking you two had split up. So are you two in an open relationship, or what?

So are you two in an open relationship, or what?

The facebook relationship status would be "It's complicated".

Basically Kahneman did find out that intuition or System I is quite useful. Various people in decision science manage to run study indicating that heuristics are important and this community is aware of that.

CFAR speaks about integrating system I and system II.

Yeah...what are the chances that in 50 years time psychologists and neurophysicists still believe system I and II are useful heuristics to describe brain processes?

Not so bad, I think. I'd give roughly equal probability to (1) substantially the same dichotomy still being convenient, though perhaps with different names, (2) more careful investigation having refined the ideas enough to require a change in terminology (e.g., maybe it will turn out that what Kahneman calls "system 1" is better considered as two related systems, or something), and (3) the idea being largely abandoned because what's really going on turns out to be very different and it's just good/bad luck that the system 1 / system 2 dichotomy looks good in the early 21st century.

Even in case 3 I would expect there to be some parallels between system 1 / system 2 and whatever replaces it. There doesn't seem to be much doubt that our brains do some things quickly and without conscious effort and some things slowly and effortfully, or that there are ways in which the quick effortless stuff can go systematically wrong.