Is Morality a Valid Preference?

In general, the ethical theory that prevails here on Less Wrong is preference utilitarianism.

What is your evidence for this? In The Preference Utilitarian’s Time Inconsistency Problem, the top voted comments didn't try to solve the problem posed for preference utilitarians, but instead made general arguments against preference utilitarianism.

[-]jimrandomh15y90

The real answer to torture vs. dust specks is to recognize that the answer to the scenario is torture, but the scenario itself has a prior probability so astronomically low that no evidence could ever convince you that you were in it, since at most k/3^^^3 people can affect the fate of 3^^^3 people at once (where k is the number of times a person's fate is affected). However, there are higher-probability scenarios that look like torture vs. 3^^^3 dust specks, but are actually torture vs. nothing or torture vs. not-enough-specks-to-care. In philosophical pr... (read more)

6DanielLC15y

The same points in Pascal's Mugging apply. 3^^^3 has a relatively low K-complexity, which means that, if someone where to just tell you it happens, the expected number would still be astronomical. There are higher-probability things that actually apply. They're just more like torture vs. significantly less torture. The bias is still enough to keep the paradox strong.

[-]mkehrt15y60

I've been thinking about this on and off for half a year or so, and I have come to the conclusion that I cannot agree with any proposed moral system that answers "torture" to dust specks and torture. If this means my morality is scope-insensitive, then so be it.

(I don't think it is; I just don't think utilitarianism with an aggregation function of summation over all individuals is correct; I think the correct aggregation function should probably be different. I am not sure what the correct aggregation function is, but maximizing the minimum ind... (read more)

[-]rstarkov15y50

I think Torture vs Dust Specks makes a hidden assumption that the two things are comparable. It appears that people don't actually think like that; even an infinite amount of dust specks are worse than a single person being tortured or dying. People arbitrarily place some bad things into a category that's infinitely worse than another category.

So, I'd say that you aren't preferring morality; you are simply placing 50 years of torture as infinitely worse than a dust speck; no number people getting dust specks can possibly be worse than 50 years of torture.

6MBlume15y

The thing is, if you think that A and B aren't comparable, with A>B, and if you don't make some simplifying assumption like "any event with P < 0.01 is unworthy of consideration, no matter how great or awful" or something, then you don't get to ever care about B for a moment. There's always some tiny chance of A that has to completely dominate your decision-making.

1TheOtherDave15y

After reading this several times, I have to conclude that I don't understand what "comparable" means in this comment. Otherwise, I have no idea how one could thinking both that A and B aren't comparable and that A > B.

0MBlume15y

I mean "comparable" as the negation of this line of thought.

0TheOtherDave15y

Ah. So, you meant something like: if I think A is worse than B, but not infinitely worse than B, and I don't have some kind of threshold (e.g., a threshold of probability) below which I no longer evaluate expected utility of events at all, then my beliefs about B are irrelevant to my decisions because my decisions are entirely driven by my beliefs about A? I mean, that's trivially true, in the sense that a false premise justifies any conclusion, and any finite system will have some threshold for which events not to evaluate. But in a less trivial sense... hm. OK, thanks for clarifying.

0rstarkov15y

This is a good point, and I've pondered on this for a while. Following your logic: we can observe that I'm not spending all my waking time caring about A (people dying somewhere for some reason). Therefore we can conclude that the death of those people is comparable to mundane things I choose to do instead - i.e. the mundane things are not infinitely less important than someone's death. But this only holds if my decision to do the mundane things in preference to saving someone's life is rational. I'm still wondering whether I do the mundane things by rationally deciding that they are more important than my contribution to saving someone's life could be, or by simply being irrational. I am leaning towards the latter - which means that someone's death could still be infinitely worse to me than something mundane, except that this fact is not accounted for in my decision making because I am not fully rational no matter how hard I try.

4endoself15y

We defined a dust speck to have nonzero negative utility. If you don't think this describes reality, then you can substitute something else, like a stubbed toe. As long as we can make a series of things, none of which is infinitely worse than the next, we can prove that nothing in the list is infinitely worse than any other. http://lesswrong.com/lw/n3/circular_altruism/u7x presents this well.

-3[anonymous]15y

This is true, but not relevant to the question of whether 50 years of torture is infinitely worse than a speck of dust in the eye.

0FAWS15y

So is a single millisecond of torture also infinitely worse than a dust speck and as well as infinitely worse than everything else that isn't infinitely worse than a dust speck itself, or is some time span of torture infinitely worse than a slightly shorter time span? If you postulate a discontinuity that discontinuity has to be somewhere.

-1[anonymous]15y

I guess this is what I get for replying to a torture post! The point I was trying to make is mathematical: for sensible definitions of "finitely greater" the statement "if we have a sequence of objects, each of which is only finitely greater than its predecessor, then every object on the list is only finitely greater than eany earlier object" is true, but not relevant to the question of whether or not there exist infinitely large objects. My goal was to flag up mathematical reasoning that doesn't hold water, which apparently I failed to do. For completeness, I should also mention that the linked post does not make the same error.

1FAWS15y

But extremely relevant to the question whether or not there exist infinitely large objects on the list.

0[anonymous]15y

Whether or not everybody has a list is precisely the question asked in the top post of this thread.

0FAWS15y

Not at all. People who treat some things as infinitely worse than others don't do so because they believe that a list that includes both somehow stops being a list, and the threat starter never implied anything in that direction. They just have inconsistent preferences (at least in the sense of being money-pumpable). Either that or they bite the bullet and admit that there is at least one particular item infinitely worse than the preceding for any such list. Denying that a list is a list is just nonsense.

0[anonymous]15y

We are in violent agreement (but I'm coming off worse!). rstarkov suggested that people may have "utility functions" that don't take real values. Endoself's comment "showed" that this cannot be, starting from the assumption that everybody has a preference system that can be encoded as a real-valued utility function. This is nonsense. My non-disagreement with you seems to have stemmed from me not wanting to be the first person to say "order-type", and us making different assumptions about how various poster's positions projected onto our own internal models of "lists" (whatever they were).

0FAWS15y

You shouldn't have used the worlds "not relevant", that implied the statement had no important implications for the problem at all, rather than proving the (very relevant since the topic is ulilitarism) hidden assumption wrong for that set of people (unless they bit the bullet).

2Nornagest15y

It absolutely assumes that the two are comparable, and most of the smarter objections to it that I've seen invoke some kind of filtering function to zero out the impact of any particular dust speck on some level of comparison. There are a number of objections to this that you could raise in practice: given a random distribution of starting values, for example, an additional dust speck would be sufficient to push a small percentage, but an unimaginably huge quantity, of victims' subjective suffering over any threshold of significance we feel like choosing. I'm not too impressed with any of these responses -- they generally seem to leverage special pleading on some level -- but I've got to admit that they don't have anything wrong with them that the filtering argument doesn't. Welcome to Less Wrong, by the way.

3rstarkov15y

Argh, I have accidentally reported your comment instead of replying. I did wonder why it asks me if I'm sure... Sorry. It does indeed appear that the only rational approach is for them to be treated as comparable. I was merely trying to suggest a possible underlying basis for people consistently picking dust specks, regardless of the hugeness of the numbers involved.

0Alicorn15y

You did report it; I've ignored the report and now it is gone.

[-]lukeprog15y40

Really? Preference utilitarianism prevails on Less Wrong? I haven't been around too long, but I would have guessed that moral anti-realism (in several forms) prevailed.

[-]komponisto15y180

Isn't this a confusion of levels, with preference utilitarianism being an ethical theory, and moral anti-realism being a metaethical theory?

0Clippy15y

I would like to hope not.

6MinibearRex15y

Generally, I doubt that many people on less wrong believe that the universe has an "inherent" moral property, so I suppose your guess is accurate. However, there is a fairly strong emphasis on (trans)humanistic ideas. There doesn't have to be a term for "rightness" in the equations of quantum mechanics. That simply doesn't matter. Humans care. However, often humanity's moral instincts cause us to actually hurt the world more than we help it. That's why Eliezer tells us so frequently to shut up and multiply.

2Dorikka15y

I would probably go for specks here. Many people, I predict, are going to get an emotional high out of thinking that their sacrifice will prevent someone from being tortured (while scope insensitivity prevents them from realizing just how small 1/3^^^3 is). If they actually receive a dust speck in the eye in a short enough time interval afterwards, I think that they would have a significant chance of taking it to mean that specks was chosen instead of torture (and thus they would get more of a high after being specked, if they knew why.) If you polled people in such a way that they wouldn't get the high, then your answer should be the same as before. Scope insensitivity still kicks in, and people will vote without understanding how big 3^^^3 is (I don't understand it myself -- the description of 'a 1 followed by a string of zeroes as long as the Bible' is just so mind-numbingly big I know I can't comprehend it.)

0lukeprog15y

Gotcha.

-2DanielLC15y

You can (sort of) be both. I remember there being at least one person on Felicifia.org who said he figures there is no morality, so he might as well maximize happiness. I'm not a morality anti-realist, but I'm not all that sure of it, and I'm pretty sure I'd stay a Utilitarian anyway. That said, I'm emphatically not a preference utilitarian. Preferences are a map. Reality is a territory. On a fundamental physical level, they are incomparable.

1NihilCredo15y

What. I'm going to say that the first statement doesn't really seem to make much more sense than the others.

4TheOtherDave15y

I understood DanielLC's acquaintance as meaning that he prefers to maximize happiness and he believes that his only reasons for action are his preferences and (if it existed) morality, so in the absence of morality he will act solely according to his preferences, which are to maximize happiness. And, sure, if his preferences had been for entering a monastery, or maximizing unhappiness, or moonwalking in a clown suit, then in the absence of morality he would act solely according to those preferences. I doubt very much that any of these accounts actually describe a real person, and I would be very nervous around anyone they did describe, but none of this is senseless.

0DanielLC15y

I'd start that I might as well maximize my happiness, and then figure that it would make more sense to maximize happiness in general.

[-]Stuart_Armstrong15y20

Namely, do other people's moral intuitions constitute a preference that we should factor into a utilitarian calculation?

If we feel like it. I personally would say yes. What would you say?

1wedrifid15y

Yes, regardless of whether it is true. Morality is for lying about.

0Stuart_Armstrong15y

If that's you particular moral judgement...

3wedrifid15y

Possibly. However there is significant semantic ambiguity centring around 'for' and the concept of purpose. There is legitimate literal meaning there in which the claim is not moral at all (although it would be rather exaggerated.) A moral judgement that I do make unambiguously is that people should not be expected to answer loaded moral questions of that kind transparently. Most people are, fortunately, equipped with finely tuned hypocrisy instincts so that they can answer with bullshit with full sincerity. I don't expect those that have defective hypocrisy instincts to self sabotage by sharing their private, internally coherent value system. I also note that questions of that form I will reply to with obfuscation, overt insincerity or outright non-response even when I would comfortably answer yes. In this case 'yes' is a rather weak answer, given that 'factor in' does not specify degree of weighting. Yet many questions (or challenges) of the same form are far less mellow, not anyone else's business unless I choose it to be and potentially have either no answer that sounds acceptable or the appropriate response (once multiplied out) sounds evil. For example if the degree of 'factoring in other's intuitions' was specified it could be the case that the factoring in others is the 'evil' response, despite being egalitarian. Kind of like I consider CEV to be an incredibly stupid plan even though it sounds kind of like the goody goody heroic altruist response at a superficial level. But the come to think of it your question was about what should be factored in to a utilitarian calculation. So my answer would really have to be null - because utilitarian morality is an abomination that I would never be using in the first place!

0Stuart_Armstrong15y

A robust and candid position.

0MinibearRex15y

That's mostly the question I wanted to discuss here. If you want my own personal opinion, I think that it should be considered, but we shouldn't assign a massive amount of weight to it. I've studied enough psychology to learn that humans are not often reliable. I also wouldn't be very inclined to count the "moral intuitions" of militant religious groups. In this case, however, I'm more unsure.

0Stuart_Armstrong15y

I mean, as you going for a "moral realist" position, where other people might have insight into the "true" morality? Or is it that other people's moral intuition might bring up issues that you hadn't thought off, or illustrate the consequences of some of your own positions? Or is it political: it would be better to have a more consensus view (for practical or moral reasons), even if we disagree with certain aspects of it?

[-]rohern15y10

I find it impossible to engage thoughtfully with philosophical questions about morality because I remain unconvinced of the soundness of the first principles that are applied in moral judgments. I am not interested in a moral claim that does not have a basis in some fundamental idea with demonstrable validity. I will try to contain my critique to those claims that do attempt at least what I think to be this basic level of intellectual rigor.

Note 1: I recognize that I introduced many terms in the above statement that are open to challenge as loaded and... (read more)

9NihilCredo15y

I support and agree with every paragraph except the last one. I cannot come up with any sensible, useful separation between "actual scenarios" and "thought experiments". Consider the following question: "You have the ability to instantly make an arbitrarily high number of copies of a book, and distribute them to billions of people, at a negligible cost to everyone. Should you compensate the author before doing so?". To us this is an everyday matter, but to a medieval scholar it is about as much of an abstraction as "torture vs. dust specks". It is likely more abstract to him than "push the fat guy on the train tracks" is to us. While I don't believe that morality is well-founded, I do believe that the word "morality" has some meaning, even if it is incorrect - in the same way that theism isn't well-founded, but "theism" still means something. And I consider that the word "morality" must indicate a function (or a set of functions) having its domain included in the set of logically possible universes, and its codomain in some sort of algebraic structure (utilitarians should think it's a totally ordered field, but it needn't be such a powerful structure). I do not think that whether we might actually experience a given universe is a relevant criterion to morality as most people intend the word.

0rohern15y

If I take you correctly, you are pointing out that thought experiments, now abstract, can become actual through progress and chance of time, circumstance, technology, etc., and thus are useful in understanding morality. If this is an unfair assessment, correct me! I agree with you, but I also hold to my original claim, as I do not think that they contradict. I agree that the thought experiment can be a useful tool for talking about morality as a set of ideas and reactions out-of-time. However, I do not agree that the thought experiments I have read have convinced me of anything about morality in actual practice. This is for one reason alone: I am not convinced that the operation of human reason is the same in all cases, and in this particular, in the two cases of the theoretical and the physical/actual. I am not convinced that if a fat man were actually standing there waiting to be shoved piteously onto the tracks that the human mind would necessarily function in the same way it does when sitting in a cafe and discussing the fate of said to-be switch-pusher. If I were to stake the distinction between the actual and the theoretical on anything, it would be on the above point. What data have we on the reliability of these -- I think you must agree that, regardless of the hypothetical opinions of medieval scholar types, the Torture vs. Dust Specks scenario abstract for us now and here -- thought experiments to predict human behavior when, to retreat to the cliche, one is actually in the trenches? This may have some connection to the often-experienced phenomenon when in conversation of casual nonchalance and liberalism about issues that do not affect the speaker and a sudden and contradictory conservatism about issues that do affect the speaker. This is a phenomenon I encounter very often as a college student. It is gratis to be easy-going about topics that never impact oneself, but when circumstances change and a price is paid, reason does not reliably produce sim

3NihilCredo15y

I... don't think the point of such thought experiments was ever to predict what a human will do. That we do not make the same choices under pressure that we do when given reflection and distance is quite obvious. If you are interested in predicting what people will do, you should look at psychological battery tests, which (should) strive to strike a balance between realism and measurability. The point of train tracks-type experiments was to force one to demand some coherence from their "moral intuition", and to this end the fact that you're making such choices sitting in a café is a feature, not a bug, because it lets you carefully figure out logical conclusions on which (at least in theory) you will then be able to unthinkingly rely once you're in the heat of the moment (probably not an actual train track scenario, but situations like giving money to beggars or voting during jury duty where you only have seconds or hours to make a choice). When you're actually in the trenches, as you put it, your brain is going to be overwhelmed by a zillion more cognitive biases than usual, so it's very much in your interest to try and pre-make as many choices as possible while you have the luxury of double-checking every one of your assumptions and implications.

0TheOtherDave15y

One problem I have with the way such thought experiments are phrased is that they often ask "what would you do?" rather than "what's the best thing to do?", which muddles this notion of being more interested in my moral intuitions about the latter than my predictions about the former. But I realize that people and cultures vary widely in how they interpret phrases like that.

5TheOtherDave15y

Moral intuitions demonstrably exist. That is, many people demonstrably do endorse and reject certain kinds of situations in a way that we are inclined to categorize as a moral (rather than an aesthetic or arbitrary or pragmatic) judgment, and demonstrably do signal those endorsements and rejections to one another. All of that behavior has demonstrable influences on how people are born and live and die, suffer and thrive, are educated and remain ignorant, discover truths and believe falsehoods, are happy and sad, etc. I believe all of that stuff matters, so I believe how the moral intuitions that influence that stuff are formed, and how they can be influenced, is worth understanding. And, of course, the SIAI folks believe it matters because they want to engineer an artificial system whose decisions have predictable relationships to our moral intuitions even when it doesn't directly consult those intuitions. Now, maybe none of that stuff is what you're talking about... it's hard to tell precisely what you are rejecting the study of, actually, so I may be talking past you. If what you mean to reject is the study of morality as something that exists in the world outside of our minds and our behavior, for example, I agree with you. I suspect the best way to encourage the rejection of that is to study the actual roots of our moral judgments; as more and more of our judgments can be rigorously explained, there will be less and less room for a "god of the moral gaps" to explain them. And I agree with NihilCredo that the distinction between "applied morality" and "theoretical morality" is not a stable one -- especially when considering large-scale engineering projects -- so refusing to consider theoretical questions simply ensures that we're unprepared for the future. Also, thought experiments are often useful tools to clarify what our intuitions actually are.

1rohern15y

I think we may indeed be talking past each other, so I will try to state my case more cogently. I am not denying that people do possess ideas about something named "morality". It would be absurd to claim otherwise, as we are here discussing such ideas. I am denying that, even if I accept all of their assumptions, individuals who claim these ideas as more-than-subjective --- by that I think I mean that they claim their ideas able to be applied to a group rather than only to one man, the holder of the ideas --- can convince me that these ideas are not wholly subjective and individual-dependent. If it is the case that morality is individual only, then that is an interesting conclusion and something to talk about, but it does seem, at least to a first approximation, that for a judgment to be considered moral, it must have some broader applicability among individuals, rather than concerning but one person. What can Justice be if it is among one man only? This seems a critical part of what is meant by "morality". It is in this latter, broad case, that moral philosophy appears null. If you possess an idea of morality desire that I consider it to have some connection with the world and with all persons --- and surely I must require that it have such a connection, as moral claims attempt to dictate the interaction between people, and thus cannot be content to be contained in one mind alone --- at least enough of a connection that you can, through reasoned argument, convince me that your claims are both valid and sound, then surely your ideas must make reference to principles that I can discover individually to both exist and serve as predicates to your ideas. If you cannot elucidate these foundations, then how can I be brought to your view through reason? This was the intent of my original criticism, to ask why these foundations are so lousy and to beg that someone make them otherwise if moral claims are to be made. I think that this is the crux of my objection. I canno

0TheOtherDave15y

I agree with your basic point that moral intuitions reflect psychological realities, and that attempts to derive moral truths without explicitly referring to those realities will inevitably turn out to implicitly embed them. That said, I think you might be introducing unnecessary confusion by talking about "subjective" and "individual." To pick a simple and trivial objection, it might be that two people, by happenstance, share a set of moral intuitions, and those intuitions might include references to other people. For example, they might each believe "it is best to satisfy the needs of others," or "it is best to believe things believed by the majority" or "it is best to believe things confirmed by experiment." Indeed, hundreds of people might share those intuitions, either by happenstance or by mutual influence. In this case, the intuition would not be inter-subjective and non-individual, but still basically the kind of thing we're talking about. I assume you mean to contrast it with objective, global things like, say, gravity. Which is fine, but it gets tricky to say that precisely. Here, again, things get slippery. First, I can have moral intuitions about non-humans... for example, I can believe that it's wrong to club cute widdle baby seals. Second, it's not obvious that non-humans can't have moral intuitions. If that is in fact your desire, then you haven't a care for it. Or, indeed, for much of anything else. Speaking personally, though, I would be loathe to give up my love of pie, despite acknowledging that it is a consequence of my own biology and history. Agreed that imprinting an AI with human notions of moral judgments, especially doing so with the same loose binding to actual behavior humans demonstrate, would be relatively foolish. This is, of course, different from building an AI that is constrained to behave consistently with human moral intuitions. Agreed that such an AI would easily conclude that humans are not bound by the same constraints t

0rohern15y

I think we might still be talking past each other, but here goes: The reason I posit and emphasize a distinction between subjective judgments and those that are otherwise -- I have a weak reason for not using the term "objective" here -- is to highlight a particular feature of moral claims that is lacking, and in thus being lacked, weakens them. That is, I take a claim to be subjective if to hold it myself I must come upon it by chance. I cannot be brought to it through reason alone. It is an opinion or intuition that I cannot trace logically in my own thought, so I cannot communicate it to you by guiding you down the same line. The reason I think that this distinction matters, is that without this logical structure, it not possible for someone to bring me to experience the same intuition through reasoned argument or demonstration. Without this feature, morality must be an island state. This is ruinous, because morality inevitably and necessarily touches upon interactions between people. If it cannot do this, it cannot do much. Perhaps we should come to common agreement, are at least agreed-upon disagreement on this point before we try other things. Other Things: I suspect -- this is an idea I have only recently invented have not entirely examined -- that any idea that is irrational needs must be essentially incommunicable. How could it be otherwise? If you can lay out the logic behind a thought and give support to its predicates carefully and patiently, and of course your logic is valid and your predicates sound, how can I not, if I am open to reason, not accept what you say as true? That is, if you can demonstrate your ideas as the logical consequences of some set of known truths, I must, because that is what logical consequence is, accept your ideas as true. I have not witnessed with done with moral notions. Hence my doubt about there existence as rational ideas. I do not doubt that people have moral ideas, but I doubt that they can be communicated to people

0TheOtherDave15y

I am fairly sure that we aren't talking past each other, I just disagree with you on some points. Just to try and clarify those points... * You seem to believe that a moral theory must, first and foremost, be compelling... if moral theory X does not convince others, then it can't do much worth doing. I am not convinced of this. For example, working out my own moral theory in detail allows me to recognize situations that present moral choices, and identify the moral choices I endorse, more accurately... which lowers my chances of doing things that, if I understood better, I would reject. This seems worth doing, even if I'm the only person who ever subscribes to that theory. * You seem to believe that if moral theory X is not rationally compelling, then we cannot come to agree on the specific claims of X except by chance. I'm unconvinced of that. People come to agree on all kinds of things where there is a payoff to agreement, even where the choices themselves are arbitrary. Heck, people often agree on things that are demonstrably false. * Relatedly, you seem to believe that if X logically entails Y, then everyone in the world who endorses X necessarily endorses Y. I'd love to live in that world, but I see no evidence that I do. (That said, it's possible that you are actually making a moral claim that having logically consistent beliefs is good, rather than a claim that people actually do have such beliefs. I'm inclined to agree with the former.) * I can have a moral intuition that bears clubbing baby seals is wrong, also. Now, I grant you that I, as a human, am less likely to have moral intuitions about things that don't affect humans in any way... but my moral intuitions might nevertheless be expressible as a general principle which turns out to apply to non-humans as well. * You seem to believe that things I'm biologically predisposed to desire, I will necessarily desire. But lots of biological predispositions are influenced by local environment. My desir

0rohern15y

I think an important part of our disagreement, at least for me, is that you are interested in people generally and morality as it is now --- at least your examples come from this set --- while I am trying to restrict my inquiry to the most rational type of person, so that I can discover a morality that all rational people can be brought to through reason alone without need for error or chance. If such a morality does not exist among people generally, then I have no interest for the morality of people generally. To bring it up is a non sequitur in such a case. I do not see that people coming to agree on things that are demonstrably false is a point against me. This fact is precisely why I am turned-off by the current state of ethical thought, as it seems infested with examples of this circumstance. I am not impressed by people who will agree to an intellectual point because it is convenient. I take truth first, at least that is the point of this inquiry. I am asking a single question: Is there (or can we build) a morality that can be derived with logic from first principles that are obvious to everyone and require no Faith?

1TheOtherDave15y

You're right, I'm concerned with morality as it applies to people generally. If you are exclusively concerned with sufficiently rational people, then we have indeed been talking past each other. Thanks for clarifying that. As to your question: I submit that for that community, there are only two principles that matter: 1. Come to agreement with the rest of the community about how to best optimize your shared environment to satisfy your collective preferences. 2. Abide by that agreement as long as doing so is in the long-term best interests of everyone you care about. ...and the justification for those principles is fairly self-evident. Perhaps that isn't a morality, but if it isn't I'm not sure what use that community would have for a morality in the first place. So I say: either of course there is, or there's no reason to care. The specifics of that agreement will, of course, depend on the particular interests of the people involved, and will therefore change regularly. There's no way to build that without actually knowing about the specific community at a specific point in time. But that's just implementation. It's like the difference between believing it's right to not let someone die, and actually having the medical knowledge to save them. That said, if this community is restricted to people who, as you implied earlier, care only for rationality, then the resulting agreement process is pretty simple. (If they invite people who also care for other things, it will get more complex.)

0rohern15y

Very well put.

0Prolorn15y

Perhaps you've already encountered this, but your question calls to mind the following piece by Yudkowsky: No Universally Compelling Arguments, which is near the start of his broader metaethics sequence. I think it's one of Yudkowsky's better articles. (On a tangential note, I'm amused to find on re-reading it that I had almost the exact same reaction to The Golden Transcendence, though I had no conscious recollection of the connection when I got around to reading it myself.)

0orthonormal15y

I agree vehemently with your comment.

1Perplexed15y

Near zero? The number of people who have given careful and analytic thought to the foundational principles of ethics easily numbers in the thousands. Comparable to the number of people who have given equivalent foundational consideration to logic or thermodynamics or statistics. Given that there are seven billion people in the world, it is understandable that you have not yet had a conversation with one of these people. But it is easy enough to find the things they have written - online or in libraries. Give it a look. I can't promise it will be convincing, or even that it will improve your opinion of the field. But you will find that a lot of serious thought has been given to the subject.

1rohern15y

Forgive me for being sloppy with my language. Given what I wrote, your objection is entirely reasonable. The idea that I meant to express is that, while it seems safe to assume that virtually everyone who has ever lived long enough to become a thinking person has encountered some kind of moral question in his life, we cannot say that an appreciable percentage of these people has sat and carefully analyzed these questions. Even if we restrict ourselves only to people alive today and living in the United States -- an enormous restriction considering the perhaps 100 billion people who have lived ever -- the population of thousands you point to is pathetically small. Certainly I agree that Socrates, Mill, Kant, & their Merry Band have approached the subject seriously, but beyond these we've but a paucity, which I think is truly surprising given the apparent universality of moral experience. The comparison to studying thermodynamics or logic (perhaps not quite so with logic), is that while we can say that everyone is of course affected by thermodynamics, almost no one attempts to think about it. The effect of thermodynamics on a person's life is not impacted by that person's ignorance of thermodynamic laws. However, a huge number of people do attempt to think and talk about morality, and I am not convinced that this latter group does so rigorously or well, which does have a real and critical effect on what morality is in actual practice. However, I hope you will notice that this is a minor point, and was not a premise to the larger objection I was putting forward.

[-]wedrifid15y00

Is Morality a Valid Preference?

Yes. It's just unpleasant to be around in large doses.

[-]prase15y00

I would predict, based on human nature, that a if the 3^^^3 people were asked if they wanted to inflict a dust speck in each one of their eyes, in exchange for not torturing another individual for 50 years, they would probably vote for dust specks.

Each one with probability of order 1/3^^^3? Well that's what I call overconfidence.

1MinibearRex15y

Note the word "vote". I would expect the number in favor of dust specks to exceed (3^^^3)/2.

0prase15y

Sorry for misunderstanding, then. But what's the point of the big number, if the claim is only about proportion?

1Pavitra15y

There's a certain intuition that one should assign greater weight to an other-moral-belief if many other people believe it.

0prase15y

Then this would a clear abuse of that intuition. Among 3^^^3 people with varied beliefs, all moral beliefs that exist today on Earth would be believed by many people. Actually I think that the intuition applies only if "many" is measured relatively to the size of population.

0Pavitra15y

I think the population-ratio measure sounds about right. Phrased in those terms, the original idea was that as the number of people unanimously agreeing with you increases, the proportion of total belief-weight represented by your own opinion approaches zero.

0prase15y

What is belief weight? Does your assertion mean that with 3^^^3 people, any person's own opinion has approximately zero value?

0Pavitra15y

You said in the 3-parent that "many" should be measured relative to the size of the population. I interpret that to mean that beliefs should be weighted by the number of people who believe them, in the sense of a weighted average (although we're computing something different from the average, the concept of "weighting" analogizes over). The weight of a belief is then the number of people who believe it divided by the number of people polled. Yes.

[-]Oligopsony15y00

I think the answer is that morality has to be counted, but we also have to count changes to morality. If moral preferences were entirely a matter of intellectual commitment, this might lead to double counting, but in fact people really do experience pride, guilt, and so on - and I doubt that morality could have any effect on their behavior if it didn't.

Counting the changes to morality can cut both ways. For instance: some people have a strong inclination to have sex with people of the same sex, while many people (sometimes the same ones) are deeply morally... (read more)

5TheOtherDave15y

Mm? If I have a strong inclination to do X, and a strong moral intuition that "X is wrong", and I suffer anguish because of the conflict, how do you conclude that the best result is that the moral intuition disappears... either in that particular case, or in general? I mean, I happen to agree with you about the particular case you mention, but I don't see how you reached it. It's worth noting, incidentally, that in addition to "eliminate the intuition" and "eliminate the inclination" there is the option of "eliminate the anguish." That is, I might reach a point where I want to do X, and I think X is wrong, and I experience conflict between those impulses, and I work out some optimal balance between those conflicting impulses, and I am not anguished by this in any way.

0Oligopsony15y

Good question; here's a formal answer: We can, in the long run, keep the inclination and eliminate the intuition, keep the intuition and drop the inclination, or keep both. The utility of each is: side with inclination: fun of (less pain of) indulging in inclination (what's intrinsic to the action itself, i.e. the pain of being tortured but not the discomfort we have that it's being inflicted), less effort needed for (and plus any second order effects of) eliminating the intuition side with the intuition: zero less effort needed for (and plus any second order effects of) eliminating the inclination status quo: fun of less pain of indulging in the inclination at the rates people will engage in given moral disapproval, less pain felt by those with the intuition I think that this adequately explains 1) why the OP felt ambivalent about counting moral intuitions - even if they count in principle, you really should ignore them sometimes - and 2) why your and my intuitions agree in the homosexuality case: gay sex is fun and doesn't intrinsically harm anyone, homophobia brings little joy to even its holders, and it would take much less effort to eliminate homophobia than homosexuality. This model grounds common moral intuitions like "if it doesn't harm anybody [implied: beyond the feelings of those who morally disapprove], why ban it?," that preferences that are inborn should take precedence over those that are inculcated, and so on. And it seems that even the other side in this debate is operating from this framework: hence their propensity to argue that homosexuality is not inborn, or that homophobia is, or that there would be very bad second-order effects of the disappearance of homophobia (you would marry your box turtle!) I don't really think this can work over the long term. An individual might be mistaken about how much moral anguish she'd experience over something, but in the long run, those moral intuitions that don't affect you emotionally aren't going to aff

0TheOtherDave15y

Ah, I see what you mean. So if the world changed such that eliminating the inclination cost less than eliminating the intuition -- say, we discover a cheap-to-produce pill that makes everybody who takes it heterosexual without any other side-effects -- a good utilitarian would, by the same token, conclude that the best long-term equilibrium is one in which the inclination disappeared. Yes?

1Oligopsony15y

In principle, I suppose so (though if you're in a relationship, a pill to change your orientation is hardly low-cost!)

0TheOtherDave15y

Agreed, though that simply extends the definition of "long-term equilibrium" a few generations. Anyway, cool; I'd misunderstood your original claim to be somewhat more sweeping than what you actually meant, which is why I was uncertain. Thanks for clarifying!

[-]David_Gerard15y00

I would predict, based on human nature, that a if the 3^^^3 people were asked if they wanted to inflict a dust speck in each one of their eyes, in exchange for not torturing another individual for 50 years, they would probably vote for dust specks.

I think you've nailed my problem with this scenario: anyone who wouldn't go for this, I would be disinclined to listen to.

1rohern15y

Perhaps this is just silliness, but I am curious how you would feel if the question were: "You have a choice: Either one person gets to experience pure, absolute joy for 50 years, or 3^^^3 people get to experience a moment of pleasure on the level experienced when eating a popsicle." Do you choose popsicle?

2David_Gerard15y

I suspect I would. But not only does utility not add linearly, you can't just flip the sign, because positive and negative are calculated by different systems.

0TheOtherDave15y

I don't think it's silly at all. Personally, I experience more or less the same internal struggle with this question as with the other: I endorse the idea that what matters is total utility, but my moral intuitions aren't entirely aligned with that idea, so I keep wanting to choose the individual benefit (joy or non-torture) despite being unable to justify choosing it. Also, as David Gerard says, it's a different function... that is, you can't derive an answer to one question from an answer to the other... but the numbers we're tossing around are so huge that the difference hardly matters.

LESSWRONG
LW

LESSWRONG
LW

19

Is Morality a Valid Preference?

19

19