Are Deontological Moral Judgments Rationalizations?

In 2007, Chris Matthews of Hardball interviewed David O'steen, executive director of a pro-life organization. Matthews asked:

I have always wondered something about the pro-life movement. If you believe that killing [a fetus] is murder, why don't you bring murder charges or seek a murder penalty against a woman who has an abortion? Why do you let her off, if you really believe it's murder?1

O'steen replied that "we have never sought criminal penalties against a woman," which isn't an answer but a re-statement of the reason for the question. When pressed, he added that we don't know "how she‘s been forced into this." When pressed again, O'steen abandoned these responses and tried to give a consequentialist answer. He claimed that implementing "civil penalties" and taking away the "financial incentives" of abortion doctors would more successfully "protect unborn children."

But this still doesn't answer the question. If you believe that killing a fetus is murder, then a woman seeking an abortion pays a doctor to commit murder. Why don't abortionists want to change the laws so that abortion is considered murder and a woman who has an abortion can be charged with paying a doctor to commit murder? Psychologist Robert Kurzban cites this as a classic case of moral rationalization.2

Pro-life demonstrators in Illinois were asked a similar question: "If [abortion] was illegal, should there be a penalty for the women who get abortions illegally?" None of them (on the video) thought that women who had illegal abortions should be punished as murders, an ample demonstration of moral rationalization. And I'm sure we can all think of examples where it looks like someone has settled on an intuitive moral judgment and then invented rationalizations later.3

More controversially, some have suggested that rule-based deontological moral judgments generally tend to be rationalizations. Perhaps we can even dissolve the debate between deontological intuitions and utilitarian intuitions if we can map the cognitive algorithms that produce them.

Long-time deontologists and utilitarians may already be up in arms to fight another war between Blues and Greens, but these are empirical questions. What do the scientific studies suggest?

 

Utilitarian and Deontological Processes

A runaway trolley is about to run over and kill five people, but you can save them by hitting a switch that will put the trolley on a side track where it will only kill one person. Do you throw the switch? When confronted with this switch dilemma, most people say it is morally good to divert the trolley,4 thereby achieving the utilitarian 'greater good'.

Now, consider the footbridge dilemma. Again, a runaway trolley threatens five people, and the only way to save them is to push a large person off a footbridge onto the tracks, which will stop the trolley but kill the person you push. (Your body is too small to stop the trolley.) Do you push the large person off the bridge? Here, most people say it's wrong to trade one life for five, allowing a deontological commitment to individual rights to trump utilitarian considerations of the greater good.

Researchers presented subjects with a variety of 'impersonal' dilemmas (including the switch dilemma) and 'up-close-and-personal' dilemmas (including the footbridge dilemma). Personal dilemmas preferentially engaged brain areas associated with emotion. Impersonal dilemmas preferentially engaged the regions of the brain associated with working memory and cognitive control.5

This suggested a dual-process theory of moral judgment, according to which the footbridge dilemma elicits a conflict between emotional intuition ("you must not push people off bridges!") and utilitarian calculation ("pushing the person off the bridge will result in the fewest deaths"). In the footbridge case, emotional intuition wins out in most people.

But now, consider the crying baby dilemma from the final episode of M.A.S.H:

It's wartime. You and your fellow villagers are hiding from nearby enemy soldiers in a basement. Your baby starts to cry, and you cover your baby's mouth to block the sound. If you remove your hand, your baby will cry loudly, and the soldiers will hear. They will find you... and they will kill all of your. If you do not remove your hand, your baby will smother to death. Is it morally acceptable to smother your baby to death in order to save yourself and the other villagers?6

Here, people take a long time to answer, and they show no consensus in their answers. If the dual-process theory of moral judgment is correct, then people considering the crying baby dilemma should exhibit increased activity in the ACC (a region associated with response conflict), and in regions associated with cognitive control (for overriding a potent emotional response with utilitarian calculation). Also, those who eventually choose the characteristically utilitarian answer (save the most lives) over the characteristically deontological answer (don't kill the baby) should exhibit comparatively more activity in brain regions associated with working memory and cognitive control. All three predictions turn out to be true.7

Moreover, patients with two different kinds of dementia or lesions that cause "emotional blunting" are disproportionately likely to approve of utilitarian action in the footbridge dilemma,8 and cognitive load manipulations that keep working memory occupied slow down utilitarian judgments but not deontological judgments.9

Studies of individual differences also seem to support the dual-process theory. Individuals who are (1) high in "need for cognition" and low in "faith in intuition", or (2) score well on the Cognitive Reflection Test, or (3) have unusually high working memory capacity... all give more utilitarian judgments.10

This leads us to Joshua Greene's bold claim:

...deontological judgments tend to be driven by emotional responses, and... deontological philosophy, rather than being grounded in moral reasoning, is to a large extent an exercise in moral rationalization. This is in contrast to consequentialism, which, I will argue, arises from rather different psychological processes, ones that are more 'cognitive,' and more likely to involve genuine moral reasoning...

[Psychologically,] deontological moral philosophy really is... an attempt to produce rational justifications for emotionally driven moral judgments, and not an attempt to reach moral conclusions on the basis of moral reasoning.11

 

Cognition and Emotion

Greene explains the difference between 'cognitive' and 'emotional' processes in the brain (though both involve information processing, and so are 'cognitive' in a broader sense):

...'cognitive' processes are especially important for reasoning, planning, manipulating information in working memory, controlling impulses, and 'higher executive functions' more generally. Moreover, these functions tend to be associated with certain parts of the brain, primarily the dorsolateral surfaces of the prefrontal cortex and parietal lobes... Emotion, in contrast, tends to be associated with other parts of the brain, such as the amygdala and the medial surfaces of the frontal and parietal lobes... And while the term 'emotion' can refer to stable states such as moods, here we will primarily be concerned with emotions subserved by processes that in addition to being valenced, are quick and automatic, though not necessarily conscious.

Since we are concerned with two kinds of moral judgment (deontological and consequentialist) and two kinds of neurological process (cognitive and emotional), we have four empirical possibilities:

First, it could be that both kinds of moral judgment are generally 'cognitive', as Kohlberg’s theories suggest (Kohlberg, 1971). At the other extreme, it could be that both kinds of moral judgment are primarily emotional, as Haidt’s view suggests (Haidt, 2001). Then there is the historical stereotype, according to which consequentialism is more emotional (emerging from the 'sentimentalist' tradition of David Hume (1740) and Adam Smith (1759) while deontology is more 'cognitive' [including the Kantian 'rationalist' tradition: see Kant (1785)]. Finally, there is the view for which I will argue, that deontology is more emotionally driven while consequentialism is more 'cognitive.'

We have already seen the neuroscientific evidence in favor of Greene's view. Now, let us turn to further evidence from the work of Jon Haidt.

 

Emotion and Deontological Judgments

Haidt & colleagues (1993) presented subjects with a sequence of harmless actions, for example:

  1. A son promises his dying mother that he will visit her grave every day after she has died, but then doesn’t because he is busy.
  2. A woman uses an old American flag to clean the bathroom.
  3. A family eats its dog after it has been killed accidentally by a car.
  4. A brother and sister kiss on the lips.
  5. A man masturbates using a dead chicken before cooking and eating it.

For each action, subjects were asked questions like: Is this action wrong? Why? Does it hurt anyone? If someone did this, would it bother you? Greene summarizes the results:

When people say that such actions are wrong, why do they say so? One hypothesis is that these actions are perceived as harmful, whether or not they really are... Kissing siblings could cause themselves psychological damage. Masturbating with a chicken could spread disease, etc. If this hypothesis is correct, then we would expect people’s answers to the question "Does this action hurt anyone?" to correlate with their degree of moral condemnation... Alternatively, if emotions drive moral condemnation in these cases, then we would expect people’s answers to the question "If you saw this, would it bother you?" to better predict their answers to the moral questions posed.

If you're following along, it may not surprise you that emotions seemed to be driving the deontological condemnation of harmless actions. Moreover, both education and adulthood were correlated with more consequentialist judgments. (Cognitive control of basic emotional reactions is something that develops during adolescence.12) Greene reminds us:

These... findings make sense in light of the model of moral judgment we have been developing, according to which intuitive emotional responses drive prepotent moral intuitions while 'cognitive' control processes sometimes rein them in.

But there is more direct evidence of the link between emotion and the deontological condemnation of harmless actions.

Wheatley & Haidt (2005) gathered hypnotizable subjects and gave some of them a hypnotic suggestion to feel disgust upon reading the word 'often', while giving others a hypnotic suggestion to feel disgust upon reading the word 'take'. The researchers then showed these subjects a variety of scenarios, some of them involving no harm. (For example, two second cousins have a relationship in which they "take weekend trips to romantic hotels" or else "often go on weekend trips to romantic hotels".) As expected, subjects who received the wordings they had been primed to feel disgust toward judged the couple's actions as more morally condemnable than other subjects did.

In a second experiment, Wheatley and Haidt used the same technique and had subjects respond to a scenario in which a person did nothing remotely wrong: a student "often picks" or "tries to take up" broad topics of discussion at meetings. Still, many subjects who were given the matching hypnotic suggestion rated the student's actions as morally wrong. When asked why, they invented rationalizations like "It just seems like he’s up to something" or "It just seems so weird and disgusting" or "I don’t know [why it’s wrong], it just is."

In other studies, researchers implemented a disgust condition by placing some subjects at a dirty desk or in the presence of fart spray. As before, those in the disgust condition were more likely to rate harmless actions as morally wrong than other subjects were.13

Finally, consider that the dual-process theory of moral judgment predicts that deontological judgments will be quicker than utilitarian ones, because deontological judgments use emotional and largely unconscious brain modules while utilitarian judgments require slow, conscious calculation. Suter & Hertwig (2011) presented subjects with a variety of moral dilemmas and alternatively prodded them to give their judgments quickly or take their time to deliberate thoroughly. As predicted, faster responses predicted more deontological judgments.

 

Summing Up

We are a species prone to emotional moral judgment, and to rationalization ('confabulation'). And, Greene writes,

What should we expect from creatures who exhibit social and moral behavior that is driven largely by intuitive emotional responses and who are prone to rationalization of their behaviors? The answer, I believe, is deontological moral philosophy...

Whether or not we can ultimately justify pushing the man off the footbridge, it will always feel wrong. And what better way to express that feeling of non-negotiable absolute wrongness than via the most central of deontological concepts, the concept of a right: You can’t push him to his death because that would be a violation of his rights.

Deontology, then, is a kind of moral confabulation. We have strong feelings that tell us in clear and uncertain terms that some things simply cannot be done and that other things simply must be done. But it is not obvious how to make sense of these feelings, and so we, with the help of some especially creative philosophers, make up a rationally appealing story: There are these things called 'rights' which people have, and when someone has a right you can’t do anything that would take it away. It doesn’t matter if the guy on the footbridge is toward the end of his natural life, or if there are seven people on the tracks below instead of five. If the man has a right, then the man has a right. As John Rawls... famously said, "Each person possesses an inviolability founded on justice that even the welfare of society as a whole cannot override"... These are applause lines because they make emotional sense.

Of course, utilitarian moral judgment is not emotionless. Emotion is probably what leads us to label harm as a 'bad' thing, for example. But utilitarian moral judgment is, as we've seen, particularly demanding of 'cognitive' processes: calculation, the weighing of competing concerns, the adding and averaging of value, and so on. Utilitarian moral judgment uses the same meso-limbic regions that track a stimulus' reward magnitude, reward probability, and expected value.14

This does not prove the case that deontological moral judgments are usually rationalizations. But many lines of converging evidence make this a decent hypothesis. And now we can draw our neural map:15

And up until March 18th of this year, Greene had a pretty compelling case for his position that deontological judgments are generally just rationalizations.

And then, Guy Kahane et al. (2011) threw Greene's theory into doubt by testing separately for the content (deontological vs. utilitarian) and the intuitiveness (intuitive vs. not-intuitive) of moral judgments. The authors summarize their results:

Previous neuroimaging studies reported that utilitarian judgments in dilemmas involving extreme harm were associated with activation in the DLPFC and parietal lobe (Greene et al., 2004). This finding has been taken as evidence that utilitarian judgment is generally driven by controlled processing (Greene, 2008). The behavioural and neural data we obtained suggest instead that differences between utilitarian and deontological judgments in dilemmas involving extreme harm largely reflect differences in intuitiveness rather than in content.

...When we controlled for content, these analyses showed considerable overlap for intuitiveness. In contrast, when we controlled for intuitiveness, only littleif anyoverlap was found for content. Our results thus speak against the influential interpretation of previous neuroimaging studies as supporting a general association between deontological judgment and automatic processing, and between utilitarian judgment and controlled processing.

[This evidence suggests...] that behavioural and neural differences in responses to such dilemmas are largely due to differences in intuitiveness, not to general differences between utilitarian and deontological judgment.

So we'll have to wait for more studies to unravel the mystery of whether deontological moral judgments are generally rationalizations.

By email, Greene told me he suspected Kahane's 'alternative theory' wasn't much of an alternative to what he (Greene) was proposing in the first place. In his paper, Greene discussed the passage where Kant says it's wrong to lie to prevent a madman from killing someone, and cites this as an example of a case in which a deontological judgment might be more controlled, while the utilitarian judgment is more automatic. Greene's central claim is that when there's a conflict between rights and duties on the one hand, and promoting the greater good on the other, it's typically controlled cognition on the utilitarian side and emotional intuition on the other. 

Update: Greene's full reply to Kahane et al. is now available.

But even if Greene's theory is right, humans may still need to use deontological rules because we run on corrupted hardware.

 

 

Notes

1 Hardball for November 13, 2007. Here is the transcript.

2 Kurzban (2011), p. 193.

3 Also see Jon Haidt's unpublished manuscript on moral dumfounding, and Hirstein (2005).

4 Petrinovich et al. (1993); Petrinovich & O’Neill (1996).

5 Greene et al. (2001, 2004).

6 Greene (2009).

7 Greene et al. (2004).

8 Mendez et al. (2005); Koenigs et al. (2007); Ciaramelli et al. (2007).

9 Greene et al. (2008).

10 Bartels (2008); Hardman (2008); Moore et al. (2008).

11 The rest of the Joshua Greene quotes from this article are from Greene (2007).

12 Anderson et al. (2001); Paus et al. (1999); Steinburg & Scott (2003).

13 Schnall et al. (2004); Baron & Thomley (1994).

14 See Cushman et al. (2010).

15 From Greene (2009).

 

References

Anderson, Anderson, Northam, Jacobs, & Catroppa (2001). Development of executive functions through late childhood and adolescence in an Australian sample. Developmental Neuropsychology, 20: 385-406.

Baron & Thomley (1994). A Whiff of Reality: Positive Affect as a Potential Mediator of the Effects of Pleasant Fragrances on Task Performance and Helping. Environment and Behavior, 26: 766-784.

Bartels (2008). Principled moral sentiment and the flexibility of moral judgment and decision making. Cognition, 108: 381-417.

Ciaramelli, Muccioli, Ladavas, & di Pellegrino (2007). Selective deficit in personal moral judgment following damage to ventromedial prefrontal cortex. Social Cognitive and Affective Neuroscience, 2: 84-92.

Cushman, Young, & Greene (2010). Multi-system moral psychology. In Doris (ed.), The Moral Psychology Handbook (pp. 47-71). Oxford University Press.

Greene, Sommerville, Nystrom Darley, & Cohen (2001). An fMRI investigation of emotional engagement in moral judgment. Science, 293: 2105-2108.

Greene, Nystrom, Engell, Darley, & Cohen (2004). The neural bases of cognitive conflict and control in moral judgment. Neuron, 44: 389-400.

Greene (2007). The secret joke of Kant's soul. In Sinnott-Armstrong (ed.), Moral Psychology Vol. 3: The Neuroscience of Morality (pp. 35-79). MIT Press.

Greene, Morelli, Lowenberg, Nystrom, & Cohen (2008). Cognitive load selectively interferes with utilitarian moral judgment. Cognition, 107: 1144-1154.

Greene (2009). The cognitive neuroscience of moral judgment. In Gazzaniga (ed.), The Cognitive Neurosciences, Fourth Edition (pp. 987–999). MIT Press.

Haidt, Koller, & Dias (1993). Affect, culture, and morality, or is it wrong to eat your dog? Journal of Personality and Social Psychology 65: 613-628.

Hardman (2008). Moral dilemmas: Who makes utilitarian choices. In Hare (ed.), Hare Psychopathy Checklist--Revised (PCL-R): 2nd Edition. Multi-Health Systems, Inc.

Haidt (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108: 814-834.

Hirstein (2005). Brain Fiction: Self-Deception and the Riddle of Confabulation. MIT Press.

Hume (1740). A Treatise of Human Nature.

Kahane, Wiech, Shackel, Farias, Savulescu, & Tracey (2011). The neural basis of intuitive and counterintuitive moral judgmentSocial Cognitive & Affective Neuroscience.

Kant (1785). Groundwork of the Metaphysics of Morals.

Koenigs, Young, Cushman, Adolphs, Tranel, Damasio, & Hauser (2007). Damage to the prefrontal cortex increases utilitarian moral judgements. Nature, 446: 908–911.

Kohlberg (1971). From is to ought: How to commit the naturalistic fallacy and get away with it in the study of moral development. In Mischel (ed.), Cognitive development and epistemology (pp. 151–235). Academic Press.

Kurzban (2011). Why Everyone (Else) Is a Hypocrite: Evolution and the Modular Mind. Princeton University Press.

Mendez, Anderson, & Shapira (2005). An investigation of moral judgment in fronto-temporal dementia. Cognitive and Behavioral Neurology, 18: 193–197.

Moore, Clark, & Kane (2008). Who shalt not kill?: Individual differences in working memory capacity, executive control, and moral judgment. Psychological Science, 19: 549-557.

Paus, Zijdenbos, Worsley, Collins, Blumenthal, Giedd, Rapoport, & Evans (1999). Structural maturation of neural pathways in children and adolescents: In vivo study. Science, 283: 1908-1911.

Petrinovich, O'Neill, Jorgensen (1993). An empirical study of moral intuitions: Toward an evolutionary ethics. Journal of Personality and Social Psychology, 64: 467-478.

Petrinovich & O’Neill (1996). Influence of wording and framing effects on moral intuitions. Ethology and Sociobiology, 17: 145-171.

Schnall, Haidt, & Clore (2004). Irrelevant disgust makes moral judgment more severe, for those who listen to their bodies. Unpublished manuscript.

Smith (1759). The Theory of Moral Sentiments

Steinburg & Scott (2003). Less guilty by reason of adolescence: Developmental immaturity, diminished responsibility, and the juvenile death penalty. American Psychologist, 58: 1009-1018.

Suter & Hertwig (2011). Time and moral judgment. Cognition, 119: 454-458.

Valdesolo & DeSteno (2006). Manipulations of emotional context shape moral judgment. Psychological Science, 17: 476-477.

Wheatley & Haidt (2005). Hypnotically induced disgust makes moral judgments more severe. Psychological Science, 16: 780-784.

168 comments, sorted by
magical algorithm
Highlighting new comments since Today at 4:56 AM
Select new highlight date
Moderation Guidelinesexpand_more

I used to think I was a very firm deontologist, but that was mainly because I didn't want ethical rules to be bent willy-nilly to maximize something simple like "number of lives saved." I didn't, for example, want torture to be legal. I wanted to live in a world with "rights" -- that is, ethical rules that ought not to be broken even when the circumstances change, for all possible circumstances with non-negligible probability. You don't want to live in a world where people are constantly reconsidering "Hm, is it worth it at this moment to not steal Sarah's property?" You want to live in a world where people understand that stealing is wrong and that's that. You want some rigidity.

I think a lot of self-identified deontologists think along these lines. They associate utilitarianism with "the greatest good for the greatest number," and then imagine things like "it is for the good of this great Nation that you be drafted to dig ditches this year" and they shudder.

That shudder isn't necessarily a "confabulation." The reason you shudder at the thought of a moral rule to "maximize utility" is that there is no definition of utility or "human value," simple enough to state in one sentence, that wouldn't result in a hell-world if you systematically maximized it. Human value is complicated, as this site has been at pains to tell us. Pick something (like "number of lives saved") and optimize for that, and you won't like the results.

People come up with deontological constraints, I think, to deal with the fact that "maximizing utility," when you visualize it, looks very, very bad. Modeling utilitarianism to low precision looks bad. Adding more subtlety to the model might not be so bad. Adding in terms like sympathy, respect for life, and so on as positive goods, so that throwing someone off a trolley is not a clear win. Or you could model human value by appealing to rights. Either way you haven't really put your finger on what you mean by "moral." If we could define morality rigorously, life would be easy, and it isn't.

This sounds like two-tier consequentialism -- "as it happens, when you take second- and third- and fourth- order consequences into account, the utility-maximizing course looks a hell of a lot like respecting some set of inherent rights of individuals"

I've sometimes thought of deontological rules as something like a sanity check on utilitarian reasoning.

If, as you are reasoning your way to maximum utility, you come up with a result that ends, "... therefore, I should kill a lot of innocent people," or for that matter "... therefore, I'm justified in scamming people out of their life savings to get the resources I need," the role of deontological rules against murder or cheating is to make you at least stop and think about it really hard. And, almost certainly, find a hole in your reasoning.

It is imaginable — I wouldn't say likely — that there are "universal moral laws" for human beings, which take the following form: "If you come to the conclusion 'Utility is maximized if I murder these innocent people', then it is more likely that your human brain has glitched and failed to reason correctly, than that your conclusion is correct." In other words, the probability of a positive-utility outcome from murder is less than the probability of erroneous reasoning leading to the belief in that outcome.

A consequence of this is that the better predictor you are, the more things can be moral for you to do if you conclude they maximize utility. It is imaginable that no human can with <50% probability of error arrive at the conclusion "I should push that fat guy in front of the trolley", but that some superhuman predictor could.

It is imaginable — I wouldn't say likely — that there are "universal moral laws" for human beings, which take the following form: "If you come to the conclusion 'Utility is maximized if I murder these innocent people', then it is more likely that your human brain has glitched and failed to reason correctly, than that your conclusion is correct." In other words, the probability of a positive-utility outcome from murder is less than the probability of erroneous reasoning leading to the belief in that outcome.

Obligatory link to relevant sequence.

They associate utilitarianism with "the greatest good for the greatest number," and then imagine things like "it is for the good of this great Nation that you be drafted to dig ditches this year" and they shudder.

That shudder isn't necessarily a "confabulation."

I don't think Luke or Greene is saying that the shudder is confabulation. The shudder is the intuitive emotional response. What they're calling "confabulation" is making up a deontological rule, such as "everyone has a right not to be drafted for anything except defense", or something like that, to explain/justify the shudder.

What they're calling "confabulation" is making up a deontological rule, such as "everyone has a right not to be drafted for anything except defense", or something like that, to explain/justify the shudder.

If you don't make a deontological rule and insist that it have no exceptions, in any particular case you will be tempted to find an excuse why it doesn't apply. As Eliezer said in his post The Ends Don't Justify the Means:

And so we have the bizarre-seeming rule: "For the good of the tribe, do not cheat to seize power even when it would provide a net benefit to the tribe."

Indeed it may be wiser to phrase it this way: If you just say, "when it seems like it would provide a net benefit to the tribe", then you get people who say, "But it doesn't just seem that way - it would provide a net benefit to the tribe if I were in charge."

Very good reply here. I used to firmly identify as a deontologist for that reason - I actually wrote a post rejecting the trolley game for ignoring secondary effects. It got a very mixed response, but I stand strongly by one of the points on there -

... everything creates secondary effects. If putting people involuntarily in harm's way to save others was an acceptable result, suddenly we'd all have to be really careful in any emergency. Imagine living in a world where anyone would be comfortable ending your life to save other people nearby - you'd have to not only be constantly checking your surroundings, but also constantly on guard against do-gooders willing to push you onto the tracks.

So I used to think I was a deontologist - "no, I wouldn't push someone onto the tracks to save others, because it's not a good idea to live in a world where people are comfortable ending each other's lives when they deem it for the greater good."

However, after a conversation with a very intelligent person with lots of training in philosophy, I was convinced I'm actually a "rules-based consequentialist" - that I want rules and protocols that produce a general set of consistently good effects rather than running the math every time a trolley is out of control (or a plane is going to crash, or a suspect you're really darn sure did it is in custody but you've got flimsy evidence...)

I like this reply, but I feel it doesn't take the next logical step. What kind of considerations could make utilitarianism correct, given that, as you suggest, a good society needs some firmer rules?

So, just suppose for a moment, a bunch of rational human beings (rational, as human beings go) get together and agree to live by some rather rigid rules. They do so with the best available evidence in front of them, after thinking things through as well as could possibly be expected. They tell each other that you shouldn't push the fat man in front of the trolley, and act accordingly.

What possible sense can it make to say that nevertheless, really and truly, the morally right thing to do is push the fat man? What would "morally right" mean in that sentence, when we have already stipulated that pro-social codes of conduct and recognized virtues, rationally agreed to in open honest well-informed discussion, recommend something else? There is nothing else for morality to "really and truly" be about.

What possible sense can it make to say that nevertheless, really and truly, the morally right thing to do is push the fat man?

In the same sense that I can say it is morally--human wrong, or morally--dog+ wrong for a dog to eat a homeless guy who smells like bacon.

It isn't morally--dog wrong - because it's just a dog. And I agree it does make little sense to talk about whether a dog's actions are morally--human wrong. But if the dog knew of a way to have a better morality according to morality--dog standards, and it was moral--dog to improve one's self in that way when there was no or little cost, then the failure to improve to morality--dog+ is a failure according to morality--dog, and so in an important sense are the dog's actions that violate morality--dog+, though they do not violate morality--dog directly.

This looks like a sketch toward an argument that utilitarianism could be right for humans+. Traditionally, utilitarianism was supposed to be about what is right for us to do.

I can't speak to what is traditional and I don't mind declaring all historical utilitarians wrong in all their debates with non-utilitarians, though I wouldn't mind saying the opposite, either.

Human morality demands a certain amount of thought, and many actions demand moral consideration or their being "good" is no more than fate, and their being bad is negligence.

Upon thinking about it, one realizes that those who think about it should (shouldthosewhothinkaboutit) push the fat man. Those who don't think about it shouldn't (shouldn'tthosewhodon'tthinkaboutit) push the fat man, but should (shouldthosewhodon'tthinkaboutit) think about it.

To ask about unclarified "should" is as to ask about unclarified "sound".

It is important to bear in mind that blame is something humans spray paint onto the unalterable causality of the world, and not to think that either the paint is unalterable because causality is, or that causality is alterable because the paint is.

We can blame humans fully, partially, or not at all for the consequences when they are unthinking, they do what unthinking people should do, there are negative consequences, thinking people should have done a different thing, and those humans should have been thinking people but weren't.

Everything has been explained. There is nothing left in asking if a person really should have done what a thinking person should have done had he or she have been thinking, when the person should have been thinking, and unthinking people were not obligated to do thing.

Upon thinking about it, one realizes that those who think about it should (shouldthosewhothinkaboutit) push the fat man.

One of us hasn't thought enough about it, because I think it takes more than thinking about it. One would also have to know oneself to be largely immune to various biases, which make most humans more prone to rationalize false conclusions about the need to kill someone for the greater good, than to correctly grasp a true utilitarian Trolley Problem. One would have to be human+, if not human(+N). (I think one would also have to live in a human+ or human(+N) community, but never mind about that.)

Note that Greene and other cognitive scientists rarely if ever spell out an airtight case, where the actions save either one life or five lives and magically have no further consequences, and where the utilitarian calculus is therefore clear. Greene simply describes the case more or less as Luke does above, and then leaves the subjects to infer or not infer whatever consequences they might.

The point is that blame can itself have the effect of decreasing the frequency of the behavior that is receiving the blame. So the right question to ask is would having been more exposed to the idea that one should be blamed for doing X have prevented the person from doing X.

So the right question to ask is would having been more exposed to the idea that one should be blamed for doing X have prevented the person from doing X.

My point is that sometimes the answer to "would having been more exposed to the idea that one should be blamed for doing X have prevented the person from doing X?" is no and the answer to "would having been more exposed to the idea that one should be blamed for not trying to improve morally have prevented the person from not improving morally?" is yes and the answer to "would having been more exposed to the idea that one should be blamed for doing X have prevented the person they would be had they tried to improve morally from doing X?" is yes.

Thank you for succinctly stating a good question to ask. The answer to that question may be "no" while the answers to two similar questions are both "yes". Yet by "morally right" many people seem to mean not just situations where the answer to the question you put is "yes", but those in which the answer to the first question is "no" but the answers to to related questions are both "yes". Others mean only cases in which the answer to the first question is "yes", period.

I think I know what people mean by phrases such as "What possible sense can it make to say that nevertheless, really and truly, the morally right thing to do is push the fat man?" or "It is morally right to push the fat man", I think I know why other people are confused, and I do not feel confused by the question, but rather an impulse to unpack it and explain why I think it is confusing.

Person 1: Is it morally right to hit random people you encounter in the street?

Person 2: No. We blame everyone who does that, and consequently people don't do that, even when they want to.

Person 3: I agree.

P1: To kidnap and eat people?

P2: Same answer as to the first question.

P3: Likewise.

P1: For a tiger to kidnap and eat people?

P2: It's not "immoral" because neither the tiger nor anyone else is affected by blame. We guard against tigers, and defend ourselves, and even seek out and kill tigers that have acquired a taste for humans, but we do not castigate tigers.

P3: I agree.

P1: An alien spacecraft has begun abducting and experimenting on people. Who the aliens abduct seems random: sometimes they go to great lengths to reach an individual, but there is no pattern at all among abductees. Every abducted person has had part of their brain removed, and has an apparently irresistible desire to eat people. It seems all abductees must be monitored or restrained for the rest of their natural lives. Is it wrong for them to eat people?

P2: No, they are like the tigers.

P3: I agree.

P1: But previously you both said it was morally wrong to eat people!

P2: It depends on the effects of blame. There is no effect of blaming the abducted cannibals. It's not even like with people who have hidden brain based biological disorders, when failing to blame them weakens the social condemnation for everyone, and there is a weighing to do. The alien case is so one-sided and distinguishable that we can easily tell that the right thing to do is to not blame the abductees, but to blame conventional cannibals.

P3: I agree

P1: Actually, I left something out when describing the aliens. They have kidnapped millions of people, but never anyone wearing anything red, or who have red tatoos, or were in red cars, or were within a few feet of anything not biological that was red. The aliens said as much when they arrived, beaming this information directly into everyone's skulls in their native language several times a day. Are people truly morally responsible for kidnapping and eating people?

P2: Obviously not if they have been abducted and altered. Blame has no effect at all on the behavior of abductees, so they are not "morally responsible". that's the true meaning of "morally responsible", just check the dictionary!

P3: I disagree. Blame may not affect abductees, but it does affect whether or not people get abducted, because the safety measure is so low-cost and easy to implement. Abductees are to blame for eating people, and are truly "morally responsible".

Person 4: If you want to define the term "morally responsible" so that it's simply the answer to one hypothetical question, I don't blame you and I'm willing to play that game. If you think the term naturally covers blaming the abductees for eating people, that's fine too. But Person 1: don't get confused and lose sight of the relationship between blame and people's actions, don't think there is only a small link or no link at all between blame and the number of cannibals just because they are not "truly morally responsible" as you define it. And Person 2: don't get confused by your having a single term such that you might think that if only we blame abductees in the right way, they will stop eating people whenever they can.

What possible sense can it make to say that nevertheless, really and truly, the morally right thing to do is push the fat man?

There is a (maybe not totally coherent, but mostly coherent and natural) way of construing the situation in a vacuum such that pushing the fat man is the right decision.

Alternately, the correct method of analysis that, when carried through to nth order, outputs morality, when carried through to zeroth order recommends pushing the fat man.

This post sums up my own position much more eloquently than I have so far been able to phrase it mentally. Thanks.

I think this whole "utilitarian vs. deontological" setup is a misleading false dichotomy. In reality, the way people make moral judgments -- and I'd also say, any moral system that is really usable in practice -- is best modeled neither by utilitarianism nor by deontology, but by virtue ethics.

All of the puzzles listed in this article are clarified once we realize that when people judge whether an act is moral, they ask primarily what sort of person would act that way, and consequently, whether they want to be (or be seen as) this sort of person and how people of this sort should be dealt with. Of course, this judgment is only partly (and sometimes not at all) in the form of conscious deliberation, but from an evolutionary and game-theoretical perspective, it's clear why the unconscious processes would have evolved to judge things from that viewpoint. (And also why their judgment is often covered in additional rationalizations at the conscious level.)

The "fat man" variant of the trolley problem is a good illustration. Try to imagine someone who actually acts that way in practice, i.e. who really goes ahead and kills in cold blood when convinced by utilitarian arithmetic that it's right to do so. Would you be comfortable working or socializing with this person, or even just being in their company? Of course, being scared and creeped out by such a person is perfectly rational: among the actually existing decision algorithms implemented by human brains, there are none (or at least very few) that would make the utilitarian decision in the fat man-trolley problem and otherwise produce reasonably predictable, cooperative, and non-threatening behavior.

It's similar with the less dramatic examples discussed by Haidt. In all of these, the negative judgment, even if not explicitly expressed that way, is ultimately about judging what kind of person would act like that. (And again, except perhaps for the ideologically polarized flag example, it is true that such behaviors signal that the person in question is likely to be otherwise weird, unpredictable, and threatening.)

I'd also add that when it comes to rationalizations, utilitarians should be the last ones to throw stones. In practice, utilitarianism has never been much more than a sophisticated framework for constructing rationalizations for ideological positions on questions where correct utilitarian answers are at worst just undefined, and at best wildly intractable to calculate. (As is the case for pretty much all questions of practical interest.)

I'd also add that when it comes to rationalizations, utilitarians should be the last ones to throw stones. In practice, utilitarianism has never been much more than a sophisticated framework for constructing rationalizations for ideological positions on questions where correct utilitarian answers are at worst just undefined, and at best wildly intractable to calculate. (As is the case for pretty much all questions of practical interest.)

The phenomenon of utilitarianism serving as a sophisticated framework for constructing rationalizations for ideological positions exists and is perhaps generic. But there's an analogous phenomenon of virtue ethics being rhetorically (think about both sides of the abortion debate). I strongly disagree that utilitarianism is ethically useless in practice. Do you disagree that VillageReach's activity has higher utilitarian expected value per dollar than that of the Make A Wish Foundation?

Yes, there are plenty of situations where game theoretic dynamics and coordination problems make utilitarian style analysis useless, but your claim seems overly broad and sweeping.

I agree that I have indulged in a bit of a rhetorical excess above. What I had in mind is primarily welfare economics -- as I indicated in another comment, I think it's quite evident that this particular kind of formalized utilitarianism is regularly used to construct arguments for various ideological positions that are seemingly rigorous but in fact clearly rationalizations.

I also agree that non-utilitarian theories of ethics are fertile grounds for rationalizations too. I merely wanted to emphasize that given all the utilitarian rationalizations being thrown around, the idea of utilitarian thinking being somehow generally less prone to rationalizations is a non-starter, under any reasonable definitions of these terms.

As for the issues of charity, I think they are also more complicated than they seem, but this is a quite complex topic in its own right, which unfortunately I don't have the time to address right now. I do agree that this area can be seen as a partial counterexample to my general thesis about uselessness of utilitarianism. (But less so than the strong proponents of utilitarian charity commonly claim.)

So I guess the takeaway is that if you care more about your status as a predictable, cooperative, and non-threatening person than about four innocent lives, don't push the fat man.

It's not just about what status you have, but what you actually are. You can view it as analogous to the Newcomb problem, where the predictor/Omega is able to model you accurately enough to predict if you're going to take one or two boxes, and there's no way to fool him into believing you'll take one and then take both. Similarly, your behavior in one situation makes it possible to predict your behavior in other situations, at least with high statistical accuracy, and humans actually have some Omega-like abilities in this regard. If you kill the fat man, this predicts with high probability that you will be non-cooperative and threatening in other situations. This is maybe not necessarily true in the space of all possible minds, but it is true in the space of human minds -- and it's this constraint that gives humans these limited Omega-like abilities for predicting each others' behavior.

(Of course, in real life this is further complicated by all sorts of higher-order strategies that humans employ to outsmart each other, both consciously and unconsciously. But when it comes to the fundamental issues like the conditions under which deadly violence is expected, things are usually simple and clear.)

And while these constraints may seem like evolutionary baggage that we'd best get rid of somehow, it must be recognized that they are essential for human cooperation. When dealing with a typical person, you can be confident that they'll be cooperative and non-threatening only because you know that their mind is somewhere within the human mind-space, which means that as long as there are no red flags, cooperative and non-threatening behavior according to the usual folk-ethics is highly probable. All human social organization rests on this ability, and if humans are to self-modify into something very different, like utility-maximizers of some sort, this is a fundamental problem that must be addressed first.

Another way of saying this (I think - Vladimir_M can correct me):

You only have two choices. You can be the kind of person who kills the fat mat in order to save four other lives and kills the fat man in order to get a million dollars for yourself. Or you can be the kind of person who refuses to kill the fat man in both situations. Because of human hardware, those are your only choices.

I don't mean to imply that the kind of person who would kill the fat man would also kill for profit. The only observation that's necessary for my argument is that killing the fat man -- by which I mean actually doing so, not merely saying you'd do so -- indicates that the decision algorithms in your brain are sufficiently remote from the human standard that you can no longer be trusted to behave in normal, cooperative, and non-dangerous ways. (Which is then correctly perceived by others when they consider you scary.)

Now, to be more precise, there are actually two different issues there. The first is whether pushing the fat man is compatible with otherwise cooperative and benevolent behavior within the human mind-space. (I'd say even if it is, the latter is highly improbable given the former.) The second one is whether minds that implement some such utilitarian (or otherwise non-human) ethic could cooperate with each other the way humans are able to thanks to the mutual predictability of our constrained minds. That's an extremely deep and complicated problem of game and decision theory, which is absolutely crucial for the future problems of artificial minds and human self-modification, but has little bearing on the contemporary problems of ideology, ethics, etc.

It seems like you can make similar arguments for virtue ethics and acausal trade.

If another agent is able to simulate you well, then it helps them to coordinate with you by knowing what you will do without communicating. When you're not able to have a good prediction of what other people will do, it takes waaay more computation to figure out how to get what you want, and if its compatible with them getting what they want.

By making yourself easily simulated, you open yourself up to ambient control, and by not being easily simulated you're difficult to trust. Lawful Stupid seems to happen when you have too many rules enforced too inflexibly, and often (in literature) other characters can take advantage of that really easily.

The second one is whether minds that implement some such utilitarian (or otherwise non-human) ethic could cooperate with each other the way humans are able to thanks to the mutual predictability of our constrained minds.

But we normally seem to see "one death as a tragedy, a million as a statistic" due to scope insensitivity, availability bias etc.

Why not trust that people only directly dealing with numbers are normal when they implement cold-blooded utilitarianism? Why not have many important decisions made abstractly by such people? Is wanting to make decisions this way, remote from the consequences and up a few meta-levels, a barbaric thing to advocate?

Why not trust that people only directly dealing with numbers are normal when they implement cold-blooded utilitarianism? Why not have many important decisions made abstractly by such people? Is wanting to make decisions this way, remote from the consequences and up a few meta-levels, a barbaric thing to advocate?

During the 20th century some societies have attempted to implement more-or-less that policy. The results certainly justify the adjective barbaric.

But most of the people remained relatively normal throughout. So virtue ethics needs a huge patch to approximate consequentialism.

You are providing a consequentialist argument for a base of virtue ethics plus making sure no one makes abstract decisions, but I don't see how preventing people from making abstract decisions emerges naturally from virtue ethics at all.

I agree with your comment in one sense and was trying to imply it, as the bad results are not prevented by virtue ethics alone. On the other hand, you have provided a consequentialist argument that I think valid and was hinting towards.

Spreading this meme, even by a believing virtue ethicist, would seem to reduce the lifespan of fat men with bounties on their heads much faster than it would spare the crowds tied to the train tracks.

U: "Ooo look, a way to rationalize killing for profit!"

VE: "No no no, the message is that you shouldn't kill the fat man in either ca-"

U: "Shush you!"

Of course, one may want to simply be the sort who tells the truth, consequences to fat men be damned.

You only have two choices. You can be the kind of person who kills the fat mat in order to save four other lives and kills the fat man in order to get a million dollars for yourself. Or you can be the kind of person who refuses to kill the fat man in both situations. Because of human hardware, those are your only choices.

This seems obviously false.

It's not just about what status you have, but what you actually are.

Is "what you actually are" equivalent to status of yourself, to yourself?

No, I don't think so. "What I actually am", if I'm understanding Vladimir correctly, refers to the actual actions I take under various situations.

For example, if I believe I'm the sort of person who would throw the fat man under the train, but in fact I would not throw the fat man under the train, then I've successfully signaled to myself my status as a fat-man-under-train-thrower (I wonder if that's an allowed construction in German), but I am not actually a fat-man-under-train-thrower.

http://lesswrong.com/lw/v2/prices_or_bindings/

(Also, please try to avoid sentences like "if you care about X more than innocent lives" — that comes across to me as sarcastic moral condemnation and probably tends to emotionally trigger people.)

I am torn on virtue ethics.

On one level it's almost akin to what a Bayesian calculation (taking "weird but harmless behaviour" as positive evidence of "weird and harmful") would feel like from the inside, and in that respect I can see the value in virtue ethics (even though it strikes me as a mind projection issue of creating a person's ethical 'character' when all you need is the likelihood of them performing this act or that).

But on another level, I can see it is as a description of a sort of hard-coded irrationality that we have evolution to thank for. All things being equal, we prefer to associate with people who will never murder us, rather than people who will only murder us when it would be good to do so - because we personally calculate good with a term for our existence. People with an irrational, compelling commitment are more trustworthy than people compelled by rational or utilitarian concerns (Schelling's Strategy of Conflict), because we are aware that there exists situations where the best outcome overall is not the best outcome personally.

So I am torn between lumping virtue ethics in with deontological ethics as "descriptions of human moral behaviour" and repairing it into a usable set of prescriptions for human moral behaviour.

(even though it strikes me as a mind projection issue of creating a person's ethical 'character' when all you need is the likelihood of them performing this act or that).

Character just is a compressed representation of patterns of likely behavior and the algorithms generating them.

All things being equal, we prefer to associate with people who will never murder us, rather than people who will only murder us when it would be good to do so - because we personally calculate good with a term for our existence. People with an irrational, compelling commitment are more trustworthy than people compelled by rational or utilitarian concerns (Schelling's Strategy of Conflict), because we are aware that there exists situations where the best outcome overall is not the best outcome personally.

This connotes that wanting others to self-bind comes from unvirtuous selfishness, which seems like the wrong connotation to apply to a phenomenon that enables very general and large Pareto improvements (yay!).

In particular (not maximally relevant in this conversation, but particularly important for LW), among fallible (including selfishly biased in their beliefs) agents that wish to pursue common non-indexical values, self-binding to cooperate in the epistemic prisoner's dilemma enables greater group success than a war of all against all who disagree, or mere refusal to cooperate given strategic disagreement.

As to "irrational" (and, come to think of it, also cooperation), see Bayesians vs. Barbarians.

So I am torn between lumping virtue ethics in with deontological ethics as "descriptions of human moral behaviour" and repairing it into a usable set of prescriptions for human moral behaviour.

Why not do both? Treat naive virtue ethics as a description of human moralizing verbal behavior, and treat the virtue-ethical things people do as human game-theoretic behavior, and, because behaviors tend to have interesting and not completely insane causes, look for any good reasons for these behaviors that you aren't already aware of and craft a set of prescriptions from them.

(even though it strikes me as a mind projection issue of creating a person's ethical 'character' when all you need is the likelihood of them performing this act or that).

It's not a fallacy if the thing your projecting onto is an actual human with an actual human mind. Another way to see this is as using the priors on how humans tend to behave that evolution has provided you.

But on another level, I can see it is as a description of a sort of hard-coded irrationality that we have evolution to thank for. All things being equal, we prefer to associate with people who will never murder us, rather than people who will only murder us when it would be good to do so - because we personally calculate good with a term for our existence. People with an irrational, compelling commitment are more trustworthy than people compelled by rational or utilitarian concerns (Schelling's Strategy of Conflict), because we are aware that there exists situations where the best outcome overall is not the best outcome personally.

The definition of "rational" you're using in that paragraph has the problem that it will cause you to regret your rationality. If having an "irrational" commitment helps you be more trusted and thus achieve your goals, it's not irrational. See the articles about decision theory for more details on this.

It's not a fallacy if the thing your projecting onto is an actual human with an actual human mind. Another way to see this is as using the priors on how humans tend to behave that evolution has provided you.

That only works if you're (a) not running in to cultural differences and (b) not dealing with someone who has major neurological differences. Using your default priors on "how humans work" to handle an autistic or a schizophrenic is probably going to produce sub-par results. Same if you assume that "homosexuality is wrong" or "steak is delicious" is culturally universal.

It's unlikely that you'll run in to someone who prioritizes prime-sized stacks of pebbles, but it's entirely likely you'll run in to people who thinks eating meat is wrong, or that gay marriage ought to be legalized :)

Using your default priors on "how humans work" to handle an autistic or a schizophrenic is probably going to produce sub-par results.

They're going to produce the result that this human's brain is wired strangely and thus he's liable to exhibit other strange and likely negative behaviors. Which is more-or-less accurate.

Why on Earth is this comment getting downvoted?

My guess is it's in response to the phrase "negative behaviors" describing a non-neurotypical person's behavior.

Because his comment is evidence for the hypothesis that he has a divergent neurology from mine, and is therefor liable to exhibit negative behaviors :P

(a) not running in to cultural differences and

Indeed, and it probably needs to be emphasized that nations are not monocultures. Americans reading mainly utilitarian blogs and Americans reading mainly deontologist blogs live in different cultures, for instance. (To say nothing about Americans reading atheist blogs and Americans reading fundamentalist blogs, let alone Americans reading any kinds of blogs and Americans who don't read period.)

(even though it strikes me as a mind projection issue of creating a person's ethical 'character' when all you need is the likelihood of them performing this act or that).

This may be part of the reason many virtue ethical theories are prescriptions on what one should do themselves, and usually disapprove of trying to apply it to other human's.On this level, it's poor for predicting, but wonderful for meaningful signalling of cooperative intent. I tend to consider virtue ethics as my low-level compressed version of consequentialist morality; it gives me the ability to develop actions for snap situations that I'd want to take for consequentialist reasons.

As is the case for pretty much all questions of practical interest.

Is it a good idea to spend money on yourself (rather than donating it)?

I don't see how you could possibly rationalize that, and the inconvenience of it would seem to outweigh any benefit it gives to rationalizing other things.

A small nitpick, and without having read the other comments, so please excuse me if this has been mentioned before.

The 5 actions listed under the heading "Emotion and Deontological Judgments" squick me. But they don't disgust me.

From Urban Dictionary:

The concept of the "squick" differs from the concept of "disgust" in that "squick" refers purely to the physical sensation of repulsion, and does not imply a moral component.

Stating that something is "disgusting" implies a judgement that it is bad or wrong. Stating that something "squicks you" is merely an observation of your reaction to it, but does not imply a judgement that such a thing is universally wrong.

It may be useful to add this to our collective vocabulary. Some might argue it's adding unnecessary labels to too-similar a concept, but I think the distinction is useful.

Please, let me know if something like this has been explored already?

Wow. I have the practice (common to sci-fi readers, I have heard) of taking unfamiliar words in my stride, attempting to figure them out in context, and taking it on faith that if I can't figure it out now, more context will soon be given. So that is how I approach new words on the internet (like 'squick'). This is only important because my internal definition for squick had developed into something very much like saying "eww" or the word disgust. It didn't have that crucial 'no moral component' tag for me. Interesting!

Likewise, but I think I have a bit of an obsession with learning obscure jargon... to the point of reading through the provided dictionaries in SF&F books a half dozen times, then referring to it when the words come up. And reading through online lists of terminology for fictional universes and technical activities.

But yes, searching for "squick" on here, I have seen it used as "eww", but I'm not quite sure from the brief glance if it had that particular tag, at least not explicitly.

because my internal definition for squick had developed into something very much like saying "eww" or the word disgust.

Same here.

I think this is more of a prescriptive than descriptive definition of squick. In my experience, people who use the term do not necessarily mean that they make no moral judgment, and in fact, many people, including those who use the term, do not seem to acknowledge a difference between "this gives me a physical sensation of repulsion" and "this is morally wrong."

That Urban Dictionary definition entails that "disgust" does imply a moral component or a judgement that something is universally wrong. However, in my experience, it does not. I can easily imagine a little kid, or a grown adult, declaring a given food or smell or sight "disgusting" without having any objection to its existence. (I can, of course, also imagine a news article in which people interviewed describe someone's immoral behavior as disgusting.) The OED Online describes the word mainly as a visceral reaction and only in passing says it may be brought about by a "disagreeable action".

Instead of creating a new word for what "disgust" currently means and making "disgust" mean something else, perhaps we should leave "disgust" as it is and come up with a word for "moral revulsion". Something like "consternation" or "appallment".

Yeah, it does seem to be phrased such as to imply that.

I can easily imagine a little kid, or a grown adult, declaring a given food or smell or sight "disgusting" without having any objection to its existence. (I can, of course, also imagine a news article in which people interviewed describe someone's immoral behavior as disgusting.)

So the denotative meaning only very mildly indicates a potential for moral revulsion. But used in certain contexts, it does have heavy (heavier) connotations of moral revulsion. I think it's useful to have words for both the physical reaction side and for the moral reaction side, but I disagree with the UD definition in that "disgust" can be more of a generic umbrella term.

So... in other words, use "disgusted" when it's clear, or you mean both. Use "squicked" when it's unclear, and you want to only imply a physical reaction. And use "appalled" when you want to heavily imply moral reaction.

This is all just speculation and suggestion, but I do still hold that the word is useful.

So... in other words, use "disgusted" when it's clear, or you mean both. Use "squicked" when it's unclear, and you want to only imply a physical reaction. And use "appalled" when you want to heavily imply moral reaction.

Yes, I think I agree completely.

I'd guess that there is at least one more variation: Sufficiently bad programming practices (e.g. hard coding "magic numbers" all over the source code) tends to inspire a feeling with a component of disgust in whoever has to maintain the code... Does this generalize? E.g. does discovering that part of the structure of a car is dependent on duct tape lead to similar reactions?

A recent study by folks at the Oxford Centre for Neuroethics suggests that Greene et. al.'s results are better explained by appeal to differences in how intuitive/counterintuitive a moral judgment is, rather than differences in how utilitarian/deontological it is. I had a look at the study, and it seems reasonably legit, but I don't have any expertise in neuroscience. As I understand it, their findings suggest that the "more cognitive" part of the brain gets recruited more when making a counterintuitive moral judgment, whether utilitarian or deontological.

Also, it is worth noting that attempts to replicate the differences in response times have failed (this was the result with the Oxford Center for Neuroethics study as well).

Here is an abstract:

Neuroimaging studies on moral decision-making have thus far largely focused on differences between moral judgments with opposing utilitarian (well-being maximizing) and deontological (duty-based) content. However, these studies have investigated moral dilemmas involving extreme situations, and did not control for two distinct dimensions of moral judgment: whether or not it is intuitive (immediately compelling to most people) and whether it is utilitarian or deontological in content. By contrasting dilemmas where utilitarian judgments are counterintuitive with dilemmas in which they are intuitive, we were able to use functional magnetic resonance imaging to identify the neural correlates of intuitive and counterintuitive judgments across a range of moral situations. Irrespective of content (utilitarian/deontological), counterintuitive moral judgments were associated with greater difficulty and with activation in the rostral anterior cingulate cortex, suggesting that such judgments may involve emotional conflict; intuitive judgments were linked to activation in the visual and premotor cortex. In addition, we obtained evidence that neural differences in moral judgment in such dilemmas are largely due to whether they are intuitive and not, as previously assumed, to differences between utilitarian and deontological judgments. Our findings therefore do not support theories that have generally associated utilitarian and deontological judgments with distinct neural systems.

An important quote from the study:

To further investigate whether neural differences were due to intuitiveness rather than content of the judgment [utilitarian vs. deontological], we performed the additional analyses....When we controlled for content, these analyses showed considerable overlap for intuitiveness. In contrast, when we controlled for intuitiveness, only little--if any--overlap was found for content. Our results thus speak against the influential interpretation of previous neuroimaging studies as supporting a general association between deontological judgment and automatic processing, and between utilitarian judgment and controlled processing.” (p. 7 my version)

Where to find the study (subscription only):

Kahane, G., K. Wiech, N. Shackel, M. Farias, J. Savulescu and I. Tracey, ‘The Neural Basis of Intuitive and Counterintuitive Moral Judgement’, forthcoming in Social, Cognitive and Affective Neuroscience.

Link on Guy Kahane's website: http://www.philosophy.ox.ac.uk/members/research_staff/guy_kahane

PDF of paper. Well done! This is a much better counter-argument to Greene's position than the ones presented in 2007. I shall update the original post accordingly.

Is this page broken for anyone else? When trying to load it, I just get a "Less Wrong broke!" message. I can still see the preview of it here, and I can even hit the 'edit' button from there and successfully update the post, and I can post new comments by replying to comments, but I can't actually load the page that contains this post! Is that happening for anyone else? It's been like this for me for more than an hour now.

It's broken for me too, in exactly the way you describe. One of the variants on the error page invites me to buy a reddit t-shirt.

Broken for me too, including when logged out. So it's probably broken for everyone.

(I hope trike has automatic notification of these page-generation-crashed events, so that there is no point in contacting them manually. A message to this effect (or to the contrary) on the "page crashed" page would be nice.)