One reason why conversations about morality are often unproductive is that there are two seperate conversations that often get mixed together. The first is principled morality: How ought perfect, incorruptible beings with no false beliefs act, further assuming the absence of any side-effects? The second is practical morality: How ought flawed, corruptible beings with false beliefs act, accounting for the possibility of side-effects?

I believe that both of these questions are ultimately valuable: the pragmatic because it tells us how to ultimately act; the principled because it is instructive. This post will focus on defending the value of discussing principled morality as people on Less Wrong generally don't object to discussing pragmatic morality. However, I grant that there might exist some very strong deontologists who would require the opposite message. More commonly though, I suspect that people who ignore pragmatic morality have just accidentally skipped a step and forgotten to account for practical considerations.

I already explained in my post on the Direct Application Fallacy that even if a hypothetical situation never occurs in real life, it can still provide opportunities for learning, be tied back to the real through further argument or make us aware of the limitations of our model. Eliezer expresses a similar sentiment in a recent Facebook post, where he identifies a particular kind of argument as stemming from, "one of those awful durable forms of misunderstanding where Msr. Toolbox doesn't see what you could possibly be telling somebody to do with a "good" or "ideal" algorithm besides executing that algorithm."

Further, separating the discussion of principled and pragmatic issues is important for the same reason that local validity is important. If you have to clearly specify your stance on principled morality and then your stance on how these considerations play out practically, you have less scope to fudge that if you refuse to seperate them.

Objections

Undoubtedly, contextualisers would object that insisting on this kind of discussion is often harmful. For example, someone might oppose torture because they believe that giving a government that much power is too risky. They might want to refuse to discuss the principled question where this consideration is excluded because they might be worried that someone could take their in principle exception and use it to support the policy in practise. However, even though we have to be conscious of the effects of our words in a public settings, if we are to have any hope of coming to the truth, we'll need contexts in which the decoupling-style of conversation is allowed to take place.

Another objection would be that only deontologists need a concept of principled morality, utilitarians don't need them. My response is that I include utilitarianism as a form of principled morality. I would also note that there are different forms of utilitarianism: hedonic/preference, total/sum, act/rule, ect. Plus there are different possible utility functions. If you've already agreed on the precise rule set in advance, then I'm perfectly fine with you skipping the principled discussion.

Another objection would be that these questions are ultimately unanswerable, so we should focus on pragmatic morality instead. I'm not going to insist on a precise model of morality, but you should have at least some idea of what kinds of things you value. Otherwise, there is no reason to think this heuristic has any value. If you say, "Utility, but also freedom and fairness", then that's a starting point. Even if we can't figure out exactly how they relate, we should be able to at least rule some theories out, such as theories that purely optimise for fairness, even if that results in everyone being miserable. I don't have an issue with proposing heuristics if you think universal rules are too hard, but you should at least have some idea of what factors that heuristic is trying to balance and over what range it is reasonable.

Pragmatic Factors

Given all of this, it seems natural to ask what specific factors are relevant in practical morality, but can be ignored in principled morality. Here is a list of some factors we might want to take into account:

• Willpower - Utilitarianism taken literally is too demanding for humans to follow and attempting to follow it would lead most humans to burn out.
• Malleability - If the rules are too loose, we can reinterpret them when convenient. For example, if my rule is, "I ought not to lie, except when I have a good reason to do so", then I will always be able to think up some kind of justification.
• Corruption - It can be dangerous to put yourself in a position where you stand to gain massively by acting immorally. For example, you might not want to become a tobacco researcher unless you are sure you won't sell out.
• Epistemic fallibility - Our beliefs about how the situation is may be misleading. Maybe you think you can steal money to donate to the poor without anyone finding out, but maybe you are overconfident.
• Simplicity - We need to be able to remember the rules or principles that we are supposed to be following and to figure out how they apply without spending hours deliberating over each decision.
• Psychological compatibility - Human motivation works in a particular way. We may be better at following some moral rules than others because they are more compatible with human psychology. For example, humans naturally care more about their family and their "tribe" than strangers.
• Selection effects - Some moral rules may not be worth following as the only people who would follow them would be exactly the kinds of people who you don't want to follow them. For example, if you tell people not to have children, probably the only people who'd take this seriously are the kinds of people who take morality seriously and who would hence likely bring up children with strong moral values.
• Virality - Some moral claims may be unpopular enough that it isn't worth spreading them. For example, trying to persuade people to support gay marriage a few hundreds years ago would have likely been ineffective.
• Verifiability - Similar to virality, but some people pretend to have adopted a specific moral system without actually following it any more than necessary for appearances.
• Chesterton's Fence - Some rules may exist for good reasons, even if we don't know what those reasons are
• Character building - Some actions might allow you to build your character better than others

I've often seen naive utilitarian analyses which have missed one of these factors when it would make a large difference in what we ought to do. Further, it is worth noting that principled discussion of morality doesn't always exclude all of these factors. Indeed, some of these may sometimes be written into the scenario being discussed. Instead, the focus is on limiting the discussion narrowly, in the hope that this makes progress easier.