I'm now definitely planning to write the above-mentioned post, discussing various definitions, types, and sources of moral uncertainty. As this will require thinking more deeply about that topic, I'll be able to answer more properly once I have. (And I'll also comment a link to post that in this thread.)
For now, some thoughts in response to your comment, which are not fully-formed and not meant to be convincing; just meant to indicate my current thinking and maybe help me formulate that other post through discussion:
Cross-posted to the EA Forum. Updated substantially since initial publication.
Overview/purpose of this sequence
While working on an (upcoming) post about a new way to think about moral uncertainty, I unexpectedly discovered that, as best I could tell:
There was no single post on LessWrong or the EA Forum that very explicitly (e.g., with concrete examples) overviewed what seem to be the most prominent approaches to making decisions under moral uncertainty (more specifically, those covered in Will MacAskill’s 2014 thesis).[1][2]
There was no (easily findable and explicit) write-up of how to handle simultaneous moral and empirical uncertainty. (What I'll propose is arguably quite obvious, but still seems worth writing up explicitly.)
There was no (easily findable and sufficiently thorough) write-up of applying sensitivity analysis and value of information analysis to situations of moral uncertainty.
I therefore decided to write a series of three posts, each of which addressed one of those apparent “gaps”. My primary aim is to synthesise and make accessible various ideas that are currently mostly buried in the philosophical literature, but I also think it’s plausible that some of the ideas in some of the posts (though not this first one) haven’t been explicitly explored before.
I expect that these posts are most easily understood if read in order, but each post should also have value if read in isolation, especially for readers who are already familiar with key ideas from work on moral uncertainty.
Epistemic status (for the whole sequence)
I've now spent several days reading about moral uncertainty, but I wouldn't consider myself an actual expert in this topic or in philosophy more broadly. Thus, while I don't expect this sequence to contain any major, central mistakes, I wouldn’t be surprised if it's inaccurate or unclear/misleading in some places.
I welcome feedback of all kinds (on these posts and in general!).
Moral uncertainty
We are often forced to make decisions under conditions of uncertainty. This uncertainty can be empirical (e.g., what is the likelihood that nuclear war would cause human extinction?) or moral (e.g., does the wellbeing of future generations matter morally?).[3][4] The issue of making decisions under empirical uncertainty has been well-studied, and expected utility theory has emerged as the typical account of how a rational agent should proceed in these situations. The issue of making decisions under moral uncertainty appears to have received less attention (though see this list of relevant papers), despite also being of clear importance.
I'll later publish a post on definitions, types, and sources of moral uncertainty. In the present post, I'll instead aim to convey a sense of what moral uncertainty is through various examples. One example (which I'll return to repeatedly) is the following:
Devon's decision
The choice-worthiness of each option according to each theory is summarised in the following table:
Given this information, what should Devon do?
"My Favourite Theory"
Multiple approaches to handling moral uncertainty have been proposed. The simplest option is the "My Favourite Theory" (MFT) approach, in which we essentially ignore our moral uncertainty, and just do whatever seems best based on the theory in which one has the highest "credence" (belief). In the above situation, MFT would suggest Devon should buy the fish curry, even though doing so is only somewhat better according to T2 (10 - 5 = 5), and is far worse (5 - -90 = 95) according to another theory in which he has substantial (25%) credence. Indeed, even if Devon had 49% credence in T1 (vs 51% in T2), and the difference in the choice-worthiness of the options was a thousand times as large according to T1 as according to T2, MFT would still ignore the fact the situation is so much "higher stakes" for T1 than T2, refuse to engage in any "moral hedging", and advise Devon proceed with whatever T2 advised.
On top of generating such counterintuitive results, MFT is subject to other quite damning objections (see pages 20-25 of Will MacAskill’s 2014 thesis). Thus, the remainder of this post will focus on other approaches to moral uncertainty, which do allow for "moral hedging".
Types of moral theories
Which approach to moral uncertainty should be used depends in part on what types of moral theories are under consideration by the decision-maker - in particular, whether the theories are cardinally measurable or only ordinally measurable, and, if cardinally measurable, whether or not they’re inter-theoretically comparable.[6]
Cardinality
Essentially, a theory is cardinally measurable if it can tell you not just which outcome is better than which, but also by how much. E.g., it can tell you not just that “X is better than Y which is better than Z”, but also that “X is 10 ‘units’ better than Y, which is 5 ‘units’ better than Z”. (Some readers may be more familiar with distinctions between ordinal, interval, and ratio scales; I'm almost certain "cardinal" scales include both interval and ratio scales.)
My understanding is that popular consequentialist theories are typically cardinal, while popular non-consequentialist theories are typically (or at least more often) ordinal. For example, a Kantian theory may simply tell you that lying is worse than not lying, but not by how much, so you cannot directly weigh that “bad” against the goodness/badness of other actions/outcomes (whereas such comparisons are relatively easy under most forms of utilitarianism).
Intertheoretic comparability
Even if a set of theories are cardinal, they still may not be inter-theoretically comparable. Roughly speaking, two theories are comparable if there's a consistent, non-arbitrary “exchange rate” between the theories' “units of choice-worthiness" (and they're non-comparable if there isn't). MacAskill explains the “problem of intertheoretic comparisons” as follows:
In his own thesis, Tarsney provides useful examples:
It's worth noting that similar issues have received attention from, and are relevant to, other fields as well. For example, MacAskill writes: "A similar problem arises in the study of social welfare in economics: it is desirable to be able to compare the strength of preferences of different people, but even if you represent preferences by cardinally measurable utility functions you need more information to make them comparable." Thus, concepts and findings from those fields could illuminate this matter, and vice versa.
Three approaches
In MacAskill's thesis, the approaches to moral uncertainty he argues for are:
I will focus on these approaches (excluding Hybrid procedures), both because these approaches seem to me to be relatively prominent, effective, and intuitive, and because I know less about other approaches. (Potentially promising alternatives include a bargaining-theoretic approach [related presentation slides here], the similar but older and less fleshed-out parliamentary model, and the approaches discussed in Tarsney's thesis.)
Maximising Expected Choice-worthiness (MEC)
MEC is essentially an extension of expected utility theory. MacAskill describes MEC as follows:
In this formula, C(Ti) represents the decision-maker’s credence (belief) in Ti (some particular moral theory), while CWi(A) represents the “choice-worthiness” (CW) of A (an “option” or action that the decision-maker can take), according to Ti.
To illustrate how MEC works, we will return to the example of Devon deciding whether to buy a fish curry or tofu curry, as summarised in the table of choice-worthiness values from earlier:
(I’ve also modelled this example in Guesstimate. In that link, for comparison purposes, this model is followed by a model of the same basic example using traditional expected utility reasoning, and another using MEC-E (an approach I'll explain in my next post).)
Using MEC in this situation, the expected choice-worthiness of buying the fish curry is 0.25 * -90 + 0.75 * 10 = -15, and the expected choice-worthiness of buying the tofu curry is 0.25 * 5 + 0.75 * 5 = 5. Thus, Devon should buy the tofu curry.
This is despite Devon believing that T2 is more likely than T1, and T2 claiming that buying the fish curry is better than purchasing the tofu curry. The reason is that, as discussed earlier, there is far “more at stake” for T1 than for T2 in this example.
To me, this seems like a good, intuitive result for MEC, and shows how it improves upon the “My Favourite Theory” approach.
There are two final things I should note about MEC:
MEC can be used in exactly the same way when more than two theories are under consideration. (The only reason most examples in this sequence will be ones in which only two moral theories are under consideration is to keep explanations simple.)
The basic idea of MEC can also be used as a heuristic, without involving actual numbers.
Another example of applying MEC (which is probably only worth reading if the approach still seems unclear to you) can be found in the following footnote.[8]
Normalised MEC and Variance Voting
(It's possible I've made mistakes in this section; if you think I have, please let me know.)
But what about cases in which, despite being cardinal, the theories you have credence in are not intertheoretically comparable? (Recall that this essentially means that there's no consistent, non-arbitrary “exchange rate” between the theories' “units of choice-worthiness".)
MacAskill argues that, in such situations, one must first "normalise" the theories in some way (which basically means "adjusting values measured on different scales to a notionally common scale"). MEC can then be applied just as we saw earlier, but now with the new, normalised choice-worthiness scores.
There are multiple ways one could normalise the theories under consideration (e.g., by range), but MacAskill argues for normalising by variance. That is, he argues that we should:
MacAskill uses the term Variance Voting to refer to this process of first normalising by variance and then using the MEC approach.
(Unfortunately, as far as I could tell, none of the three theses/papers I read that referred to normalising moral theories by variance actually provided a clear, worked example. I've attempted to construct such a worked example based on an extension of the scenario with Devon deciding what meal to buy; that can be found here, and here is a simpler and I think effectively identical method, suggested in a private message.)
In arguing for Variance Voting over its alternatives, MacAskill states that the basic principle normalisation aims to capture is the “principle of equal say: the idea, stated imprecisely for now, that we want to give equally likely moral theories equal weight when considering what it’s appropriate to do” (emphasis in original). He further writes:
(Note that it’s not a problem for one theory to have much more influence on decisions due to higher credence in that theory. The principle of equal say is only violated if additional influence is unrelated to additional credence in a theory, and instead has to do with what are basically arbitrary/accidental choices about exchange rates between units of choice-worthiness.)
MacAskill (pages 110-116) provides two arguments that VV is the approach that satisfies the principle of equal say, and Owen Cotton-Barratt similarly argues for the superiority of normalisation by variance over alternative normalisations. (But note that this approach does seem to have its flaws, as discussed in, e.g., pages 222-223 of Tarsney's thesis.)
The Borda Rule (BR)
Finally, what about cases in which all moral theories you have credence in are only ordinal, rather than cardinal (i.e., they say only whether each option is more, equally, or less choice-worthy than each other option, but not by how much)? For such cases, MacAskill recommends a voting method called the Borda Rule (BR; also known as a "Borda count"), with "moral theories play[ing] the part of voters and practical options the part of candidates" (Tarsney). I will first quote MacAskill’s formal explanation of BR (which may be somewhat confusing by itself), before quoting an example he gives and showing what applying BR to that looks like:
I will now show, following MacAskill, how this rule applies to an example he gives in his thesis:
The options’ Borda Scores according to each theory, and their Credence-Weighted Borda Scores, are therefore as shown in the following table:
(For example, G has a score of 2 - 0 = 2 according to utilitarianism because that theory views two options as less choice-worthy than G, and 0 options as more choice-worthy than G.)
The calculations that provided the Credence-Weighted Borda Scores shown in the above table are as follows:
BR would therefore claim that Julia should call for a retrial. This is the case even though passing a guilty verdict was seen as best by Julia’s “favourite theory” (the variant of utilitarianism). Essentially, calling for a retrial is preferred because both passing a guilty verdict and passing an innocent verdict were seen as least preferred by some theory Julia has substantial credence in, whereas calling for a retrial is not least preferred by any theory.
MacAskill notes that preferring this sort of a compromise option in a case like this seems intuitively right. He also argues that alternatives to BR fail to give us the sort of answers we’d want in these or other sorts of cases. (Though Tarsney raises some objections to BR which I won't get into.)
Closing remarks
I hope you have found this post a useful, clear summary of key ideas around what moral uncertainty is, why it matters, and how to make decisions when morally uncertain. Personally, I believe that an understanding of moral uncertainty - particularly a sort of heuristic version of MEC - has usefully enriched my thinking, and influenced some of the biggest decisions I’ve made over the last year.[10]
In the next post, I will discuss (possibly novel, arguably obvious) extensions of each of the three approaches discussed here, in order to allow for modelling both moral and empirical uncertainty, explicitly and simultaneously. The post after that will discuss how we can combine the approaches in the first two posts with sensitivity analysis and value of information analysis.[11][12]
I genuinely mean no disrespect to the several posts on moral uncertainty I did discover (e.g., here, here, and here). All did meet some of those criteria, and I’d say most were well-written but just weren’t highly explicit (e.g., didn’t include enough concrete examples), and/or didn’t cover (in the one post) each of the prominent approaches and the related ideas necessary to understand them. ↩︎
Other terms/concepts that are sometimes used and are similar to “moral uncertainty” are normative, axiological, and value uncertainty. In this sequence, I’ll use “moral uncertainty” in a general sense that also incorporates axiological and value uncertainty, and at least a large part of normative uncertainty.
Also, throughout this sequence, I will use the term "approach" in a way that I believe aligns with MacAskill's use of the term "metanormative theory". ↩︎
It seems to me that there are many cases where it’s not entirely clear whether the uncertainty is empirical or moral. For example, I might wonder “Are fish conscious?”, which seems on the face of it an empirical question. However, I might not yet know precisely what I mean by “conscious”, and only really want to know whether fish are “conscious in a sense I would morally care about”. In this case, the seemingly empirical question becomes hard to disentangle from the (seemingly moral) question “What forms of consciousness are morally important?”
(Furthermore, my answers to that question in turn may be influenced by empirical discoveries. For example, I may initially believe avoidance of painful stimuli demonstrates consciousness in a morally relevant sense, but then change that belief after learning that this behaviour can be displayed in a stimulus-response way by certain extremely simple organisms.)
In such cases, I believe the approach suggested in the next post of this sequence will still work well, as that approach does not really require empirical and moral uncertainty to be treated fundamentally differently. (Another approach, which presents itself differently but I think is basically the same in effect, is to consider uncertainty over “worldviews”, with those worldviews combining moral and empirical claims.) ↩︎
In various places in this sequence, I will use language that may appear to endorse or presume moral realism (e.g., referring to “moral information” or to probability of a particular moral theory being “true”). But this is essentially just for convenience; I intend this sequence to be neutral on the matter of moral realism vs antirealism, and I believe this post can be useful in mostly similar ways regardless of one’s position on that matter. I discuss the matter of "moral uncertainty for antirealists" in more detail in this separate post. ↩︎
The matter of how to actually assign “units” or “magnitudes” of choice-worthiness to different options, and what these things would even mean, is complex, and I won’t really get into it in this sequence. ↩︎
Christian Tarsney's 2017 thesis thesis (e.g., pages 175-176) explains other ways the "structure" of moral theories can differ, and potential implications of these other differences. These were among the juicy complexities I had to resist cramming in this originally-intended-as-bitesized post (but I may write another post about Tarsney's ideas later; please let me know if you think that'd be worthwhile). ↩︎
It's worth noting that similar issues have received attention from, and are relevant to, other fields as well. For example, MacAskill writes: "A similar problem arises in the study of social welfare in economics: it is desirable to be able to compare the strength of preferences of different people, but even if you represent preferences by cardinally measurable utility functions you need more information to make them comparable." Thus, concepts and findings from those fields could illuminate this matter, and vice versa. ↩︎
Suppose Alice assigns a 60% probability to hedonistic utilitarianism (HU) being true and a 40% probability to preference utilitarianism (PU) being true. Suppose also that Bob wants to play video games, but would actually get slightly more joy out of a day at the beach. Thus, according to HU, letting Bob play video games has a CW of 5, and taking him to the beach has a CW of 6; while according to PU, letting Bob play video games has a CW of 15, and taking him to the beach has a CW of -20.
Under these conditions, the expected choice-worthiness of letting Bob play video games is 0.6 * 5 + 0.4 * 15 = 9, and the expected choice-worthiness of taking Bob to the beach is 0.6 * 6 + 0.4 * -20 = -4.4. Therefore, Alice should let Bob play video games.
Analogously to the situation with the Devon example, this is despite Alice believing HU is more likely than PU, and despite HU positing that taking Bob to the beach being better than letting him play video games. As before, the reason is that there is “more at stake” in this decision for the less-believed theory than for the more-believed theory; HU considers there to only be a very small difference between the choice-worthiness of the options, while PU considers there to be a large difference. ↩︎
MacAskill later notes that a simpler method (which doesn’t subtract the number of options that are more choice-worthy) can be used when there are no ties. His calculations for the example I quote and work through in this post use that simpler method. But in this post, I’ll stick to the method MacAskill describes in this quote (which is guaranteed to give the same final answer in this example anyway). ↩︎
However, these concepts are of course not an instant fix or cure-all. In a (readable and interesting) 2019 paper, MacAskill writes “so far, the implications for practical ethics have been drawn too simplistically [by some philosophers.] First, the implications of moral uncertainty for normative ethics are far more wide-ranging than has been noted so far. Second, one can't straightforwardly argue from moral uncertainty to particular conclusions in practical ethics, both because of ‘interaction’ effects between moral issues, and because of the variety of different possible intertheoretic comparisons that one can reasonably endorse.”
For a personal example, a heuristic version of MEC still leaves me unsure whether I should move from being a vegetarian-flirting-with-veganism to a strict vegan, or even whether I should spend much time making that decision, because that might trade off to some extent with time and money I could put towards longtermist efforts (which seem more choice-worthy according to other moral theories I have some credence in). I suspect any quantitative modelling simple enough to be done in a reasonable amount of time would still leave me unsure.
That said, I, like MacAskill (in the same paper), “do believe, however, that consideration of moral uncertainty should have major impacts for how practical ethics is conducted. [...] It would be surprising if the conclusions [of approaches taking moral uncertainty into account] were the same as those that practical ethicists typically draw.”
In particular, I’d note that considering moral uncertainty can reveal some “low-hanging fruit”: some “trades” between moral theories that are relatively clearly advantageous, due to large differences in the “stakes” different moral theories see the situation as having. (Personally, cases of apparent low-hanging fruit of this kind have included becoming at least vegetarian, switching my career aims to longtermist ones, and yet engaging in global-poverty-related movement-building when an unusual opportunity arose and it wouldn’t take up too much of my time.) ↩︎
To foreshadow: Basically, my idea is that, once you’ve made explicit your degree of belief in various moral theories and how good/bad outcomes appear to each of those theories, you can work out which updates to your beliefs in moral theories or to your understandings of those moral theories are most likely to change your decisions, and thus which “moral learning” to prioritise and how much resources to expend on it. ↩︎
I’m also considering later adding posts on:
I’d be interested in hearing whether people think those threads are likely to be worth pursuing. ↩︎