If an act-utilitarian surgeon believes they could save the lives of five patients in need of organ transplants by killing one patient and harvesting their organs, should they do so?[1]
The main reason the surgeon should not harvest someone's organs is because of a distinction between how one determines the best outcome (defined by some criterion of rightness, i.e., maximizing utility) and how one makes decisions with that outcome in mind (i.e., a decision procedure, which involves considerations beyond ethics). This is by no means a novel realization. For some short posts on two-level utilitarianism, see Amanda Askell’s or Richard Y. Chappell's EA Forum posts. Still, there are some useful specifics I haven't seen talked about regarding why and how your decision procedure should differ from simply maximizing utility. The Feeling of Value, a philosophy dissertation by Dr. Sharon Hewitt Rawlette, provides a good starting point to think through these questions, and most of these thoughts are from that.
Time and cognitive capacity constraints are among the most obvious ways a simple utilitarian decision procedure doesn’t work. There are just far too many considerations to think through for most decisions! But even under more charitable conditions, it fails. Here’s a short numbered list of these failures.
Consideration 1: Uncertainty
When deciding what course of action is best, we might think about past experiences we had that were similar. For instance, when deciding whether to meet up with friends, you could consider how much fun you had hanging out with them in the past. There’s an impulse to think that considering more specific information in your decision will lead to a better decision. However, an area of research in the normative decision-making literature argues that you are often more accurate when focusing on a few simple factors; heuristics can actually improve outcomes in situations of uncertainty[2]. You might be better off not considering the fun you have when you hang out with friends on Thursdays specifically, or even with that specific group of people. Simply thinking about how much fun you have hanging out with friends in general might be most accurate.
This is because whenever you add additional qualifiers to the class of experiences you are referencing, you reduce the number of past experiences you have to base your decision on. This makes your final decision more susceptible to random variation. By making decisions based on rules, you accept some of the bias that comes from a heuristic in exchange for reducing the high variance from your limited information. Things are often complicated enough that even if we think we have come to the correct decision, a simpler, rule-based analysis would actually have been more likely to be correct.[3]
It turns out naive utilitarians have no way of coordinating with each other.
Say you and your friend Dave want to meet up for dinner, and you both promise to meet at the pizza place near your house. But you’re both only making decisions by maximizing utility, and you both know the other is as well. That means the only reason you have to follow through on your promise is if there’s some actual harm in breaking it. At first glance, it seems like there would be harm: if you went somewhere besides the pizza place, Dave wouldn’t be there! However, since Dave knows you don’t actually care about promises,[4] he has no evidence you’ll actually go to the pizza place. In which case, he also has no reason to go to the pizza place.
This is the same problem as grounding coordination in game theory with purely rational players. To reasonably coordinate, you need to be able to make inferences about the other actor that aren’t based on what’s strictly optimal for them.
However, if you act with even a slight disposition to keep your promise to go to the pizza place, Dave suddenly gains an incentive to join you. Then you know he’s incentivized to go, so you’re even morelikely to go, and so he’s more likely, and so on. If anyone has any tendency to trust others’ promises or follow through on the promises they make, both parties can have reasonable expectations of the other.[5]
And even if no one expects you to keep your word, breaking promises still erodes trust. While the impact is diffuse, breaking a single promise ever so slightly affects how common people believe promise-breaking to be.
Consideration 3: Motivation
The third problem is motivation. Even if we want to maximize utility, humans are biased toward our own interests and the interests of our loved ones. One way to leverage people’s tendency to prioritize themselves is to allow each person to have control over their own well-being. In other words, rights. In a world without rights, you have people uninterested in the welfare of others making decisions for those others. Since people are worse at making decisions that prioritize others, this world is probably worse off. It’s best to have a strong connection between someone’s decisions and their ability to positively affect themselves and those they love. Rights like autonomy over one’s body, property, etc., allow individuals to take ownership of their own welfare in a way that’s welfare-maximizing for the world as a whole.[6]
This extends beyond the human rights you might be immediately thinking of. Rawlette brings up the example of individuals in a corporate setting who “own” a certain sphere of influence. Think of someone being in charge of a certain feature of a website, for example. The decision to give the worker ownership of the feature was probably openly made by maximizing utility for the company. But still, once that decision is made, others making decisions for the feature might be considered an infringement on the worker’s “right” to control the feature. Setting this norm allows the company to work more smoothly overall, even if in a specific instance someone thinks they could improve things by autonomously making changes in someone else’s sphere of influence.
Beyond respecting the rights of others, we should also make personal rules against selfish actions generally. It’s much easier to convince ourselves that an action is good if it benefits us. This becomes more relevant the more uncertain a situation is: the complexity gives us more opportunities to rationalize. But if we make a personal rule to never steal, for instance, we prevent ourselves from rationalizing a bad decision into something that seems good. These sorts of rules also force us to consider alternatives that we might not have thought of otherwise.
It’s also good for society to have these rules. If stealing makes everyone mad at you, you’ll have a self-interested reason not to do it. Your selfish motivations have been hijacked to force you to internalize harms to others. Even if stealing is the utilitarian best decision for someone at some given time, society should still condemn it ~universally to maintain a consistent incentive structure.
Conclusions
The best way to make decisions isn’t always the one that thinks exclusively about utilities. Even if your end goal is utility maximization, when you’re uncertain about outcomes, it’s best to prioritize trust, cooperation, rights, and broad rules against selfish actions. The rules aren’t given any direct moral weight in themselves; we’re still in act-utilitarian land, but they act as guidelines on how to arrive at the best outcome when knowledge or motivations aren’t perfect. With that being said, I am somewhat uncertain about all this. Yielding your moral considerations to common norms feels antithetical to rationalism and effective altruism if over-applied. I'm especially not confident about the uncertainty point, somewhat ironically.
If you want more of the specifics as to why organ-stealing exists in the domain of uncertainty where norms against murder and such should come into play, you can read my original post. The most important takeaways are that committing ahead of time to not stealing organs forces you to look for alternatives, and that both secrecy and discovery have their own major drawbacks. It’s easy to think of the utilitarian actions in these decisions as being at odds with patient rights. But the more nuanced view is that considering rights when making decisions is valuable even when thinking consequentially. And similarly, if you care about rights, that doesn’t preclude you from thinking about consequences.
I also think this sort of multi-level framing of utilitarianism is useful to keep in mind when having discussions about ethics. I genuinely believe most people are more or less act utilitarians in their latent criterion of rightness, but interpret ethical questions as pressing on their heavily rule-based decision procedure instead. For example, some research finds that 95% of people learn from consequences when making ethical decisions. But if you ask the average person why someone has a right not to get their organs stolen, I seriously doubt they will be willing to justify it as anything more than self-evident. People value consequences, but tend to think and speak in rules. In my observation, self-described utilitarians are more comfortable separating these levels, while less philosophically-minded people tend to strongly resist conceptualizing an act as instrumentally but not intrinsically valuable. This isn’t to say real deontologists don’t exist, but utilitarians should be a little more forgiving of people who say things like “equality is intrinsically valuable.” Most of the time, those sorts of statements should be read as normative claims about decision procedures, not value criteria.
Rawlette cites Gigerenzer & Gaissmaier, 2011 here. I also found Hafenbrädl et al., 2016 and Ehrig & Schmidt, 2019, who have more recent and focused discussions of the topic. Maier et al., 2026 has a neat paper about how humans instinctively do this already, though there’s a lot of work on how moral decisions function differently from other decisions, so that likely doesn’t apply in this case. In fact, I strongly assume most people have the opposite problem from what we’re solving here and rely too strongly on heuristic rules or instincts when making moral decisions.
You could phrase this as simply having a strong prior towards common-sense norms being best, and that would be fine. It would be slightly different in framing, however; in the approach I’ve described, you can ignore whatever you believe the best action to be rather than change what you believe that best action is, but that distinction is a little murky.
... and also knows that you know that he doesn’t care about promises, and also knows that you know that he knows that you don’t care about promises, et cetera. This relies on arbitrarily deep common knowledge about the other person’s beliefs.
This story isn’t how it works in the real world. In reality, since no one is strictly a naive act-util decision maker, the problem never arises. You immediately have grounds to trust others as well as an incentive to provide grounds for others to trust you. But it’s still a cautionary tale against becoming a naive util decision maker.
(This is a heavily edited version of a post on my Substack)
If an act-utilitarian surgeon believes they could save the lives of five patients in need of organ transplants by killing one patient and harvesting their organs, should they do so?[1]
The main reason the surgeon should not harvest someone's organs is because of a distinction between how one determines the best outcome (defined by some criterion of rightness, i.e., maximizing utility) and how one makes decisions with that outcome in mind (i.e., a decision procedure, which involves considerations beyond ethics). This is by no means a novel realization. For some short posts on two-level utilitarianism, see Amanda Askell’s or Richard Y. Chappell's EA Forum posts. Still, there are some useful specifics I haven't seen talked about regarding why and how your decision procedure should differ from simply maximizing utility. The Feeling of Value, a philosophy dissertation by Dr. Sharon Hewitt Rawlette, provides a good starting point to think through these questions, and most of these thoughts are from that.
Time and cognitive capacity constraints are among the most obvious ways a simple utilitarian decision procedure doesn’t work. There are just far too many considerations to think through for most decisions! But even under more charitable conditions, it fails. Here’s a short numbered list of these failures.
Consideration 1: Uncertainty
When deciding what course of action is best, we might think about past experiences we had that were similar. For instance, when deciding whether to meet up with friends, you could consider how much fun you had hanging out with them in the past. There’s an impulse to think that considering more specific information in your decision will lead to a better decision. However, an area of research in the normative decision-making literature argues that you are often more accurate when focusing on a few simple factors; heuristics can actually improve outcomes in situations of uncertainty[2]. You might be better off not considering the fun you have when you hang out with friends on Thursdays specifically, or even with that specific group of people. Simply thinking about how much fun you have hanging out with friends in general might be most accurate.
This is because whenever you add additional qualifiers to the class of experiences you are referencing, you reduce the number of past experiences you have to base your decision on. This makes your final decision more susceptible to random variation. By making decisions based on rules, you accept some of the bias that comes from a heuristic in exchange for reducing the high variance from your limited information. Things are often complicated enough that even if we think we have come to the correct decision, a simpler, rule-based analysis would actually have been more likely to be correct.[3]
This is a contentious topic, both in psychology and among rationalists. I'm not taking a strong stance on the rationality wars, but I think it is reasonable to default to norms in situations of deep uncertainty.
Consideration 2: Trust
It turns out naive utilitarians have no way of coordinating with each other.
Say you and your friend Dave want to meet up for dinner, and you both promise to meet at the pizza place near your house. But you’re both only making decisions by maximizing utility, and you both know the other is as well. That means the only reason you have to follow through on your promise is if there’s some actual harm in breaking it. At first glance, it seems like there would be harm: if you went somewhere besides the pizza place, Dave wouldn’t be there! However, since Dave knows you don’t actually care about promises,[4] he has no evidence you’ll actually go to the pizza place. In which case, he also has no reason to go to the pizza place.
This is the same problem as grounding coordination in game theory with purely rational players. To reasonably coordinate, you need to be able to make inferences about the other actor that aren’t based on what’s strictly optimal for them.
However, if you act with even a slight disposition to keep your promise to go to the pizza place, Dave suddenly gains an incentive to join you. Then you know he’s incentivized to go, so you’re even more likely to go, and so he’s more likely, and so on. If anyone has any tendency to trust others’ promises or follow through on the promises they make, both parties can have reasonable expectations of the other.[5]
And even if no one expects you to keep your word, breaking promises still erodes trust. While the impact is diffuse, breaking a single promise ever so slightly affects how common people believe promise-breaking to be.
Consideration 3: Motivation
The third problem is motivation. Even if we want to maximize utility, humans are biased toward our own interests and the interests of our loved ones. One way to leverage people’s tendency to prioritize themselves is to allow each person to have control over their own well-being. In other words, rights. In a world without rights, you have people uninterested in the welfare of others making decisions for those others. Since people are worse at making decisions that prioritize others, this world is probably worse off. It’s best to have a strong connection between someone’s decisions and their ability to positively affect themselves and those they love. Rights like autonomy over one’s body, property, etc., allow individuals to take ownership of their own welfare in a way that’s welfare-maximizing for the world as a whole.[6]
This extends beyond the human rights you might be immediately thinking of. Rawlette brings up the example of individuals in a corporate setting who “own” a certain sphere of influence. Think of someone being in charge of a certain feature of a website, for example. The decision to give the worker ownership of the feature was probably openly made by maximizing utility for the company. But still, once that decision is made, others making decisions for the feature might be considered an infringement on the worker’s “right” to control the feature. Setting this norm allows the company to work more smoothly overall, even if in a specific instance someone thinks they could improve things by autonomously making changes in someone else’s sphere of influence.
Beyond respecting the rights of others, we should also make personal rules against selfish actions generally. It’s much easier to convince ourselves that an action is good if it benefits us. This becomes more relevant the more uncertain a situation is: the complexity gives us more opportunities to rationalize. But if we make a personal rule to never steal, for instance, we prevent ourselves from rationalizing a bad decision into something that seems good. These sorts of rules also force us to consider alternatives that we might not have thought of otherwise.
It’s also good for society to have these rules. If stealing makes everyone mad at you, you’ll have a self-interested reason not to do it. Your selfish motivations have been hijacked to force you to internalize harms to others. Even if stealing is the utilitarian best decision for someone at some given time, society should still condemn it ~universally to maintain a consistent incentive structure.
Conclusions
The best way to make decisions isn’t always the one that thinks exclusively about utilities. Even if your end goal is utility maximization, when you’re uncertain about outcomes, it’s best to prioritize trust, cooperation, rights, and broad rules against selfish actions. The rules aren’t given any direct moral weight in themselves; we’re still in act-utilitarian land, but they act as guidelines on how to arrive at the best outcome when knowledge or motivations aren’t perfect. With that being said, I am somewhat uncertain about all this. Yielding your moral considerations to common norms feels antithetical to rationalism and effective altruism if over-applied. I'm especially not confident about the uncertainty point, somewhat ironically.
If you want more of the specifics as to why organ-stealing exists in the domain of uncertainty where norms against murder and such should come into play, you can read my original post. The most important takeaways are that committing ahead of time to not stealing organs forces you to look for alternatives, and that both secrecy and discovery have their own major drawbacks. It’s easy to think of the utilitarian actions in these decisions as being at odds with patient rights. But the more nuanced view is that considering rights when making decisions is valuable even when thinking consequentially. And similarly, if you care about rights, that doesn’t preclude you from thinking about consequences.
I also think this sort of multi-level framing of utilitarianism is useful to keep in mind when having discussions about ethics. I genuinely believe most people are more or less act utilitarians in their latent criterion of rightness, but interpret ethical questions as pressing on their heavily rule-based decision procedure instead. For example, some research finds that 95% of people learn from consequences when making ethical decisions. But if you ask the average person why someone has a right not to get their organs stolen, I seriously doubt they will be willing to justify it as anything more than self-evident. People value consequences, but tend to think and speak in rules. In my observation, self-described utilitarians are more comfortable separating these levels, while less philosophically-minded people tend to strongly resist conceptualizing an act as instrumentally but not intrinsically valuable. This isn’t to say real deontologists don’t exist, but utilitarians should be a little more forgiving of people who say things like “equality is intrinsically valuable.” Most of the time, those sorts of statements should be read as normative claims about decision procedures, not value criteria.
Benthem's Bulldog recently responded to this, and while I broadly agree, I think he missed some nuance.
Rawlette cites Gigerenzer & Gaissmaier, 2011 here. I also found Hafenbrädl et al., 2016 and Ehrig & Schmidt, 2019, who have more recent and focused discussions of the topic. Maier et al., 2026 has a neat paper about how humans instinctively do this already, though there’s a lot of work on how moral decisions function differently from other decisions, so that likely doesn’t apply in this case. In fact, I strongly assume most people have the opposite problem from what we’re solving here and rely too strongly on heuristic rules or instincts when making moral decisions.
You could phrase this as simply having a strong prior towards common-sense norms being best, and that would be fine. It would be slightly different in framing, however; in the approach I’ve described, you can ignore whatever you believe the best action to be rather than change what you believe that best action is, but that distinction is a little murky.
... and also knows that you know that he doesn’t care about promises, and also knows that you know that he knows that you don’t care about promises, et cetera. This relies on arbitrarily deep common knowledge about the other person’s beliefs.
This story isn’t how it works in the real world. In reality, since no one is strictly a naive act-util decision maker, the problem never arises. You immediately have grounds to trust others as well as an incentive to provide grounds for others to trust you. But it’s still a cautionary tale against becoming a naive util decision maker.
There are legitimate defenses of paternalism. See Paul Bloom’s substack post on related topics.