A (small) critique of total utilitarianism

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness (or preference satisfaction or welfare). In fact if one can kill a billion people to create a billion and one, one is morally compelled to do so. And this is true for real people, not just thought experiment people - living people with dreams, aspirations, grudges and annoying or endearing quirks. To avoid causing extra pain to those left behind, it is better that you kill off whole families and communities, so that no one is left to mourn the dead. In fact the most morally compelling act would be to kill off the whole of the human species, and replace it with a slightly larger population.

We have many real world analogues to this thought experiment. For instance, it seems that there is only a small difference between the happiness of richer nations and poorer nations, while the first consume many more resources than the second. Hence to increase utility we should simply kill off all the rich, and let the poor multiply to take their place (continually bumping off any of the poor that gets too rich). Of course, the rich world also produces most of the farming surplus and the technology innovation, which allow us to support a larger population. So we should aim to kill everyone in the rich world apart from farmers and scientists - and enough support staff to keep these professions running (Carl Shulman correctly points out that we may require most of the rest of the economy as "support staff". Still, it's very likely that we could kill off a significant segment of the population - those with the highest consumption relative to their impact of farming and science - and still "improve" the situation).

Even if turns out to be problematic to implement in practice, a true total utilitarian should be thinking: "I really, really wish there was a way to do targeted killing of many people in the USA, Europe and Japan, large parts of Asia and Latin America and some parts of Africa - it makes me sick to the stomach to think that I can't do that!" Or maybe: "I really really wish I could make everyone much poorer without affecting the size of the economy - I wake up at night with nightmare because these people remain above the poverty line!"

I won't belabour the point. I find those actions personally repellent, and I believe that nearly everyone finds them somewhat repellent or at least did so at some point in their past. This doesn't mean that it's the wrong thing to do - after all, the accepted answer to the torture vs dust speck dilemma feels intuitively wrong, at least the first time. It does mean, however, that there must be very strong countervailing arguments to balance out this initial repulsion (maybe even a mathematical theorem). For without that... how to justify all this killing?

Hence for the rest of this post, I'll be arguing that total utilitarianism is built on a foundation of dust, and thus provides no reason to go against your initial intuitive judgement in these problems. The points will be:

  1. Bayesianism and the fact that you should follow a utility function in no way compel you towards total utilitarianism. The similarity in names does not mean the concepts are on similarly rigorous foundations.
  2. Total utilitarianism is neither a simple, nor an elegant theory. In fact, it is under-defined and arbitrary.
  3. The most compelling argument for total utilitarianism (basically the one that establishes the repugnant conclusion), is a very long chain of imperfect reasoning, so there is no reason for the conclusion to be solid.
  4. Considering the preferences of non-existent beings does not establish total utilitarianism.
  5. When considering competing moral theories, total utilitarianism does not "win by default" thanks to its large values as the population increases.
  6. Population ethics is hard, just as normal ethics is.

 

A utility function does not compel total (or average) utilitarianism

There are strong reasons to suspect that the best decision process is one that maximises expected utility for a particular utility function. Any process that does not do so, leaves itself open to be money pumped or taken advantage of. This point has been reiterated again and again on Less Wrong, and rightly so.

Your utility function must be over states of the universe - but that's the only restriction. The theorem says nothing further about the content of your utility function. If you prefer a world with a trillion ecstatic super-humans to one with a septillion subsistence farmers - or vice versa - then as long you maximise your expected utility, the money pumps can't touch you, and the standard Bayesian arguments don't influence you to change your mind. Your values are fully rigorous.

For instance, in the torture vs dust speck scenario, average utilitarianism also compels you to take torture, as do a host of other possible utility functions. A lot of arguments around this subject, that may implicitly feel to be in favour of total utilitarianism, turn out to be nothing of the sort. For instance, avoiding scope insensitivity does not compel you to total utilitarianism, and you can perfectly allow birth-death asymmetries or similar intuitions, while remaining an expected utility maximiser.

 

Total utilitarianism is not simple nor elegant, but is arbitrary

Total utilitarianism is defined as maximising the sum of everyone's individual utility function. That's a simple definition. But what are these individual utility functions? Do people act like expected utility maximisers? In a word... no. In another word... NO. In yet another word... NO!

So what are these utilities? Are they the utility that the individuals "should have"? According to what and who's criteria? Is it "welfare"? How is that defined? Is it happiness? Again, how is that defined? Is it preferences? On what scale? And what if the individual disagrees with the utility they are supposed to have? What if their revealed preferences are different again?

There are (various different) ways to start resolving these problems, and philosophers have spent a lot of ink and time doing so. The point remains that total utilitarianism cannot claim to be a simple theory, if the objects that it sums over are so poorly and controversially defined.

And the sum itself is a huge problem. There is no natural scale on which to compare utility functions. Divide one utility function by a billion, multiply the other by eπ, and they are still perfectly valid utility functions. In a study group at the FHI, we've been looking at various ways of combining utility functions - equivalently, of doing interpersonal utility comparisons (IUC). Turns out it's very hard, there seems no natural way of doing this, and a lot has also been written about this, concluding little. Unless your theory comes with a particular IUC method, the only way of summing these utilities is to do an essentially arbitrary choice for each individual before summing. Thus standard total utilitarianism is an arbitrary sum of ill defined, non-natural objects.

Why then is it so popular? Well, one reason is that there are models that make use of something like total utilitarianism to great effect. Classical economic theory, for instance, models everyone as perfectly rational expected utility maximisers. It gives good predictions - but it remains a model, with a domain of validity. You wouldn't conclude from that economic model that, say, mental illnesses don't exist. Similarly, modelling each life as having the same value and maximising expected lives saved is sensible and intuitive in many scenarios - but not necessarily all.

Maybe if we had a bit more information about the affected populations, we could use a more sophisticated model, such as one incorporating quality adjusted life years (QALY). Or maybe we could let other factors affect our thinking - what if we had to choose between saving a population of 1000 versus a population of 1001, of same average QALYs, but where the first set contained the entire Awá tribe/culture of 300 people, and the second is made up of representatives from much larger ethnic groups, much more culturally replaceable? Should we let that influence our decision? Well maybe we should, maybe we shouldn't, but it would be wrong to say "well, I would really like to save the Awá, but the model I settled on earlier won't allow me to, so I best follow the model". The models are there precisely to model our moral intuitions (the clue is in the name), not freeze them.

 

The repugnant conclusion is at the end of a flimsy chain

There is a seemingly sound argument for the repugnant conclusion, which goes some way towards making total utilitarianism plausible. It goes like this:

  1. Start with a population of very happy/utilitied/welfared/preference satisfied people.
  2. Add other people whose lives are worth living, but whose average "utility" is less than that of the initial population.
  3. Redistribute "utility" in an egalitarian way across the whole population, increasing the average a little as you do so (but making sure the top rank have their utility lowered).
  4. Repeat as often as required.
  5. End up with a huge population whose lives are barely worth living.

If all these steps increase the quality of the outcome (and it seems intuitively that they do), then the end state much be better than the starting state, agreeing with total utilitarianism. So, what could go wrong with this reasoning? Well, as seen before, the term "utility" is very much undefined, as is its scale - hence egalitarian is extremely undefined. So this argument is not mathematically precise, its rigour is illusionary. And when you recast the argument in qualitative terms, as you must, it become much weaker.

Going through the iteration, there will come a point when the human world is going to lose its last anime, its last opera, its last copy of the Lord of the Rings, its last mathematics, its last online discussion board, its last football game - anything that might cause more-than-appropriate enjoyment. At that stage, would you be entirely sure that the loss was worthwhile, in exchange of a weakly defined "more equal" society? More to the point, would you be sure that when iterating this process billions of times, every redistribution will be an improvement? This is a conjunctive statement, so you have to be nearly entirely certain of every link in the chain, if you want to believe the outcome. And, to reiterate, these links cannot be reduced to simple mathematical statements - you have to be certain that each step is qualitatively better than the previous one.

And you also have to be certain that your theory does not allow path dependency. One can take the perfectly valid position that "If there were an existing poorer population, then the right thing to do would be to redistribute wealth, and thus lose the last copy of Akira. However, currently there is no existing poor population, hence I would oppose it coming into being, precisely because it would result in the lose of Akira." You can reject this type of reasoning, and a variety of others that block the repugnant conclusion at some stage of the chain (the Stanford Encyclopaedia of Philosophy has a good entry on the Repugnant Conclusion and the arguments surrounding it). But most reasons for doing so already pre-suppose total utilitarianism. In that case, you cannot use the above as an argument for your theory.

 

Hypothetical beings have hypothetical (and complicated) things say to you

There is another major strand of argument for total utilitarianism, which claims that we owe it to non-existent beings to satisfy their preferences, that they would prefer to exist rather than remain non-existent, and hence we should bring them into existence. How does this argument fare?

First of all, it should be emphasised that one is free to accept or reject that argument without any fear of inconsistency. If one maintains that never-existent beings have no relevant preferences, then one will never stumble over a problem. They don't exist, they can't make decisions, they can't contradict anything. In order to raise them to the point where their decisions are relevant, one has to raise them to existence, in reality or in simulation. By the time they can answer "would you like to exist?", they already do, so you are talking about whether or not to kill them, not whether or not to let them exist.

But secondly, it seems that the "non-existent beings" argument is often advanced for the sole purpose of arguing for total utilitarianism, rather than as a defensible position in its own right. Rarely are its implication analysed. What would a proper theory of non-existent beings look like?

Well, for a start the whole happiness/utility/preference problem comes back with extra sting. It's hard enough to make a utility function out of real world people, but how to do so with hypothetical people? Is it an essentially arbitrary process (dependent entirely on "which types of people we think of first"), or is it done properly, teasing out the "choices" and "life experiences" of the hypotheticals? In that last case, if we do it in too much detail, we could argue that we've already created the being in simulation, so it comes back to the death issue.

But imagine that we've somehow extracted a utility function from the preferences of non-existent beings. Apparently, they would prefer to exist rather than not exist. But is this true? There are many people in the world who would prefer not to commit suicide, but would not mind much if external events ended their lives - they cling to life as a habit. Presumably non-existent versions of them "would not mind" remaining non-existent.

Even for those that would prefer to exist, we can ask questions about the intensity of that desire, and how it compares with their other desires. For instance, among these hypothetical beings, some would be mothers of hypothetical infants, leaders of hypothetical religions, inmates of hypothetical prisons, and would only prefer to exist if they could bring/couldn't bring the rest of their hypothetical world with them. But this is ridiculous - we can't bring the hypothetical world with them, they would grow up in ours - so are we only really talking about the preferences of hypothetical babies, or hypothetical (and non-conscious) foetuses?

If we do look at adults, bracketing the issue above, then we get some that would prefer that they not exist, as long as certain others do - or conversely that they not exist, as long as others also not exist. How should we take that into account? Assuming the universe infinite, any hypothetical being would exist somewhere. Is mere existence enough, or do we have to have a large measure or density of existence? Do we need them to exist close to us? Are their own preferences relevant - ie we only have a duty to bring into the world, those beings that would desire to exist in multiple copies everywhere? Or do we feel these have already "enough existence" and select the under-counted beings? What if very few hypothetical beings are total utilitarians - is that relevant?

On a more personal note, every time we make a decision, we eliminate a particular being. We can not longer be the person who took the other job offer, or read the other book at that time and place. As these differences accumulate, we diverge quite a bit from what we could have been. When we do so, do we feel that we're killing off these extra hypothetical beings? Why not? Should we be compelled to lead double lives, assuming two (or more) completely separate identities, to increase the number of beings in the world? If not, why not?

These are some of the questions that a theory of non-existent beings would have to grapple with, before it can become an "obvious" argument for total utilitarianism.

 

Moral uncertainty: total utilitarianism doesn't win by default

An argument that I have met occasionally is that while other ethical theories such as average utilitarianism, birth-death asymmetry, path dependence, preferences of non-loss of culture, etc... may have some validity, total utilitarianism wins as the population increases because the others don't scale in the same way. By the time we reach the trillion trillion trillion mark, total utilitarianism will completely dominate, even if we gave it little weight at the beginning.

But this is the wrong way to compare competing moral theories. Just as different people's utilities don't have a common scale, different moral utilities don't have a common scale. For instance, would you say that square-total utilitarianism is certainly wrong? This theory is simply total utilitarianism further multiplied by the population; it would correspond roughly to the number of connections between people. Or what about exponential-square-total utilitarianism? This would correspond roughly to the set of possible connections between people. As long as we think that exponential-square-total utilitarianism is not certainly completely wrong, then the same argument as above would show it quickly dominating as population increases.

Or what about 3^^^3 average utilitarianism - which is simply average utilitarianism, multiplied by 3^^^3? Obviously that example is silly - we know that rescaling shouldn't change anything about the theory. But similarly, dividing total utilitarianism by 3^^^3 shouldn't change anything, so total utilitarianism's scaling advantage is illusory.

As mentioned before, comparing different utility functions is a hard and subtle process. One method that seems to have surprisingly nice properties (to such an extent that I recommend always using as a first try) is to normalise the lowest possible attainable utility to zero, the highest attainable utility to one, multiply by the weight you give to the theory, and then add the normalised utilities together.

For instance, assume you equally valued average utilitarianism and total utilitarianism, giving them both weights of one (and you had solved all the definitional problems above). Among the choices you were facing, the worst outcome for both theories is an empty world. The best outcome for average utilitarianism would be ten people with an average "utility" of 100. The best outcome for total utilitarianism would be a quadrillion people with an average "utility" of 1. Then how would either of those compare to ten trillion people with an average utility of 60? Well, the normalised utility of this for the average utilitarian is 0.6, while for the total utilitarian it's also 60/100=0.6, and 0.6+0.6=1.2. This is better that the utility for the small world (1+10-9) or the large world (0.01+1), so it beats either of the extremal choices.

Extending this method, we can bring in such theories as exponential-square-total utilitarianism (probably with small weights!), without needing to fear that it will swamp all other moral theories. And with this normalisation (or similar ones), even small weights to moral theories such as "culture has some intrinsic value" will often prevent total utilitarianism from walking away with all of the marbles.

 

(Population) ethics is still hard

What is the conclusion? At Less Wrong, we're used to realising that ethics is hard, that value is fragile, that there is no single easy moral theory to safely program the AI with. But it seemed for a while that population ethics might be different - that there may be natural and easy ways to determine what to do with large groups, even though we couldn't decide what to do with individuals. I've argued strongly here that it's not the case - that population ethics remain hard, that we have to figure out what theory we want to have without access to easy shortcuts.

But in another way it's liberating. To those who are mainly total utilitarians but internally doubt that a world with infinitely many barely happy people surrounded by nothing but "muzak and potatoes" is really among the best of the best - well, you don't have to convince yourself of that. You may choose to believe it, or you may choose not to. No voice in the sky or in the math will force you either way. You can start putting together a moral theory that incorporates all your moral intuitions - those that drove you to total utilitarianism, and those that don't quite fit in that framework.

237 comments, sorted by
magical algorithm
Highlighting new comments since Today at 2:46 PM
Select new highlight date
Moderation Guidelinesexpand_more

For instance, it seems that there is only a small difference between the happiness of richer nations and poorer nations, while the first consume many more resources than the second. Hence to increase utility we should simply kill off all the rich, and let the poor multiply to take their place (continually bumping off any of the poor that gets too rich).

This empirical claim seems ludicrously wrong, which I find distracting from the ethical claims. Most people in rich countries (except for those unable or unwilling to work or produce kids who will) are increasing the rate of technological advance by creating demand for improved versions of products, paying taxes, contributing to the above-average local political cultures, and similar. Such advance dominates resource consumption in affecting the welfare of the global poor (and long-term welfare of future people). They make charitable donations or buy products that enrich people like Bill Gates and Warren Buffett who make highly effective donations, and pay taxes for international aid.

The scientists and farmers use thousands of products and infrastructure provided by the rest of society, and this neglects industry, resource extraction, and the many supporting sectors that make productivity in primary and secondary production so far (accountants, financial markets, policing, public health, firefighting...). Even "frivolous" sectors like Hollywood generate a lot of consumer surplus around the world (they watch Hollywood movies in sub-Saharan Africa), and sometimes create net rewards for working harder to afford more luxuries (sometimes they may encourage leisure too much by a utilitarian standard).

Regarding other points:

fact that you should follow a utility function in no way compel you towards total utilitarianism

Yes, this is silly equivocation exacerbated by the use of similar-sounding words for several concepts, and one does occasionally see people making this error.

interpersonal utility comparisons (IUC)

The whole piece assumes preference utilitarianism, but much of it also applies to hedonistic utilitarianism: you need to make seemingly-arbitrary choices in interpersonal happiness/pleasure comparison as well.

When considering competing moral theories, total utilitarianism does not "win by default" thanks to its large values as the population increases.

I agree.

The most compelling argument for total utilitarianism (basically the one that establishes the repugnant conclusion), is a very long chain of imperfect reasoning, so there is no reason for the conclusion to be solid. Considering the preferences of non-existent beings does not establish total utilitarianism.

Maybe just point to the Stanford Encyclopedia of Philosophy entry and a few standard sources on this? This has been covered very heavily by philosophers, if not ad nauseam.

Whatever the piece assumes, I don't think it's preference utilitarianism because then the first sentence doesn't make sense:

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness.

Assuming most people have a preference to go on living, as well as various other preferences for the future, then killing them would violate all these preferences, and simply creating a new, equally happy being would still leave you with less overall utility, because all the unsatisfied preferences count negatively. (Or is there a version of preference utilitarianism where unsatisfied preferences don't count negatively?) The being would have to be substantially happier, or you'd need a lot more beings to make up for the unsatisfied preferences caused by the killing. Unless we're talking about beings that live "in the moment", where their preferences correspond to momentary hedonism.

Peter Singer wrote a chapter on killing and replaceability in Practical Ethics. His view is prior-existence, not total preference utilitarianism, but the points on replaceability apply to both.

Maybe just point to the Stanford Encyclopedia of Philosophy entry and a few standard sources on this? This has been covered very heavily by philosophers, if not ad nauseam.

Will add a link. But I haven't yet seen my particular angle of attack on the repugnant conclusion, and it isn't in the Stanford Encyclopaedia. The existence/non-existence seems to have more study, though.

There is no natural scale on which to compare utility functions. [...] Unless your theory comes with a particular [interpersonal utility comparison] method, the only way of summing these utilities is to do an essentially arbitrary choice for each individual before summing. Thus standard total utilitarianism is an arbitrary sum of ill defined, non-natural objects.

This, in my opinion, is by itself a decisive argument against utilitarianism. Without these ghostly "utilities" that are supposed to be measurable and comparable interpersonally, the whole concept doesn't even being to make sense. And yet the problem is commonly ignored routinely and nonchalantly, even here, where people pride themselves on fearless and consistent reductionism.

Note that the problem is much more fundamental than just the mathematical difficulties and counter-intuitive implications of formal utilitarian theories. Even if there were no such problems, it would still be the case that the whole theory rests on an entirely imaginary foundation. Ultimately, it's a system that postulates some metaphysical entities and a categorical moral imperative stated in terms of the supposed state of these entities. Why would we privilege that over systems that postulate metaphysical entities and associated categorical imperatives of different kinds, like e.g. traditional religions?

(If someone believes that there is a way how these interpersonally comparable utilities could actually be grounded in physical reality, I'd be extremely curious to hear it.)

(If someone believes that there is a way how these interpersonally comparable utilities could actually be grounded in physical reality, I'd be extremely curious to hear it.)

I asked about this before in the context of one of Julia Galef's posts about utilitarian puzzles and received several responses. What is your evaluation of the responses (personally, I was very underwhelmed)?

The only reasonable attempt at a response in that sub-thread is this comment. I don't think the argument works, though. The problem is not just disagreement between different people's intuitions, but also the fact that humans don't do anything like utility comparisons when it comes to decisions that affect other people. What people do in reality is intuitive folk ethics, which is basically virtue ethics, and has very little concern with utility comparisons.

That said, there are indeed some intuitions about utility comparison, but they are far too weak, underspecified, and inconsistent to serve as basis for extracting an interpersonal utility function, even if we ignore disagreements between people.

Intuitive utilitarian ethics are very helpful in everyday life.

There is the oft-repeated anecdote of the utilitarian moral philosopher weighing up whether to accept a job at Columbia. It would get more money, but it would uproot his family, but it might help his career... familiar kind of moral dilemma. Asking his colleague for advice, he got told "Just maximise total utility." "Come on," he is supposed to have replied, "this is serious!"

I struggle to think of any moral dilemma I have faced where utilitarian ethics even provide a practical framework for addressing the problem, let alone a potential answer.

That anecdote is about a decision theorist, not a moral philosopher. The dilemma you describe is a decision theoretic one, not a moral utilitarian one.

Writing out costs and benefits is a technique that is sometimes helpful.

Sure, but "costs" and "benefits" are themselves value-laden terms, which depend on the ethical framework you are using. And then comparing the costs and the benefits is itself value-laden.

In other words, people using non-utilitarian ethics can get plenty of value out of writing down costs and benefits. And people using utilitarian ethics don't necessarily get much value (doesn't really help the philosopher in the anecdote). This is therefore not an example of how utilitarian ethics are useful.

Writing down costs and benefits is clearly an application of consequentialist ethics, unless things are so muddied that any action might be an example of any ethic. Consequentialist ethics need not be utilitarian, true, but they are usually pretty close to utilitarian. Certainly closer to utilitarianism than to virtue ethics.

Writing down costs and benefits is clearly an application of consequentialism ethics.

No, because "costs" and "benefits" are value-laden terms.

Suppose I am facing a standard moral dilemma; should I give my brother proper funerary rites, even though the city's ruler has forbidden it. So I take your advice and write down costs and benefits. Costs - breaching my duty to obey the law, punishment for me, possible reigniting of the city's civil war. Benefits - upholding my duty to my family, proper funeral rites for my brother, restored honour. By writing this down I haven't committed to any ethical system, all I've done is clarify what's at stake. For example, if I'm a deontologist, perhaps this helps clarify that it comes down to duty to the law versus duty to my family. If I'm a virtue ethicist, perhaps this shows it's about whether I want to be the kind of person who is loyal to their family above tawdry concerns of politics, or the kind of person who is willing to put their city above petty personal concerns. This even works if I'm just an egoist with no ethics; is the suffering of being imprisoned in a cave greater or less than the suffering I'll experience knowing my brother's corpse is being eaten by crows?

Ironically, the only person this doesn't help is the utilitarian, because he has absolutely no way of comparing the costs and the benefits - "maximise utility" is a slogan, not a procedure.

What are you arguing here? First you argue that "just maximize utility" is not enough to make a decision. This is of course true, since utilitarianism is not a fully specified theory. There are many different utilitarian systems of ethics, just as there are many different deontological ethics and many different egoist ethics.

Second you are arguing that working out the costs and benefits is not an indicator of consequentialism. Perhaps this is not perfectly true, but if you follow these arguments to their conclusion then basically nothing is an indicator of any ethical system. Writing a list of costs and benefits, as these terms are usually understood, focuses one's attention on the consequences of the action rather than the reasons for the action (as the virtue ethicists care about) or the rules mandating or forbidding an action (as the deontologists care about). Yes, the users of different ethical theories can use pretty much any tool to help them decide, but some tools are more useful for some theories because they push your thinking into the directions that theory considers relevant.

Are you arguing anything else?

I am thinking about petty personal disputes, say if one person finds something that another person does annoying. A common gut reaction is to immediately start staking territory about what is just and what is virtuous and so on, while the correct thing to do is focus on concrete benefits and costs of actions. The main reason this is better is not because it maximizes utility but because it minimizes argumentativeness.

Another good example is competition for a resource. Sometimes one feels like one deserves a fair share and this is very important, but if you have no special need for it, nor are there significant diminishing marginal returns, then it's really not that big of a deal.

In general, intuitive deontological tendencies can be jerks sometimes, and utilitarianism fights that.

Thanks for the link, I am very underwhelmed too.

If I understand it correctly, one suggestion is equivalent to choosing some X, and re-scaling everyone's utility function so that X has value 1. Obvious problem is the arbitrary choice of X, and the fact that in some people's original scale X may have positive, negative, or zero value.

The other suggestion is equivalent to choosing a hypothetical person P with infinite empathy towards all people, and using the utility function of P as absolute utility. I am not sure about this, but seems to me that the result depends on P's own preferences, and this cannot be fixed because without preferences there could be no empathy.

And yet the problem is commonly ignored routinely and nonchalantly, even here, where people pride themselves on fearless and consistent reductionism.

Yes. To be honest it looks like local version of reductionism takes the 'everything is reducible' in declarative sense, declaring that concepts it uses are reducible regardless of their reducibility.

Thanks! That's spot on. It's what I think much of those 'utility functions' here are. Number of paperclips in the universe, too. Haven't seen anything like that reduced to formal definition of any kind.

The way humans actually decide on actions, is by evaluating the world-difference that the action causes in world-model, everything being very partial depending to available time. The probabilities are rarely possible to employ in the world model because of the combinatorial space exploding real hard. (also, Bayesian propagation on arbitrary graphs is np-complete, in very practical way of being computationally expensive). Hence there isn't some utility function deep inside governing the choices. Doing the best is mostly about putting limited computing time to best use.

Then there's some odd use of abstractions - like, every agent can be represented with utility function therefore whatever we talk about utilities is relevant. Never mind that this utility function is trivial 1 for doing what agent chooses 0 otherwise and everything just gets tautological.

(If someone believes that there is a way how these interpersonally comparable utilities could actually be grounded in physical reality, I'd be extremely curious to hear it.)

I wonder if I am misunderstanding what you are asking, because interpersonal utility comparison seems like an easy thing that people do every day, using our inborn systems for sympathy and empathy.

When I am trying to make a decision that involves the conflicting desires of myself and another person; I generally use empathy to put myself in their shoes and try to think about desires that I have that are probably similar to theirs. Then I compare how strong those two desires of mine are and base my decision on that. Now, obviously I don't make all ethical decisions like that, there are many where I just follow common rules of thumb. But I do make some decisions in this fashion, and it seems quite workable, the more fair-minded of my acquaintances don't really complain about it unless they think I've made a mistake. Obviously it has scaling problems when attempting to base any type of utilitarian ethics on it, but I don't think they are insurmountable.

Now, of course you could object that this method is unreliable, and ask whether I really know for sure if other people's desires are that similar to mine. But this seems to me to just be a variant of the age-old problem of skepticism and doesn't really deserve any more attention than the possibility that all the people I meet are illusions created by an evil demon. It's infinitesimally possible that everyone I know doesn't really have mental states similar to mine at all, that in fact they are all really robot drones controlled by a non-conscious AI that is basing their behavior on a giant lookup table. But it seems much more likely that other people are conscious human beings with mental states similar to mine that can be modeled and compared via empathy, and that this allows me to compare their utilities.

In fact, it's hard to understand how empathy and sympathy could have evolved if it they weren't reasonably good at interpersonal utility comparison. If interpersonal utility comparison was truly impossible then anyone who tried to use empathy to inform their behavior towards others would end up being disastrously wrong at figuring out how to properly treat others, find themselves grievously offending the rest of their tribe, and would hence likely have their genes for empathy selected against. It seems like if interpersonal utility comparison was impossible humans would have never evolved the ability or desire to make decisions based on empathy.

I am also curious as to why you refer to to utility as "ghostly." It seems to me that utility is commonly defined as the sum of the various desires and feelings that people have. Desires and feelings are computations and other processes in our brains, which are very solid real physical objects. So it seems like utility is at least as real as software. Of course, it's entirely possible that you are using the word "utility" to refer to a slightly different concept than I am and that is where my confusion is coming from.

This, in my opinion, is by itself a decisive argument against utilitarianism.

You mean against preference-utilitarianism.

The vast majority of utilitarians I know are hedonistic utilitarians, where this criticism doesn't apply at all. (For some reason LW seems to be totally focused on preference-utilitarianism, as I've noticed by now.) As for the criticism itself: I agree! Preference-utiltiarians can come up with sensible estimates and intuitive judgements, but when you actually try to show that in theory there is one right answer, you just find a huge mess.

I agree. I'm fairly confident that, within the next several decades, we will have the technology to accurately measure and sum hedons and that hedonic utilitarianism can escape the conceptual problems inherent in preference utilitarianism. On the other hand, I do not want to maximize (my) hedons (for these kinds of reasons, among others).

we will have the technology to accurately measure and sum hedons

Err...what? Technology will tell you things about how brains (and computer programs) vary, but not which differences to count as "more pleasure" or "less pleasure." If evaluations of pleasure happen over 10x as many neurons is there 10x as much pleasure? Or is it the causal-functional role pleasure plays in determining the behavior of a body? What if we connect many brains or programs to different sorts of virtual bodies? Probabilistically?

A rule to get a cardinal measure of pleasure across brains is going to require almost as much specification as a broader preference measure. Dualists can think of this as guesstimating "psychophysical laws" and physicalists can think of it as seeking reflective equilibrium in our stances towards different physical systems, but it's not going to be "read out" of neuroscience without deciding a bunch of evaluative (or philosophy of mind) questions.

but it's not going to be "read out" of neuroscience without deciding a bunch of evaluative (or philosophy of mind) questions.

Sure, but I don't think we can predict that there will be a lot of room for deciding those philosophy of mind questions whichever way one wants to. One simply has to wait for the research results to come in. With more data to constrain the interpretations, the number and spread of plausible stable reflective equilibria might be very small.

I agree with Jayson that it is not mandatory or wise to maximize hedons. And that is because hedons are not the only valuable things. But they do constitute one valuable category. And in seeking them, the total utilitarians are closer to the right approach than the average utilitarians (I will argue in a separate reply).

I'm fairly confident that, within the next several decades, we will have the technology to accurately measure and sum hedons

OK, I've got to ask: what's your confidence based in, in detail? It's not clear to me that "sum hedons" even means anything.

Why do you believe that interpersonal comparison of pleasure is straightforward? To me this doesn't seem to be the case.

Is intrapersonal comparison possible? Personal boundaries don't matter for hedonistic utilitarianism, they only matter insofar as you may have spatio-temporally connected clusters of hedons (lives). The difficulties in comparison seem to be of an empirical nature, not a fundamental one (unlike the problems with preference-utilitarianism). If we had a good enough theory of consciousness, we could quantitatively describe the possible states of consciousness and their hedonic tones. Or not?

One common argument against hedonistic utiltiarianism is that there are "different kinds of pleasures", and that they are "incommensurable". But if that we're the case, it would be irrational to accept a trade-off of the lowest pleasure of one sort for the highest pleasure of another sort, and no one would actually claim that. So even if pleasures "differ in kind", there'd be an empirical trade-off value based on how pleasant the hedonic states actually are.