Some people think that they have unbounded utility functions. This isn't necessarily crazy, but it presents serious challenges to conventional decision theory. I think it probably leads to abandoning probability itself as a representation of uncertainty (or at least any hope of basing decision theory on such probabilities). This may seem like a drastic response, but we are talking about some pretty drastic inconsistencies.
This result is closely related to standard impossibility results in infinite ethics. I assume it has appeared in the philosophy literature, but I couldn't find it in the SEP entry on the St. Petersburg paradox so I'm posting it here. (Even if it's well known, I want something simple to link to.)
(ETA: this argument is extremely similar to Beckstead and Thomas' argument against Recklessness in A paradox for tiny probabilities and enormous values. The main difference is that they use transitivity +"recklessness" to get a contradiction whereas I argue directly from "non-timidity." I also end up violating a dominance principle which seems even more surprising to violate, but at this point it's kind of like splitting hairs. I give a slightly stronger set of arguments in Better impossibility results for unbounded utilities.)
Weak version
We'll think of preferences as relations over probability distributions over some implicit space of outcomes (and we'll identify outcomes with the constant probability distribution). We'll show that there is no relation which satisfies three properties: Antisymmetry, Unbounded Utilities, and Dominance.
Note that we assume nothing about the existence of an underlying utility function. We don't even assume that the preference relation is complete or transitive.
The properties
Antisymmetry: It's never the case that both and .
Unbounded Utilities: there is an infinite sequence of outcomes each "more than twice as good" as the last.^{[1]} More formally, there exists an outcome such that:
- for every .
- ^{[2]}
That is, is not as good as a chance of , which is not as good as a chance of , which is not as good as a chance of ... This is nearly the weakest possible version of unbounded utilities.^{[3]}
Dominance: let and be sequences of lotteries, and be a sequence of probabilities that sum to 1. If for all , then .
Inconsistency proof
Consider the lottery
We can write as a mixture:
By definition . And for each , Unbounded Utilities implies that . Thus Dominance implies , contradicting Antisymmetry.
How to avoid the paradox?
By far the easiest way out is to reject Unbounded Utilities. But that's just a statement about our preferences, so it's not clear we get to "reject" it.
Another common way out is to assume that any two "infinitely good" outcomes are incomparable, and therefore to reject Dominance.^{[4]} This results in being indifferent to receiving $1 in every world (if the expectation is already infinite), or doubling the probability of all good worlds, which seems pretty unsatisfying.
Another option is to simply ignore small probabilities, which again leads to rejecting even the finite version of Dominance---sometimes when you mix together lotteries something will fall below the "ignore it" threshold leading the direction of your preference to reverse. I think this is pretty bizarre behavior, and in general ignoring small probabilities is much less appealing than rejecting Unbounded Utilities.
All of these options seem pretty bad to me. But in the next section, we'll show that if the unbounded utilities are symmetric---if there are both arbitrarily good and arbitrarily bad outcomes---then things get even worse.
Strong version
I expect this argument is also known in the literature; but I don't feel like people around LW usually grapple with exactly how bad it gets.
In this section we'll show there is no relation which satisfies three properties: Antisymmetry, Symmetric Unbounded Utilities, and Weak Dominance.
(ETA: actually I think that even with only positive utilities you already violate something very close to Weak Dominance, which Beckstead and Thomas call Prospect-Outcome dominance. I find this version of Weak Dominance slightly more compelling, but Symmetric Unbounded Utilities is a much stronger assumption than Unbounded Utilities or non-Timidity, so it's probably worth being aware of both versions. In a footnote^{[5]} I also define an even weaker dominance principle that we are forced to violate.)
The properties
Antisymmetry: It's never the case that both and .
Symmetric Unbounded Utilities. There is an infinite sequence of outcomes each of which is "more than twice as important" as the last but with opposite sign. More formally, there is an outcome such that:
- For every even :
- For every odd :
That is, a certainty of is outweighed by a chance of , which is outweighed by a chance of , which is outweighed by a chance of ....
Weak Dominance.^{[5]} For any outcome , any sequence of lotteries , and any sequence of probabilities that sum to 1:
- If for every , then .
- If for every , then .
Inconsistency proof
Now consider the lottery
We can write as the mixture:
By Unbounded Utilities each of these terms is . So by Weak Dominance,
But we can also write as the mixture:
By Unbounded Utilities each of these terms is . So by Weak Dominance . This contradicts Antisymmetry.
Now what?
As usual, the easiest way out is to abandon Unbounded Utilities. But if that's just the way you feel about extreme outcomes, then you're in a sticky situation.
You could allow for unbounded utilities as long as they only go in one direction. For example, you might be open to the possibility of arbitrarily bad outcomes but not the possibility of arbitrarily good outcomes.^{[6]} But the asymmetric version of unbounded utilities doesn't seem very intuitively appealing, and you still have to give up the ability to compare any two infinitely good outcomes (violating Dominance).
People like talking about extensions of the real numbers, but those don't help you avoid any of the contradictions above. For example, if you want to extend to a preference order over hyperreal lotteries, it's just even harder for it to be consistent.
Giving up on Weak Dominance seems pretty drastic. At that point you are talking about probability distributions, but I don't think you're really using them for decision theory---it's hard to think of a more fundamental axiom to violate. Other than Antisymmetry, which is your other option.
At this point I think the most appealing option, for someone committed to unbounded utilities, is actually much more drastic: I think you should give up on probabilities as an abstraction for describing uncertainty, and should not try to have a preference relation over lotteries at all.^{[7]} There are no ontologically fundamental lotteries to decide between, so this isn't necessarily so bad. Instead you can go back to talking directly about preferences over uncertain states of affairs, and build a totally different kind of machinery to understand or analyze those preferences.
ETA: replacing dominance
Since writing the above I've become more sympathetic to violations of Dominance and even Weak Dominance---it would be pretty jarring to give up on them, but I can at least imagine it. I still think violating "Very Weak Dominance"^{[5]} is pretty bad, but I don't think it captures the full weirdness of the situation.
So in this section I'll try to replace Weak Dominance by a principle I find even more robust: if I am indifferent between and any of the lotteries , then I'm also indifferent between X and any mixture of the lotteries . This isn't strictly weaker than Weak Dominance, but violating it feels even weirder to me. At any rate, it's another fairly strong impossibility result constraining unbounded utilities.
The properties
We'll work with a relation over lotteries. We write if both and . We write if but not . We'll show that can't satisfy four properties: Transitivity, Intermediate mixtures, Continuous symmetric unbounded utilities, and Indifference to homogeneous mixtures.
Intermediate mixtures. If , then .
Transitivity. If and then .
Continuous symmetric unbounded utilities. There is an infinite sequence of lotteries each of which is "exactly twice as important" as the last but with opposite sign. More formally, there is an outcome such that:
- For every even :
- For every odd :
That is, a certainty of is exactly offset by a chance of , which is exactly offset by a chance of , which is exactly offset by a chance of ....
Intuitively, this principle is kind of like symmetric unbounded utilities, but we assume that it's possible to dial down each of the outcomes in the sequence (perhaps by mixing it with ) until the inequalities become exact equalities.
Homogeneous mixtures. Let be an outcome, , a sequence of lotteries, and be a sequence of probabilities summing to 1. If for all , then .
Inconsistency proof
Consider the lottery
We can write as the mixture:
By Unbounded Utilities each of these terms is . So by homogeneous mixtures,
But we can also write as the mixture:
By Unbounded Utilities each of these terms other than the first is . So by Homogenous Mixtures, the combination of all terms other than the first is . Together with the fact that , Intermediate Mixtures and Transitivity imply . But that contradicts .
- ^{^}
Note that we could replace "more than twice as good" with "at least 0.00001% better" and obtain exactly the same result. You may find this modified version of the principle more appealing, and it is closer to non-timidity as defined in Beckstead and Thomas. Note that the modified principle implies the original by applying transitivity 100000 times, but you don't actually need to apply transitivity to get a contradiction, you can just apply Dominance to a different mixture.
- ^{^}
You may wonder why we don't just write . If we did this, we'd need to introduce an additional assumption that if , . This would be fine, but it seemed nicer to save some symbols and make a slightly weaker assumption.
- ^{^}
The only plausibly-weaker definition I see is to say that there are outcomes and an infinite sequence such that for all : . If we replaced the with then this would be stronger than our version, but with the inequality it's not actually sufficient for a paradox.
To see this, consider a universe with three outcomes and a preference order that always prefers lotteries with higher probability of and breaks ties using by preferring a higher probability of . This satisfies all of our other properties. It satisfies the weaker version of the axiom by taking for all , and it wouldn't be crazy to say that it has "unbounded" utilities. - ^{^}
For realistic agents who think unbounded utilities are possible, it seems like they should assign positive probability to encountering a St. Petersburg paradox such that all decisions have infinite expected utility. So this is quite a drastic thing to give up on. See also: Pascal's mugging.
- ^{^}
I find this principle pretty solid, but it's worth noting that the same inconsistency proof would work for the even weaker "Very Weak Dominance": for any pair of outcomes with , and any sequence of lotteries each strictly better than , any mixture of the should at least be strictly better than !
- ^{^}
Technically you can also violate Symmetric Unbalanced Utility while having both arbitrarily good and arbitrarily bad outcomes, as long as those outcomes aren't comparable to one another. For example, suppose that worlds have a real-valued amount of suffering and a real-valued amount of pleasure. Then we could have a lexical preference for minimizing expected suffering (considering all worlds with infinite expected suffering as incomparable), and try to maximize pleasure only as a tie-breaker (considering all worlds with infinite expected pleasure as incomparable).
- ^{^}
Instead you could keep probabilities but abandon infinite probability distributions. But at this point I'm not exactly sure what unbounded utilities means---if each decision involves only finitely many outcomes, then in what sense do all the other outcomes exist? Perhaps I may face infinitely many possible decisions, but each involves only finitely many outcomes? But then what am I to make of my parent's decisions while raising me, which affected my behavior in each of those infinitely many possible decisions? It seems like they face an infinite mixture of possible outcomes. Overall, it seems to me like giving up on infinitely big probability distributions implies giving up on the spirit of unbounded utilities, or else going down an even stranger road.
I am not a fan of unbounded utilities, but it is worth noting that most (all?) the problems with unbounded utilties are actually a problem with utility functions that are not integrable with respect to your probabilities. It feels basically okay to me to have unbounded utilities as long as extremely good/bad events are also sufficiently unlikely.
The space of allowable probability functions that go with an unbounded utility can still be closed under finite mixtures and conditioning on positive probability events.
Indeed, if you think of utility functions as coming from VNM, and you a space of lotteries closed under finite mixtures but not arbitrary mixtures, I think there are VNM preferences that can only correspond to unbounded utility functions, and the space of lotteries is such that you can't make St. Petersburg paradoxes. (I am guessing, I didn't check this.)
I think bounded functions are the only computable functions that are integrable WRT the universal semi-measure. I think this is equivalent to de Blanc 2007?
The construction is just the obvious one: for any unbounded computable utility function, any universal semi-measure must assign reasonable probability to a St Petersburg game for that utility function (since we can construct a computable St Petersburg game by picking a utility randomly then looping over outcomes until we find one of at least the desired utility).
I think this argument is cool, and I appreciate how distilled it is.
Basically just repeating what Scott said but in my own tongue: this argument leaves open the option of denying that (epistemic) probabilities are closed under countable combination, and deploying some sort of "leverage penalty" that penalizes extremely high-utility outcomes as extremely unlikely a priori.
I agree with your note that the simplicitly prior doesn't implement leverage penalties. I also note that I'm pretty uncertain myself about how to pull off leverage penalties correctly, assuming they're a good idea (which isn't clear to me).
I note further that the issue as I see it arises even when all utilities are finite, but some are ("mathematically", not merely cosmically) large (where numbers like 10^100 are cosmically large, and numbers like 3^^^3 are mathematically large). Like, why are our actions not dominated by situations where the universe is mathematically large? When I introspect, it doesn't quite feel like the answer is "because we're certain it isn't", nor "because utility maxes out at the cosmological scale", but rather something more like "how would you learn that there may or may not be 3^^^3 hap... (read more)
TL;DR: I think that the discussion in this post is most relevant when we talk about the utility of whole universes. And for that purpose, I think a leverage penalty doesn't make sense.
A leverage penalty seems more appropriate for saying something like "it's very unlikely that my decisions would have such a giant impact," but I don't think that should be handled in the utility function or decision theory.
Instead, I'd say: if it's possible to have "pivotal" decisions that affect 3^^^3 people, then it's also possible to have 3^^^3 people in "normal" situations all making their separate (correlated) decisions, eating 3^^^3 sandwiches, and so the stakes of everything are similarly mathematically big.
I think that if utilities are large but bounded, then I feel like everything "adds up to normality"---if there is a way to get 3^^^3 utility, it seems like "maximize option value, figure out what's going on, stay sane" is a reasonable bet for maximizing EV (e.g. by maximizing probability of the great outcome).
Intuitively, this also seems like what you should end up doing even if... (read more)
I'm not particulalry enthusiastic about "artificial leverage penalties" that manually penalize the hypothesis you can get 3^^^3 happy people by a factor of 1/3^^^3 (and so insofar as that's what you're saying, I agree).
From my end, the core of my objection feels more like "you have an extra implicit assumption that lotteries are closed under countable combination, and I'm not sold on that." The part where I go "and maybe some sufficiently naturalistic prior ends up thinking long St. Petersburg games are ultimately less likely than they are simple???" feels to me more like a parenthetical, and a wild guess about how the weakpoint in your argument could resolve.
(My guess is that you mean something more narrow and specific by "leverage penalty" than I did, and that me using those words caused confusion. I'm happy to retreat to a broader term, that includes things like "big gambles just turn out not to unbalance naturalistic reasoning when you're doing it properly (eg. b/c finding-yourself-in-the-universe correctly handles this sort of thing somehow)", if you have one.)
(My guess is that part of the ... (read more)
My formal argument is even worse than that: I assume you have preferences over totally arbitrary probability distributions over outcomes!
I don't think this is unlisted though---right at the beginning I said we were proving theorems about a preference ordering < defined over the space of probability distributions over a space of outcomes Ω. I absolutely think it's plausible to reject that starting premise (and indeed I suggest that someone with "unbounded utilities" ought to reject this premise in an even more dramatic way).
It seems to me that our actual situation (i.e. my actual subjective distribution over possible worlds) is divergent in the same way as the St Petersburg lottery, at least with respect to quantiti... (read more)
Ok, cool, I think I see where you're coming from now.
Fair! To a large degree, I was just being daft. Thanks for the clarification.
I think this is a good point, and I hadn't had this thought quite this explicitly myself, and it shifts me a little. (Thanks!)
(I'm not terribly sold on this point myself, but I agree that it's a crux of the matter, and I'm sympathetic.)
This might be where we part ways? I'm not sure. A bunch of my guesses do kinda look like things you might describe as "preferences not being defined over probability distributions" (eg, "utility is a number, not a function"). But simultaneously, I feel solid in my ability to use probabliity distributions and utility functions in day-to-day reasoning problems after I've chunked the world into a small finite number of possible acti... (read more)
The proof doesn't run for me. The only way I know to be able to rearrange the terms in a infinite series is if the starting starting series converges and the resultant series converges. The series doesn't fullfill the condition so I am not convinced the rewrite is a safe step.
I am a bit unsure about my maths so I am going to hyberbole the kind of flawed logic I read into the proof. Start with series that might not converge 1+1+1+1+1+1... (oh it indeed blatantly diverges) then split each term to have a non-effective addition (1+0)+(1+0)+(1+0)+(1+0)... . Blatantly disregard safety rules about paranthesis messing with series and just treat them as paranthesis that follow familiar rules 1+0+1+0+1+0+1+0+1... so 1+1+1+1... is not equal to itself. (unsafe step leads to non-sense)
With converging series it doesn't matter whether we get "twice as fast" to the limit but the "rate of ascension" might matter to whatever analog a divergent series would have to a value.
Maybe I'm just a less charitable person - it seems very easy to me for someone to say the words "I have unbounded utility" without actually connecting any such referent to their decision-making process.
We can show that there's a tension between that verbal statement and the basic machinery of decision-making, and also illustrates how the practical decision-making process people use every day doesn't act like expected utilities diverge.
And I think the proper response to seeing something like this happen to you is definitely not to double down on the verbal statement that sounded good. It's to stop and think very skeptically about whether this verbal statement fits with what you can actually ask of reality, and what you might want to ask for that you can actually get. (I've written too many posts about why it's the wrong move to want an AI to "just maximize my utility function." Saying that you want to be modeled as if you have unbounded utility [of this sort that lets you get divergent EV] is the same order of mistake.)
If you think people can make verbal statements that are "not up for grabs," this probably seems like gross uncharitableness.
I'm a recent proponent of hyperreal utilities. I totally agree that hyperreals don't solve issues with divergent / St. Petersburg-style lotteries. I just think hyperreals are perfect for describing and comparing potentially-infinite-utility universes, though not necessarily lotteries over those universes. (This doesn't contradict Paul; I'm just clarifying.)
Separately, while cases like this do make it feel like we "should give up on probabilities as an abstraction for describing uncertainty," this conclusion makes me feel quite nihilistic about decision-under-uncertainty; I will be utterly shocked if "a totally different kind of machinery to understand or analyze those preferences" is satisfactory.
The examples in this post all work by not just having divergent sums but unbounded single elements. When I look at this, my immediate takeaway is that we need a model that includes time. Outcomes should be of the form (x,t)∈R×N, where t indicates at which timestep they happen.
You are then allowed to look at infinite sequences (xi,ti),(xi+1,ti+1),... iff the ti are strictly increasing. The sums ∑xi do not need to converge, but there does need to be a global bound for all individual utilities, i.e. ∃B∈R that upper-bounds all xi.
This avoids all the problems i... (read more)
Note: I've turned on two-axis voting for this post to see if it's helpful for this kind of discussion.
Relevant comment from the sequences (I had this in mind when writing parts of the OP but didn't remember who wrote it, and failed to recognize the link because it was about Newcomb's problem):
... (read more)(Note: I've edited this comment a lot since first posting it, mostly small corrections, improving the definition of the order to be better behaved and adding more points.)
I think orders which satisfy (Symmetric) Unbounded Utilities but not (Weak) Dominance or Homogeneous Mixtures can be relatively nice, so we shouldn't be too upset about giving up (Weak) Dominance and Homogenous Mixtures. Basically no nice order will be sensitive to the kinds of probability rearrangements you've done (which we can of course conclude from your results, if we want "nice" to ... (read more)
Maybe some of the problem is coming from trying to extend dominance to infinite combinations of lotteries. If we're saying that the utility function is the thing that witnesses some coherence in the choices we make between lotteries, maybe it makes sense to ask for choices between finite combinations of lotteries but not infinite ones? Any choice we actually make, we end up making by doing some "finitary" sort of computation (not sure what this really means, if anything), and perhaps in particular it's always understandable as a choice between finite lotte... (read more)
Why are these unsatisfying? Intuitively, if I already have infinite money (in expectation) then why should I care about getting to infinite + 1 or infinite x 2 money?
The point you mention about all decisions having infinite utility in exp... (read more)
If I understand correctly, this argument also appeared in Eliezer Yudkowsky's post "The Lifespan Dilemma", which itself credits one of Wei Dai's comment here. The argument given in The Lifespan Dilemma is essentially identical to the argument in Beckstead and Thomas' paper.
I spent some time trying to fight these results, but have failed!
Specifically, my intuition said we should just be able to look at the flattened distributions-over-outcomes. Then obviously the rewriting makes no difference, and the question is whether we can still provide a reasonable decision criterion when the probabilities and utilities don't line up exactly. To do so we need some defined order or limiting process for comparing these infinite lotteries.
My thought was to use something like "choose the lottery whose samples look better". For instance, exa... (read more)
One interesting case where this theorem doesn't apply would be if there are only finitely many possible outcomes. This is physically plausible: consider multiplying the maximum data density¹ by the spacetime hypervolume of your future light cone from now until the heat death of the universe.
¹ <https://physics.stackexchange.com/questions/2281/maximum-theoretical-data-density>
I just quickly browsed this post. Based on the overall topic, you might also be interested in these inconsistency results in infinitary utiliatarianism written by my PhD advisor (a set theorist) and his wife (a philosopher).
http://jdh.hamkins.org/infinitary-utilitarianism/
Your proofs all rely on lotteries over infinite numbers of outcomes. Is that necessary? Maybe a restriction to finite lotteries avoids the paradox.
Thank you for making explicit the idea that the problems with "unbounded utilities" don't even require the existence of a utility function, just a very weak assumption about preference ordering with respect to probability mixtures and the existence of "arbitrarily strong" outcomes.
I should note that this is more or less the same thing that Alex Mennen and I have been pointing out for quite some time, even if the exact framework is a little different. You can't both have unbounded utilities, and insist that expected utility works for infinite gambles.
IMO the correct thing to abandon is unbounded utilities, but whatever assumption you choose to abandon, the basic argument is an old one due to Fisher, and I've discussed it in previous posts! (Even if the framework is a little different here, this seems essentially similar.)
I'm glad t... (read more)
Promoted to curated: I've been thinking about unbounded utilities for a while, and while I agree with a bunch of the top commenters on the actual relevant element being the integrateability of utilities, and that this can save some unbounded utility assignments (in a way that feels relevant to my current beliefs on the topic).
I do think nevertheless that this post is the best distillation of a bunch of the impossibility arguments I've seen floating around for the last decade, and I think it is an actually important question when trying to decide how to relate to the future. So curating it seems appropriate.
The examples and results in your post are very interesting and surprising. Thanks for writing this.
I'm inclined to reject the dominance axioms you've assumed, at least for mixtures of infinitely many lotteries. I think stochastic dominance is a more fundamental axiom, avoids inconsistency and doesn't give any obviously wrong answers on finite payoff lotteries (even mixtures of infinitely many lotteries or outcomes with intuitively infinite expected value, including St. Petersburg and Pasadena). See Christian Tarsney's "Exceeding Expectations: Stochastic Do... (read more)
This doesn't hold if you restrict to utility functions that asymptote to a finite value, correct?
Very nice, thanks. I agree with others that if one were intent on keeping unbounded utilities, it seems simplest to give up on probability distributions that have infinite support (it seems one can avoid all these paradoxes by restricting oneself to finite-support distributions only). I guess this is similar to what you mean by "abandoning probability itself", but do note that you can keep probability so long as all the supports are finite.