Summary: the problem with Pascal's Mugging arguments is that, intuitively, some probabilities are just too small to care about. There might be a principled reason for ignoring some probabilities, namely that they violate an implicit assumption behind expected utility theory. This suggests a possible approach for formally defining a "probability small enough to ignore", though there's still a bit of arbitrariness in it.
This post is about finding a way to resolve the paradox inherent in Pascal's Mugging. Note that I'm not talking about the bastardized version of Pascal's Mugging that's gotten popular of late, where it's used to refer to any argument involving low probabilities and huge stakes (e.g. low chance of thwarting unsafe AI vs. astronomical stakes). Neither am I talking specifically about the "mugging" illustration, where a "mugger" shows up to threaten you.
Rather I'm talking about the general decision-theoretic problem, where it makes no difference how low of a probability you put on some deal paying off, because one can always choose a humongous enough payoff to make "make this deal" be the dominating option. This is a problem that needs to be solved in order to build e.g. an AI system that uses expected utility and will behave in a reasonable manner.
Intuition: how Pascal's Mugging breaks implicit assumptions in expected utility theory
Intuitively, the problem with Pascal's Mugging type arguments is that some probabilities are just too low to care about. And we need a way to look at just the probability part component in the expected utility calculation and ignore the utility component, since the core of PM is that the utility can always be arbitrarily increased to overwhelm the low probability.
Let's look at the concept of expected utility a bit. If you have a 10% chance of getting a dollar each time when you make a deal, and this has an expected value of 0.1, then this is just a different way of saying that if you took the deal ten times, then you would on average have 1 dollar at the end of that deal.
More generally, it means that if you had the opportunity to make ten different deals that all had the same expected value, then after making all of those, you would on average end up with one dollar. This is the justification for why it makes sense to follow expected value even for unique non-repeating events: because even if that particular event wouldn't repeat, if your general strategy is to accept other bets with the same EV, then you will end up with the same outcome as if you'd taken the same repeating bet many times. And even though you only get the dollar after ten deals on average, if you repeat the trials sufficiently many times, your probability of having the average payout will approach one.
Now consider a Pascal's Mugging scenario. Say someone offers to create 10^100 happy lives in exchange for something, and you assign them a 0.000000000000000000001 probability to them being capable and willing to carry through their promise. Naively, this has an overwhelmingly positive expected value.
But is it really a beneficial trade? Suppose that you could make one deal like this per second, and you expect to live for 60 more years, for about 1,9 billion trades in total. Then, there would be a probability of 0,999999999998 that the deal would never once have paid off for you. Which suggests that the EU calculation's implicit assumption - that you can repeat this often enough for the utility to converge to the expected value - would be violated.
Our first attempt
This suggests an initial way of defining a "probability small enough to be ignored":
1. Define a "probability small enough to be ignored" (PSET, or by slight rearranging of letters, PEST) such that, over your lifetime, the expected times that the event happens will be less than one.
2. Ignore deals where the probability component of the EU calculation involves a PEST.
Looking at the first attempt in detail
To calculate PEST, we need to know how often we might be offered a deal with such a probability. E.g. a 10% chance for something might be a PEST if we only lived for a short enough time that we could make a deal with a 10% chance once. So, a more precise definition of a PEST might be that it's a probability such that
(amount of deals that you can make in your life that have this probability) * (PEST) < 1
But defining "one" as the minimum times we should expect the event to happen for the probability to not be a PEST feels a little arbitrary. Intuitively, it feels like the threshold should depend on our degree of risk aversion: maybe if we're risk averse, we want to reduce the expected amount of times something happens during our lives to (say) 0,001 before we're ready to ignore it. But part of our motivation was that we wanted a way to ignore the utility part of the calculation: bringing in our degree of risk aversion seems like it might introduce the utility again.
What if redefined risk aversion/neutrality/preference (at least in this context) as how low one would be willing to let the "expected amount of times this might happen" fall before considering a probability a PEST?
Let's use this idea to define an Expected Lifetime Utility:
ELU(S,L,R) = the ELU of a strategy S over a lifetime L is the expected utility you would get if you could make L deals in your life, and were only willing to accept deals with a minimum probability P of at least S, taking into account your risk aversion R and assuming that each deal will pay off approximately P*L times.
ELU example
Suppose that we a have a world where we can take three kinds of actions.
- Action A takes 1 unit of time and has an expected utility of 2 and probability 1/3 of paying off on any one occasion.
- Action B takes 3 units of time and has an expected utility of 10^(Graham's number) and probability 1/100000000000000 of paying off one any one occasion.
- Action C takes 5 units of time and has an expected utility of 20 and probability 1/100 of paying off on an one occasion.
Assuming that the world's lifetime is fixed at L = 1000 and R = 1:
ELU("always choose A"): we expect A to pay off on ((1000 / 1) * 1/3) = 333 individual occasions, so with R = 1, we deem it acceptable to consider the utility of A. The ELU of this strategy becomes (1000 / 1) * 2 = 2000.
ELU("always choose B"): we expect B to pay off on ((1000 / 3) * 1/100000000000000) = 0.00000000000333 occasions, so with R = 1, we consider the expected utility of B to be 0. The ELU of this strategy thus becomes ((1000 / 3) * 0) = 0.
ELU("always choose C"): we expect C to pay off on ((1000 / 5) * 1/100) = 2 individual occasions, so with R = 1, we consider the expected utility of C to be ((1000 / 5) * 20) = 4000.
Thus, "always choose C" is the best strategy.
Defining R
Is R something totally arbitrary, or can we determine some more objective criteria for it?
Here's where I'm stuck. Thoughts are welcome. I do know that while setting R = 1 was a convenient example, it's most likely too high, because it would suggest things like not using seat belts.
General thoughts on this approach
An interesting thing about this approach is that the threshold for a PEST becomes dependent on one's expected lifetime. This is surprising at first, but actually makes some intuitive sense. If you're living in a dangerous environment where you might be killed anytime soon, you won't be very interested in speculative low-probability options; rather you want to focus on making sure you survive now. Whereas if you live in a modern-day Western society, you may be willing to invest some amount of effort in weird low-probability high-payoff deals, like cryonics.
On the other hand, whereas investing in that low-probability, high-utility option might not be good for you individually, it could still be a good evolutionary strategy for your genes. You yourself might be very likely to die, but someone else carrying the risk-taking genes might hit big and be very successful in spreading their genes. So it seems like our definition of L, lifetime length, should vary based on what we want: are we looking to implement this strategy just in ourselves, our whole species, or something else? Exactly what are we maximizing over?
I don't know if this solves very much. As you say, if we use the number 1, then we shouldn't wear seatbelts, get fire insurance, or eat healthy to avoid getting cancer, since all of those can be classified as Pascal's Muggings. But if we start going for less than one, then we're just defining away Pascal's Mugging by fiat, saying "this is the level at which I am willing to stop worrying about this".
Also, as some people elsewhere in the comments have pointed out, this makes probability non-additive in an awkward sort of way. Suppose that if you eat unhealthy, you increase your risk of one million different diseases by plus one-in-a-million chance of getting each. Suppose also that eating healthy is a mildly unpleasant sacrifice, but getting a disease is much worse. If we calculate this out disease-by-disease, each disease is a Pascal's Mugging and we should choose to eat unhealthy. But if we calculate this out in the broad category of "getting some disease or other", then our chances are quite high and we should eat healthy. But it's very strange that our ontology/categorization scheme should affect our decision-making. This becomes much more dangerous when we st... (read more)
A rule-of-thumb I've found use for in similar situations: There are approximately ten billion people alive, of whom it's a safe conclusion that at least one is having a subjective experience that is completely disconnected from objective reality. There is no way to tell that I'm not that one-in-ten-billion. Thus, I can never be more than one minus one-in-ten-billion sure that my sensory experience is even roughly correlated with reality. Thus, it would require extraordinary circumstances for me to have any reason to worry about any probability of less than... (read more)
Thanks for these thoughts, Kaj.
It's a worthwhile effort to overcome this problem, but let me offer a mode of criticising it. A lot of people are not going to want the principles of rationality to be contingent on how long you expect to live. There are a bunch of reasons for this. One is that how long you expect to live might not be well-defined. In particular, some people will want to say that there's no right answer to the question of whether you become a new person each time you wake up in the morning, or each time some of your brain cells die. On the o... (read more)
I think this simplifies. Not sure, but here's the reasoning:
L (or expected L) is a consequence of S, so not an independent parameter. If R=1, then is this median maximalisation?http://lesswrong.com/r/discussion/lw/mqa/median_utility_rather_than_mean/ it feels close to that, anyway.
I'll think some more...
This is just a complicated way of saying, "Let's use bounded utility." In other words, the fact that people don't want to take deals where they will overall expect to get nothing out of it (in fact), means that they don't value bets of that kind enough to take them. Which means they have bounded utility. Bounded utility is the correct reponse to PM.
I'll need some background here. Why aren't bounded utilities the default assumption? You'd need some extraordinary arguments to convince me that anyone has an unbounded utility function. Yet this post and many others on LW seem to implicitly assume unbounded utility functions.
So, um:
Which axiom does this violate?
I like Scott Aaronson's approach for resolving paradoxes that seemingly violate intuitions -- see if the situation makes physical sense.
Like people bring up "blockhead," a big lookup table that can hold an intelligent conversation with you for [length of time], and wonder whether this has ramifications for the Turing test. But blockhead is not really physically realizable for reasonable lengths.
Similarly for creating 10^100 happy lives, how exactly would you go about doing that in our Universe?
By some alternative theory of physics that has a, say, .000000000000000000001 probability of being true.
What if one considers the following approach: Let e be a probability small enough that if I were to accept all bets offered to me with probability p<= e then the expected number of such bets that I win is less than one. The approach is to ignore any bet where p <=e.
This solve's Yvain's problem with wearing seatbelts or eating unhealthy for example. It also solves the problem that "sub-dividing" a risk no longer changes whether you ignore the risk.
Rolling all 60 years of bets up into one probability distribution as in your example, we get:
I think what this shows is that the aggregating technique you propose is no different than just dealing with a 1-shot bet. So if you can't solve the one-shot Pascal's mugging, aggregating it won't help in general.
Would that mean that if I expect to have to use transport n times throughout the next m years, with probability p of dying during commuting; and I want to calculate the PEST of, for example, fatal poisoning from canned food f, which I estimate to be able to happen about t times during the same m years, I have to lump the two dangers together and see if it is still <1? I mean, I can work from home and never eat canned food... But this doesn't seem to be what you write about when you talk about different deals.
(Sorry for possibly stupid question.)
I'm probably way late to this thread, but I was thinking about this the other day in the response to a different thread, and considered using the Kelly Criterion to address something like Pascal's Mugging.
Trying to figure out your current 'bankroll' in terms of utility is probably open to intepretation, but for some broad estimates, you could probably use your assets, or your expected free time, or some utility function that included those plus whatever else.
When calculating optimal bet size using the Kelly criterion, you end up with a percentage of yo... (read more)
I think the mugger can modify their offer to include "...and I will offer you this deal X times today, so it's in your interest to take the deal every time," where X is sufficiently large, and the amount requested in each individual offer is tiny but calibrated to add up to the amount that the mugger wants. If the odds are a million to one, then to gain $1000, the mugger can request $0.001 a million times.
The way I see it, if one believes that the range of possible extremes of positive or negative values is greater than the range of possible probabilities then one would have reason to treat rare high/low value events as more important than more frequent events.
Some more extreme possibilities on the lifespan problem: Should you figure in the possibility of life extension? The possibility of immortality?
What about Many Worlds? If you count alternate versions of yourself as you, then low probability bets make more sense.
This is an interesting heuristic, but I don't believe that it answers the question of, "What should a rational agent do here?"
The reasoning why one should rely on expected value on one offs can be used to circumvent the reasoning. It is mentioned int he article but I would like to raise it explicitly.
If I personally have a 0.1 chance of getting a high reward within my lifetime then 10 persons like me would on average hit the jackpot once.
Or in the reverse if one takes the conclusion seriously one needs to start rejecting one-offs because there isn't sufficient repetition to tend to the mean. Well you could say that value is personal and thus relevant repetition class is lifetime decisions. But if we take life to be "human value" then the relevant repetition class is choices made by homo sapiens (and possibly beyond).
If the stated probability is what you really assign then yes, positive expected value.
I see the key flaw in that the more exceptional the promise is, the lower the probability you must assign to it.
Would you give more credibility to someone offering you 10^2 US$ or 10^7 US$?
As an alternative to probability thresholds and bounded utilities (already mentioned in the comments), you could constrain the epistemic model such that for any state and any candidate action, the probability distribution of utility is light-tailed.
The effect is similar to a probability threshold: the tails of the distribution don't dominate the expectation, but this way it is "softer" and more theoretically principled, since light-tailed distributions, like those in the exponential family, are in a certain sense, "natural".