*When interpreted convservatively, the von Neumann-Morgenstern rationality axioms and utility theorem are an indispensible tool for the normative study of rationality, deserving of many thought experiments and attentive decision theory. It's one more reason I'm glad to be born after the 1940s. Yet there is apprehension about its validity, aside from merely confusing it with Bentham utilitarianism (as highlighted by Matt Simpson). I want to describe not only what VNM utility is really meant for, but a contextual reinterpretation of its meaning, so that it may hopefully be used more frequently, confidently, and appropriately.*

- Preliminary discussion and precautions
- Sharing decision utility is sharing power, not welfare
- Contextual Strength (CS) of preferences, and VNM-preference as "strong" preference
- Hausner (lexicographic) decision utility
- The independence axiom isn't bad either
- Application to earlier LessWrong discussions of utility

### 1. Preliminary discussion and precautions

The idea of John von Neumann and Oskar Mogernstern is that, *if* you behave a certain way, *then* it turns out you're maximizing the expected value of a particular function. Very cool! And their description of "a certain way" is very compelling: a list of four, reasonable-seeming axioms. If you haven't already, check out the Von Neumann-Morgenstern utility theorem, a mathematical result which makes their claim rigorous, and true.

VNM utility is a *decision utility*, in that it aims to characterize the decision-making of a rational agent. One great feature is that it implicitly accounts for risk aversion: not risking $100 for a 10% chance to win $1000 and 90% chance to win $0 just means that for you, utility($100) > 10%utility($1000) + 90%utility($0).

But as the Wikipedia article explains nicely, VNM utility is:

- not designed to predict the behavior of "irrational" individuals (like real people in a real economy);
- not designed to characterize
*well-being*, but to characterize*decisions*; - not designed to measure the value of
*items*, but the value of*outcomes*; - only defined up to a scalar multiple and additive constant (acting with utility function U(X) is the same as acting with a·U(X)+b, if a>0);
- not designed to be added up or compared between a number of individuals;
- not something that can be "sacrificed" in favor of others in a meaningful way.

[**ETA**] Additionally, in the VNM theorem the probabilities are understood to be known to the agent *as they are presented*, and to come from a source of randomness whose outcomes are *not significant* to the agent. Without these assumptions, its proof doesn't work.

Because of (4), one often considers *marginal utilities* of the form U(X)-U(Y), to cancel the ambiguity in the additive constant b. This is totally legitimate, and faithful to the mathematical conception of VNM utility.

Because of (5), people often "normalize" VNM utility to eliminate ambiguity in both constants, so that utilities are unique numbers that can be added accross multiple agents. One way is to declare that every person in some situation values $1 at 1 utilon (a fictional unit of measure of utility), and $0 at 0. I think a more meaningful and applicable normalization is to fix mean and variance with respect to certain outcomes (next section).

Because of (6), characterizing the altruism of a VNM-rational agent by how he sacrifices his own VNM utility is the wrong approach. Indeed, such a sacrifice is a contradiction. Kahneman suggests^{1}, and I agree, that *something else* should be added or substracted to determine the total, comparative, or average well-being of individuals. I'd call it "**welfare**", to avoid confusing it with VNM utility. Kahneman calls it E-utility, for "experienced utility", a connotation I'll avoid. Intuitively, this is certainly something you could sacrifice for others, or have more of compared to others. True, a given person's VNM utility is likely highly correlated with her personal "welfare", but I wouldn't consider it an accurate approximation.

So if not *collective welfare*, then what could cross-agent comparisons or sums of VNM utilities indicate? Well, they're meant to characterize *decisions*, so one meaningful application is to *collective decision-making*:

### 2. Sharing decision utility is sharing power, not welfare

Suppose decisions are to be made by or on behalf of a group. The decision could equally be about the welfare of group members, or something else. E.g.,

- How much vacation each member gets, or
- Which charity the group should invest its funds in.

Say each member expresses a VNM utility value—a decision utility—for each outcome, and the decision is made to maximize the total. Over time, mandating or adjusting each member's expressed VNM utilities to have a given mean and variance could ensure that no one person dominates all the decisions by shouting giant numbers all the time. Incidentally, this is a way of normalizing their utilities: it will eliminate ambiguity in the constants ''a'' and ''b'' in (4) of section 1, which is exactly what we need for cross-agent comparisons and sums to make sense.

Without thought as to whether this is a good system, the two decision examples illustrate how allotment of normalized VNM utility signifies sharing *power* in a collective decision, rather than sharing *well-being*. As such, the latter is better described by other metrics, in my opinion and in Kahneman's.

### 3. Contextual strength (CS) of preferences, and VNM-preference as "strong" preference

As a normative thory, I think VNM utility's biggest shortcomming is in its Archimedian (or "Continuity") axiom, which as we'll see, actually isn't very limiting. In its harshest interpretation, it says that if you won't sacrifice a small chance at X in order to get Y over Z, then you're *not allowed* to prefer Y over Z. For example, if you prefer green socks over red socks, then you must be willing to sacrifice some small, real probability of fulfilling immortality to favor that outcome. I wouldn't say this is *necessary* to be considered rational. Eliezer has noted implicitly in this post (excerpt below) that he also has a problem with the Archimedean requirement.

I think this can be fixed directly with reinterpretation. For a given context C of possible outcomes, let's *intuitively* define a "strong preference" in that context to be one which is comparable in some non-zero ratio to the strongest preferences in the context. For example, other things being equal, you might consistently prefer green socks to red socks, but this may be completely undetectable on a scale that includes immortal hapiness, making it not a "strong preference" in that context. You might think of the socks as "infinitely less significant", but infinity is confusing. Perhaps less daunting is to think of them as a "strictly secondary concern" (see next section).

I suggest that the four VNM axioms can work more broadly as axioms for *strong preference* in a given context. That is, we consider VNM-preference and VNM-utility

- to be defined only for a given context C of varying possible outcomes, and
- to intuitively only indicate those preferences
*finitely-comparable to the strongest ones*in the given context.

Then VNM-indifference, which they denote by equality, would simply mean a lack of *strong preference* in the given context, i.e. *not caring enough to sacrifice likelihoods of important things*. This is a Contextual Strength (CS) interpretation of VNM utility theory: in bigger contexts, VNM-preference indicates *stronger* preferences and *weaker* indifferences.

**(CS) Henceforth, I explicitly distinguish the terms VNM-preference and VNM-indifference as those axiomatized by VNM, interpreted as above.**

### 4. Hausner (lexicographic) decision utility

[**ETA**] To see the broad applicability of VNM utility, let's examine the flexibility of a theory without the Archimedean axiom, and see that they differ only mildly in result:

In the socks vs. immortality example, we could suppose that context "Big" includes such possible outcomes as immortal happiness, human extinction, getting socks, and ice-cream, and context "Small" includes only getting socks and ice-cream. You could have two VNM-like utility functions: U_{Small} for evaluating gambles in the Small context, and U_{Big} for the Big context. You could act to maximize EU_{Big} whenever possible (EU=expected utility), and when two gambles have the same EU_{Big}, you could default to choosing between them by their EU_{Small} values. This is essentially acting to maximize the pair (EU_{Big}, EU_{Small}), ordered *lexicographically*, meaning that a difference in the former value EU_{Big} trumps a difference in the latter value. We thus have a sensible numerical way to treat EU_{Big} as "infinitely more valuable" without really involving infinities in the calculations; there is no need for that interpretation if you don't like it, though.

Since we have the VNM axioms to imply when someone is maximizing *one* expectation value, you might ask, can we give some nice *weaker* axioms under which someone is maximizing a *lexicographic tuple* of expectations?

Hearteningly, this has been taken care of, too. By weakening—indeed, effectively eliminating— the Archimedean axiom, Melvin Hausner^{2} developed this theory in 1952 for Rand Corporation, and Peter Fishburn^{3} provides a nice exposition of Hausner's axioms. So now we have Hausner-rational agents maximizing Hausner utility.

[**ETA**] But the difference between Hausner and VNM utility comes into effect only in the rare event when you *know* you can't distinguish EU_{Big} values, otherwise the Hausner-rational behavior is to "keep thinking" to make sure you're not sacrificing EU_{Big}. The most plausible scenario I can imagine where this might actually happen to a human is when making a decision on a *precisely known time limit*, like say sniping on one of two simultaneous ebay auctions for socks. CronoDAS might say the time limit creates "noise in your expectations". If the time runs out and you have failed to distinguish which sock color results in higher chances of immortality or other EU_{Big} concerns, then I'd say it wouldn't be irrational to make the choice according to some secondary utility EU_{Small} that any detectable difference in EU_{Big} would otherwise trump.

Moreover, it turns out^{3} that the primary, i.e. most dominant, function in the Hausner utility tuple behaves almost exactly like VNM utility, and has the same uniqueness property (up to the constants ''a'' and ''b''). So except in rare circumstances, you can just think in terms of VNM utility and get the same answer, and even the rare exceptions involve considerations that are necessarily "unimportant" relative to the context.

Thus, a lot of apparent flexibility in Hausner utility theory might simply demonstrate that VNM utility is more applicable to you than it fist appeared. This situation favors the (CS) interpretation: even when the Archimedean axiom isn't quite satisfied, we can use VNM utility liberally as indicating "strong" preferences in a given context.

### 5. The independence axiom isn't so bad

"A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom." (Wikipedia) But I think the independence axiom (which Hausner also assumes) is a non-issue if we're talking about "strong preferences". The following, in various forms, is what seems to be the best argument against it:

Suppose a parent has no VNM preference between S: her son or her daughter gets a free car, and D: her daughter gets it. In the original VNM formulation, this is written "S=D". She is also presented with a third option, F=.5S+.5D. Descriptively, a fair coin would be flipped, and her son or daughter gets a car accordingly.

By writing S=.5S+.5S and D=.5D+.5D, the original independence axiom says that S=D implies S=F=D, so she must be VNM-indfferent between F and the others. However, a desire for "fair chances" might result in preferring F, which we might want to allow as "rational".

[**ETA**] I think the most natural fix within the VNM *theory* is to just say S' and D' are the events "car is awarded so son/daughter based on a coin toss", which are slightly better than S and D themselves, and that F is really 0.5S' + 0.5D'. Unfortunately, such modifications undermine the applicability of the VNM *theorem*, which implicitly assumes that the source of probabilities itself is insignificant to the outcomes for the agent. Luckily, Bolker^{4} has divised an axiomatic theory whose theorems will apply without such assumptions, at the expense of some uniqueness results. I'll have another occasion to post on this later.

Anyway, under the (CS) interpretation, the requirement "S=F=D" just means the parent lacks a *VNM-preference*, i.e. a *strong preference*, so it's not too big of a problem. Assuming she's VNM-rational just means that, in the implicit context, she is unwilling to make certain probabilitstic sacrifices to favor F over S and D.

- If the context is Big and includes something like death, the VNM-indifference "S=D" is a weak claim: it might just indicate an unwillingness to increase risk of things finitely-comparable to death in order to obtain F over S or D. She is still allowed to prefer F if no such sacrifice is involved.
- If the context is Small, and say only includes her kids getting cars, then "S=D" is a strong claim: it indicates an unwillingless to risk her kids not getting cars to favor S over D in a gamble. Then she can
*still*prefer F, but she couldn't prefer F'=.49S+.49D+.02(no car) over S or D, since it would contradict what "S=D" means in terms of car-sacrifice. I think that's reasonable, since if she simply flips a mental coin and gives her son the car, she can prefer to favor her daughter in later circumstances.

You might say VNM tells you to *"Be the fairness that you want to see in the world."*

### 6. Application to earlier other LessWrong discussions of utility

This contextual strength interpretation of VNM utility is directly relevant to resolving Eliezer's point linked above:

"...The utility function is not up for grabs.I love life without limit or upper bound: There is no finite amount of life lived N where I would prefer a 80.0001% probability of living N years to an 0.0001% chance of living a googolplex years and an 80% chance of living forever."

This could just indicate that Eliezer ranks immortality on a scale that trumps finite lifespan preferences, a-la-Hausner utility theory. In a context of differing positive likelihoods of immortality, these other factors are not strong enough to constitute VNM-preferences.

As well, Stuart Armstrong has written a thoughtful article "Extreme risks: when not to use expected utility", and argues against Independence. I'd like to recast his ideas context-relatively, which I think alleviates the difficulty:

In his paragraph 5, he considers various existential disasters. In my view, this is a case for a "Big" context utility function, not a case against independence. If you were gambling *only* between eistential distasters, then you have might have an "existential-context utility function", U_{Existential}. For example, would you prefer

- 90%(extinction by nuclear war) + 10%(nothing), or
- 60%(extinction by nuclear war) + 30%(extinction by asteroids) + 10%(nothing)?

If you prefer the latter enough to make some comparable sacrifice in the «nothing» term, contextual VNM just says you assign a higher U_{Existential} to «extinction by asteroids» than to «extinction by nuclear war».^{5} There's no need to be freaked out by assigning finite numbers here, since for example Hausner would allow the value of U_{Existential} to completely trump the value of U_{Everyday} if you started worrying about socks or ice cream. You could be both *extremely risk averse* regarding existential outcomes, and *absolutely unwilling* to gamble with them for more trivial gains.

In his paragraph 6, Stuart talks about giving out (necessarily normalized) VNM utility to people, which I described in section 2 as a model for sharing power rather than well-being. I think he gives a good argument against blindly maximizing the *total normalized VNM utility of a collective* in a one-shot decision:

"...imagine having to choose between a project that gave one util to each person on the planet, and one that handed slightly over twelve billion utils to a randomly chosen human and took away one util from everyone else. If there were trillions of such projects, then it wouldn’t matter what option you chose. But if you only had one shot, it would be peculiar to argue that there are no rational grounds to prefer one over the other, simply because the trillion-iterated versions are identical."

(Indeed, practically, the mean and variance normalization I described doesn't apply to provide the same "fairness" in a one-shot deal.)

I'd call the latter of Stuart's projects an unfair distribution of power in a collective decision process, something you might personally assign a low VNM utility to, and therefore avoid. Thus I wouldn't consider it an argument not to *use expected utility*, but an argument not to blindly favor *total normalized VNM utility of a population* in your own decision utility function. The same argument—Parfit's Repugnant Conclusion—is made against total normalized welfare.

*The expected utility model of rationality is alive and normatively kicking, and is highly adaptable to modelling very weak assumptions of rationality. I hope this post can serve to marginally persuade others in that direction.*

**References, notes, and further reading:**

^{1} Kahneman, Wakker and Sarin, 1997, Back to Bentham? Explorations of experienced utility, The quarterly journal of economics.

^{2} Hausner, 1952, Multidimensional utilities, Rand Corporation.

^{3} Fishburn, 1971, A Study of Lexicographic Expected Utility, Management Science.

^{4} Bolker, 1967, A simultaneous axiomatization of utility and probability, Philosophy of Science Association.

^{5} As wedrifid pointed out, you might instead just prefer uncertainty in your impending doom. Just as in section 5, neither VNM nor Hausner can model this usefully (i.e. in way that allows calculating utilities), though I don't consider this much of a limitation. In fact, I'd consider it a normative step backward to admit "rational" agents who actually *prefer uncertainty in itself*.

There is still a serious problem with this: you might call it the dust speck-torture problem. If you have a preference regarding socks, and another regarding immortality, and you say that your preference for immortality is infinitely more important than your preference regarding socks, then you are not allowed to find some third term which is comparable both with the socks and with immortality, nor even any finite sequence of terms which will connect them. But in practice this would not apply to any human being; there will always be such a sequence of terms, and thus the person would have to compare socks and immortality in order to remain rational.

Indeed, the question is whether non-Archimedean values should be

normatively outlawedfor rational agents.Do you think they should? How about for example, an AI whose primary goal is maximize the brightness of a certain light, with a secondary goal is to adjust its color closer to red. If it is (say, by design) totally unwilling to sacrifice some of the latter for the former, would you say it is necessarily irrational? How about a human with the same preferences?

Yeah I think non-Archimedian values are normatively irrational, i.e. make an AI less effective as an optimizer. The obvious problem is that when A is "infinitely preferable" to B, then even A's most half-baked subgoal is infinitely preferable to B. Even if I gave the AI a wrapped box with B inside, it wouldn't dedicate any resources to opening it -- better to just flap its arms so that the butterfly effect will increase the chance of A by 1/googol.

Heh, yeah that's roughly how I feel when noting Archimedeanity of my values. But then I wonder... maybe I wouldn't "flap my arms just so" to increase P(A) because I'm running on hostile hardware that makes my belief probabilities coarse-grained.... i.e. maybe I'm forced to treat 1/googol like 0, and open the box with B in it. I certainly feel that way.

Such reflection leads me to think that humans aren't precise enough for the difference between VNM utility and Hausner utility to really manifest decisively. When would a human really be convinced enough that EUBig(X)

precisely equalsEUBig(Y), so to start optimizing EUSmall? It seems like the difference between VNM and Hausner utility only happens in a measure-0 class of scenarios that humans couldn't practically detect anyway. ETA: except maybe when there's a time limit...This is actually one reason I posted on Hausner utility: if you like it, then note it's 0% likely to give you different answers from VNM utility, and then just use VNM because you're not precise enough to know the difference :)

Is there any use for introducing the concept of "noise terms" in lexicographical expected value?

Here's an illustration of what I mean by "noise terms":

Suppose that you have a warm cup of coffee sitting in a room. How long will it take before the coffee is the same temperature as the room? Well, the rate of heat transfer between two objects is proportional to the temperature difference between them. That means that the temperature difference between the coffee and the room undergoes exponential decay, and exponential decay never actually reaches its final value. It'll take

infinitely longfor the coffee to reach the same temperature as the room!Except that the real world doesn't quite work that way. If you try to measure temperature precisely enough, you'll start finding that your thermometer readings are fluctuating. All those "random" molecular motions create

noise. You can't measure the difference between 100 degrees and 100 + epsilon degrees as long as epsilon is less than thenoise levelof the system. So the coffee really does reach room temperature, because the difference becomes so small that it disappears into the noise.If the parameters that I use to calculate expected value are

noisy, then as long as the difference between EUBig(X) and EUBig(Y) is small enough - below the noise level of the system - I can't tell if it's positive or negative, so I can't know if I should prefer X to Y. As with the coffee in the room, the difference vanishes into the noise. So I'll resort to the tiebreaker, and optimize EUSmall instead.Does this make any sense?

I just added a paragraph about this. Good stuff!

This certainly has an intuitive appeal. Where would the noise be? In the outcome values, or the probabilities, or both?

To have an effect, the noise would have to something that you

somehow knowcan't be eliminated by allocating more thinking/computing time (perhaps you're on a time limit) or resources, else the rational thing to do would be to just "'think harder" to make sure you're not sacrificing your EUBig...Is that knowledge plausible?

Okay, I just added a paragraph about this. It seems to me that time limits are the biggest noise factor for humans.

Good stuff!

It's not really an issue of impracticality. It's just that, any time you have a higher-level class of utility, the lower-level classes of utility stop being relevant to your decisions. No matter how precise the algorithm is. That's why I say it's extra complexity with no optimization benefit. Since the extra structure doesn't even map better to my intuition about preference, I just Occam-shave it away.

Wait... certainly, if you lexicographically value (brightness, redness) of a light, and somehow manage to

bein a scenario where you can't make the light brighter,andsomehow manage toknowthat, then the redness value becomes relevant.What I mean is that the environment itself makes such precise situations rare (a non-practical issue), and an imprecise algorithm makes it hard to detect when, if ever, they occur (a practical issue).

I wasn't claiming that they should be normatively outlawed, just that in practice in human beings, they lead to logical inconsistency. On the other hand, in a perfect AI, they wouldn't necessarily lead to inconsistency, but the less important goal would be completely ignored, as Liron says, and therefore effectively you would still have Archimedean values.

Secondary preferences may as well not exist. The only time they have an effect on behavior is in the case of two or more possible optimal actions with identical expected utilities in terms of the primary good. How likely is that?

ETA: Douglas Knight said it first.

Yes (see above). It seems well agreed upon; I think I'll ETA a note about this to the post proper.

ETA: "noise" in your expected utility calculations might constitute positive-likelihood occasions to treat two expectations as equal...

Yes, if you drop continuity, all you get is lexicographic preferences, which isn't much of a change. But situations where you want to break ties don't come up much. There is usually the opportunity to measure more closely the first-order goal, rather than to try to control the tie-breaker.

I don't see that you've offered a better alternative to VNM utility.

Your examples assert that VNM does a bad job of capturing subtleties in the structure of human preference. But they can all be fixed by using the appropriate outcome-space as your utility function's domain.

For example, if I prefer to decide which child gets a car with a fair coin flip, then I can represent my utility function like this:

To me this seems like an elegant enough way to capture my value of fairness. And this kind of VNM formulation is so precise that I think it is useful for defining exactly

howI value fairness.I think the alternate utility theories introduce complexity without improving decision theory.

Unfortunately, what you describe deviates from the VNM model, which does not allow utilities that depend on the source of "objective randomness" in the lotteries; in this case, the coin. If we allow this dependence, it leads to difficulty in defining what probability is, or at least characterizing it behaviorally, and in proving the VNM theorem itself. This is all explained Bolker (1967) - A simultaneous axiomatization of utility and probability, which is probably my favorite paper on utility theory.

The key to building an axiomatic theory where your approach makes sense, due to Bolker, is to axiomatize

expectation itself. I originally planned to post on Bolker's theory, but unfortunately, the axioms themselves are a bit abstract, so I decided I should at least start with VNM utility. But Bolker's axioms allow one to work in the exactly the way you describe, which is why I like it so much.You didn't mention what I think is most important, though, and why I wrote this post: what do you think of the issues regarding the Archimedean axiom?

When it comes to biting VNM axioms, the Archimedian/continuity axiom is a non-bullet. Why do you have to insist that 1/3^^^3 human lives is an insultingly high price for green socks? I think it's a bargain. Like I said, this infinitesimal business adds complexity with no decision theory payoff.

I don't see why the son-daughter-car formulation I described isn't good enough as is. The domain of my utility function is the set of spans of future time, and the function works by analyzing the causal relationship between my brain and the person who gets the car.

And I don't see why "objective randomness" or anything else needs to come into the picture. I already have a structure that captures what I intuitively mean by "preference" for this example, and an algorithm that makes good decisions accordingly.

At no point have I insisted that, or anything analogous. But that's not the question at hand. I haven't found any of

myvalues to be non-Archimedean. The normative question is whether they should beoutlawedfor rational agents. From the post:What's your take? Do you think it should be normatively illegal to have non-Archimedian values (like say, a pair of Hausner utilities) and be considered rational? Please share you thoughts on this thread if you're interested.

As far as I'm concerned, it

isgood enough. So then the challenge is to provide reasonable assumptions whichallow the kind of analysis you describe, and

still manage to imply

mathematical theoremscomparable in strength to those of VNM.VNM utility theory isn't just an intuitive model. It's special: it's actually a consistent mathematical theory, with

theorems. And the task of weakening assumptions while maintaining strong theorems is formidable. Luckily, Bolker has done this, by axiomatizing expectation itself, so when we want rigorous theorems to fall back on while reasoning as you describe, they're to be found.AFAIC, it doesn't. But whether we like it or not, the VNM theory

needsthe source of randomness to not be a source of utility in order for the proof of the VNM utility theorem to work. I find this unsatisfactory, as you seem to. Luckily, Bolker's theory doesn't require this, which is great. Instead, Bolker pays a different price:themselvesbecome behaviorally ambiguous.This doesn't bother me much, nor you probably; I consider it a price easily worth paying to pass from VNM to Bolker.

I admit this comment is not as in-depth an explanation of some concepts as I'd like; if I find it ties together with enough interesting topics, and I think it will, then I'll write a top level post better explaining this stuff.

But I did upvote this post, because I think it's a good discussion to have.

...

I'm just going to say "tiered utilities" here, so that the next person who thinks like me and tries to see if anyone's written anything about (what turns out to be) Hausner utility can find this post easily.

Perhaps I'm missing something, but it seems to me this doesn't cover risk-aversion properly.

Suppose for me, utility($100) = 1, and utility($1000) = 100, and utility($0) = 0. Then, utility($100) < 10%utility($1000) + 90%utility($0); (1 < 10). Now suppose I am extremely risk-averse; I prefer to never wager any money I actually have and will practically always take a certain $100 over any uncertain $1000. It does not seem this configuration is impossible in an agent, but is not supported by your model of risk-aversion.

Does this merely mean that this type of risk-aversion is considered irrational and therefore not covered in the VNM model?

Yes. If you are risk-averse (or loss-averse) in terms of

marginal changes in your money, then you're not optimizing any consistent function of your total amount of money.I wasn't (directly) talking about utilitarianism in that post. I was talking about using utility functions to model two different things: behavior and preferences. Utilitarianism is a specific example of the latter.

Under the system you describe it is possible to weight each of the below scenarios equally:

You have allowed for agents to assign different weights for different probabilities of certain stuff happening. This does more than allow for risk-aversion. It allows risk-seeking and arbitrary risk appreciation such as preferring a any split of two kinds of existential catastrophe to 90% of either.

You can conclude nothing about whether you prefer asteroids to nuclear war based on preference between the scenarios that either of us just listed.You can not shut up and multiply (except when the agents whims it as such).In the VNM model, the probabilities are assumed to be from a source which is indifferent to the agent, and numerically agreed upon. That's how the proof of the theorem works. We already discussed this above, but thanks for writing out an example to explicitly demonstrate.

The discussion you linked to does not appear to be related to my comment. Unless for some reason you have rejected parts of your original post this claim is mistaken:

When I say

I mean that it would consider you modifying the probabilities as simply presenting a different problem.

Maybe you're asserting that, in principle, the assumption that we be sure of the agent's estimation of the probabilities can't be met. I would agree.

I didn't modify the probabilities. Unless you consider making the 'e' that you expressed

in wordsexplicit or converting percentage to decimal modification 'modificaiton'. I also added a third scenario to demonstrate a compatible alternative.I have concluded that the opening post contains a

trivial errorthat is unlikely to be corrected. The worst place for errors is in an official looking context with otherwise high quality content, so I have reversed my upvote.Ah! I see what you mean, and now agree that you didn't modify the scenario. So I can better understand, when you say

does the "except" clause mean "except when the agent is VNM rational", or some similar assumption?

If your point is that I can't

deduceyou are VNM-rational from your response to the extinction question alone, then you're definitely right. If I came across to the contrary, I should reword the post.What I meant to point out in that post paragraph is just that

1) there is no need to be "freaked out" by assigning finite utilities to huge events, because

2) even if you insist on treating them as "infinitely more important" than everyday events, you can just use Hausner utility to break ties in more trivial decisions (where the EUexistential values are equal, or "noisily indistinguishable" due for example to a precise time limit on a decision, after which time you've failed to distinguish existential risks from noise). I consider this an argument in favor of using in-context VNM utility normatively, since it is simpler than Hausner, and differs from it rarely, over relatively unimportant matters in the context.

I want to convince as many people as possible to use VNM utility confidently. I'd value any suggestions you have to rephrase the post to that effect, since (I think) you noticed the above unclarity in the first place.

Exactly! Perhaps just change "I'd say you assign a higher" to "then maybe you assign a higher". That would be enough to stop people (me) twisting their minds in a knot trying to work out whether there is an oversight or whether they just don't understand the whole VNM thing. I would also add something like "(or perhaps you just value unpredictability in your impending doom!)", which would help the curious mind (or, me again!) to confirm their understanding of now the VNM structure allows for valuing uncertain events.

I think I agree with your point. I am dubious about the whole 'infinitely more important' thing but if VNM handles that in a sane way then the problem of beliefs about infinity rests on the agent and his map and VNM can just handle whatever values and expectations the agent happens to have.

(Thanks for taking the time to take another look at what I was trying to say. It isn't often that I see conversations that go "No -> huh? Actually. -> wtf? No really" resolve themselves to agreement. So my prediction is one I am glad to be mistaken on!)

Okay, I rewrote the ending, and added a footnote about "valuing uncertainty". Thanks for all the input!

Maybe it helps that we're both more concerned with figuring stuff out than "winning an argument" :)

Possible typo :)

(Emphasis added.)

Thanks! Fixed.

Knowing that Eliezer is an "infinite set atheist" and uses infinity as the limit of a sequence of finite values, I would interpret his statement as:

You can replace 0.0001% with epsilon, giving:

This does not violate continuity, because M depends on epsilon.

This is not necessary to give meaning to immortality without reference to infinite sets. Immortality is the non-existence of a finite time of death. The statement can be regarded as valuing that scenario directly, without mentioning M or epsilon.

This depends on the assumption that there is an infinite amount of time. You have only hidden, not eliminated, your reference to an infinite set.

Certainly not. There are no infinite sets in first-order Peano arithmetic, and no largest number, either.

Perhaps you mean that first-order Peano arithmetic cannot prove that the set of natural numbers is infinite.

Well, first-order Peano arithmetic doesn't have any notion of "set", much less "infinite"...

Right, that is my point. If an axiomatic system has no concept of "set" or "infinite", it will not notice if the thing it is describing is in fact an infinite set. But we can use a more complete system, and notice that it is an infinite set and make deductions from it. Which is why first-order Peano arithmetic does not invalidate my claim that Academian's characterization of immortality relies on a hidden reference to an infinite set.

The standard model of Peano arithmetic is 0={} and S(X)=Xunion{X}; its objects are all finite sets.

I fail to see any meaningful sense in which saying "There does not exist a time T after the present such that I am not alive" has a

hidden reference to infinitywhich is somehow avoided by saying "For any duration N and probability epsilon, there exists a longer duration M such that..."The set of all objects in this standard model is infinite, even though Peano arithmetic does not refer to this set.

The difference is that in my version, utility is assigned to states of a finite universe. To the extent that there is a reference to infinity, the infinity describes an abstract mathematical object. In your version, utility is assigned to a state of an infinite universe. The infinity describes physical time, it says that there is an infinite amount of time you will be alive.

That's true. You prefer to assign infinitely many utilities to infinitely many states of possible finite universes, and I allow assigning one utility to one state of a possibly infinite universe.

I think the reasons for favoring each case are probably clear to us both, so now I vote for not generating further infinities of comments in the recent comments feed :)

No, I mean that there are no infinite sets in first-order Peano arithmetic. The class "natural number" is not an object in Peano arithmetic.

First-order Peano arithmetic does not explicitly refer to sets, infinite sets, or natural numbers, but it describes them.

Correct. Nor does "There is no time of death" refer to a set. You can model it as quantifying over a "set of times" if you like, just as Peano arithmetic can be modeled as quantifying over a "set of numbers", but this does not mean the theory refers to sets, or infinity.

There are no sets

at allin first-order Peano arithmetic--sets are simply not a thing that the theory talks about.(People talking

aboutfirst-order Peano arithmetic, however, do talk about sets a lot.)Unless for some reason time is constructed in a way that it would not end in a way that implies 'death'. Loops, or something that just hasn't occurred to me. Stranger things in physics than that are true.

I am not sure how to interpret your comment.

I am claiming that the reason someone might assign infinite utility to infinite life is that it is infinite, and that ways of describing infinite life that don't mention infinity, and rely on unspoken beliefs that something is infinite.

Do you contradict this claim?

No. I assert the claim that

Imade, which is relevant to that comment's parent.A related background assumption that I don't make is that 'no finite time of death' implies 'infinite life'. There are specifications of the physics of time that I assign non zero probability of being the correct map for which a reasonable specification of 'immortal' relies no infinite duration of time.