## LESSWRONGLW

I think that, depending on what the v's are, choosing a Pareto optimum is actually quite undesirable.

For example, let v1 be min(1000, how much food you have), and let v2 be min(1000, how much water you have). Suppose you can survive for days equal to a soft minimum of v1 and v2 (for example, 0.001 v1 + 0.001 v2 + min(v1, v2)). All else being equal, more v1 is good and more v2 is good. But maximizing a convex combination of v1 and v2 can lead to avoidable dehydration or starvation. Suppose you assign weights to v1 and v2, and are offered either 1000 of the more valued resource, or 100 of each. Then you will pick the 1000 of the one resource, causing starvation or dehydration after 1 day when you could have lasted over 100. If which resource is chosen is selected randomly, then any convex optimizer will die early at least half the time.

A non-convex aggregate utility function, for example the number of days survived (0.001 v1 + 0.001 v2 + min(v1, v2)), is much more sensible. However, it will not select Pareto optima. It will always select the 100 of each option; always selecting 1000 of one leads to greater expected v1 and expected v2 (500 for each).

This example doesn't satisfy the hypotheses of the theorem because you wouldn't want to optimize for v1 if your water was held fixed. Presumably, if you have 3 units of water and no food, you'd prefer 3 units of food to a 50% chance of 7 units of food, even though the latter leads to a higher expectation of v1.

2DaFranker7yWha...? I believe your Game is badly-formed. This doesn't sound at all like how Games should be modeled. Here, you don't have two agents each trying to maximize something that they value of their own, so you can't use those tricks. As a result, apparently you're not properly representing utility in this model. You're implicitly assuming the thing to be maximized is health and life duration, without modeling it at all. With the model you make, there are only two values, food and water. The agent does not care about survival with only those two Vs. So for this agent, yes, picking one of the "1000" options really truly spectacularly trivially is better. The agent just doesn't represent your own preferences properly, that's all. If your agent cares at all about survival, there should be a value for survival in there too, probably conditionally dependent on how much water and food is obtained. Better yet, you seem to be implying that the amount of food and water obtained isn't really important, only surviving longer is - strike out the food and water values, only keep a "days survived" value dependent upon food and water obtained, and then form the Game properly.

# 21

Restatement of: If you don't know the name of the game, just tell me what I mean to youAlternative to: Why you must maximize expected utility. Related to: Harsanyi's Social Aggregation Theorem.

Summary: This article describes a theorem, previously described by Stuart Armstrong, that tells you to maximize the expectation of a linear aggregation of your values. Unlike the von Neumann-Morgenstern theorem, this theorem gives you a reason to behave rationally.1

The von Neumann-Morgenstern theorem is great, but it is descriptive rather than prescriptive. It tells you that if you obey four axioms, then you are an optimizer. (Let us call an "optimizer" any agent that always chooses an action that maximizes the expected value of some function of outcomes.) But you are a human and you don't obey the axioms; the VNM theorem doesn't say anything about you.

There are Dutch-book theorems that give us reason to want to obey the four VNM axioms: E.g., if we violate the axiom of transitivity, then we can be money-pumped, and we don't want that; therefore we shouldn't want to violate the axiom of transitivity. The VNM theorem is somewhat helpful here: It tells us that the only way to obey the four axioms is to be an optimizer.2

So now you have a reason to become an optimizer. But there are an infinitude of decision-theoretic utility functions3 to adopt — which, if any, ought you adopt? And there is an even bigger problem: If you are not already an optimizer, than any utility function that you're considering will recommend actions that run counter to your preferences!

To give a silly example, suppose you'd rather be an astronaut when you grow up than a mermaid, and you'd rather be a dinosaur than an astronaut, and you'd rather be a mermaid than a dinosaur. You have circular preferences. There's a decision-theoretic utility function that says

$\mbox{mermaid} \prec \mbox{astronaut} \prec \mbox{dinosaur}$

which preserves some of your preferences, but if you have to choose between being a mermaid and being a dinosaur, it will tell you to become a dinosaur, even though you really really want to choose the mermaid. There's another decision-theoretic utility function that will tell you to pass up being a dinosaur in favor of being an astronaut even though you really really don't want to. Not being an optimizer means that any rational decision theory will tell you to do things you don't want to do.

So why would you ever want to be an optimizer? What theorem could possibly convince you to become one?

# Stuart Armstrong's theorem

Suppose there is a set $P$ (for "policies") and some functions $v_1, \dots, v_n$ ("values") from $P$ to $\mathbb{R}$. We want these functions to satisfy the following convexity property:

For any policies $p, q \in P$ and any $\alpha \in [0, 1]$, there is a policy $r \in P$ such that for all $i$, we have $v_i(r) = \alpha v_i(p) + (1 - \alpha) v_i(q)$.

For policies $p, q \in P$, say that $p$ is a Pareto improvement over $q$ if for all $i$, we have $v_i(p) \geq v_i(q)$. Say that it is a strong Pareto improvement if in addition there is some $i$ for which $v_i(p) > v_i(q)$. Call $p$Pareto optimum if no policy is a strong Pareto improvement over it.

Theorem. Suppose $P$ and $v_1, \dots, v_n$ satisfy the convexity property. If a policy in $P$ is a Pareto optimum, then it is a maximum of the function $c_1 v_1 + \cdots + c_n v_n$ for some nonnegative constants $c_1, \dots, c_n$.

This theorem previously appeared in If you don't know the name of the game, just tell me what I mean to you. I don't know whether there is a source prior to that post that uses the hyperplane separation theorem to justify being an optimizer. The proof is basically the same as the proof for the complete class theorem and the hyperplane separation theorem and the second fundamental theorem of welfare economics. Harsanyi's utilitarian theorem has a similar conclusion, but it assumes that you already have a decision-theoretic utility function. The second fundamental theorem of welfare economics is virtually the same theorem, but it's interpreted in a different way.

# What does the theorem mean?

Suppose you are a consequentialist who subscribes to Bayesian epistemology. And in violation of the VNM axioms, you are torn between multiple incompatible decision-theoretic utility functions. Suppose you can list all the things you care about, and the list looks like this:

3. Everyone's total welfare
4. The continued existence of human civilization
5. All mammals' total welfare
7. Everyone's average welfare
8. ...

Suppose further that you can quantify each item on that list with a function $v_1, v_2, \dots$ from world-histories to real numbers, and you want to optimize for each function, all other things being equal. E.g., $v_1(x)$ is large if $x$ is a world-history where your welfare is great; and $v_5(x)$ somehow counts up the welfare of all mammals in world-history $x$If the expected value of $v_1$ is at stake (but none of the other values are at stake), then you want to act so as to maximize the expected value of $v_1$.4 And if only $v_5$ is at stake, you want to act so as to maximize the expected value of $v_5$. What I've said so far doesn't specify what you do when you're forced to trade off value 1 against value 5.

If you're VNM-rational, then you are an optimizer whose decision-theoretic utility function is a linear aggregation $\sum_i c_i v_i$ of your values and you just optimize for that function. (The $c_i$ are nonnegative constants.) But suppose you make decisions in a way that does not optimize for any such aggregation.

You will make many decisions throughout your life, depending on the observations you make and on random chance. If you're capable of making precommitments and we don't worry about computational difficulties, it is as if today you get to choose a policy for the rest of your life that specifies a distribution of actions for each sequence of observations you can make.5 Let $P$ be the set of all possible policies. If $p \in P$, and for any $i$, let us say that $v_i(p)$ is the expected value of $v_i$ given that we adopt policy $p$. Let's assume that these expected values are all finite. Note that if $p_f$ is a policy where you make every decision by maximizing a decision-theoretic utility function $f$, then the policy $p_f$ itself maximizes the expected value of $f$, compared to other policies.

In order to apply the theorem, we must check that the convexity property holds. That's easy: If $p$ and $q$ are two policies and $\alpha \in [0, 1]$, the mixed policy where today you randomly choose policy $p$ with probability $\alpha$ and policy $q$ with probability $1-\alpha$, is also a policy.

What the theorem says is that if you really care about the values on that list (and the other assumptions in this post hold), then there are linear aggregations $\sum_i c_i v_i$ that you have reason to start optimizing for. That is, there are a set of linear aggregations and if you choose one of them and start optimizing for it, you will get more expected welfare for yourself, more expected welfare for others, less risk of the fall of civilization, ....

Adopting one of these decision-theoretic utility functions $\sum_i c_i v_i$ in the sense that doing so will get you more of the things you value without sacrificing any of them.

What's more, once you've chosen a linear aggregation, optimizing for it is easy. The ratio $c_i/c_j$ is a price at which you should be willing to trade off value $j$ against value $i$. E.g., a particular hour of your time should be worth some number of marginal dollars to you.

Addendum: Wei_Dai and other commenters point out that the set of decision-theoretic utility functions that will Pareto dominate your current policy very much depends on your beliefs. So a policy that seems Pareto dominant today will not have seemed Pareto dominant yesterday. It's not clear if you should use your current (posterior) beliefs for this purpose or your past (prior) beliefs.

# More applications

There's a lot more that could be said about the applications of this theorem. Each of the following bullet points could be expanded into a post of its own:

• Philanthropy: There's a good reason to not split your charitable donations among charities.
• Moral uncertainty: There's a good reason to linearly aggregate conflicting desires or moral theories that you endorse.
• Population ethics: There's a good reason to aggregate the welfare or decision-theoretic utility functions of a population, even though there's no canonical way of doing so.
• Population ethics: It's difficult to sidestep Parfit's Repugnant Conclusion if your only desiderata are total welfare and average welfare.

1This post evolved out of discussions with Andrew Critch and Julia Galef. They are not responsible for any deficiencies in the content of this post. The theorem appeared previously in Stuart Armstrong's post If you don't know the name of the game, just tell me what I mean to you.

2That is, the VNM theorem says that being an optimizer is necessary for obeying the axioms. The easier-to-prove converse of the VNM theorem says that being an optimizer is sufficient.

3Decision-theoretic utility functions are completely unrelated to hedonistic utilitarianism.

4More specifically, if you have to choose between a bunch of actions and for all $i>1$ the expected value of $v_i$ is independent of which actions you take, then you'll choose an action that maximizes the expected value of $v_1$.

5We could formalize this by saying that for each sequence of observations $o_1, \dots, o_k$, the policy determines a distribution over the possible actions at time $k+1$.