This is presenting some old work on combining different possible utility functions, that is worth revealing to the world.

I've written before about the problem of reaching an agreement between agents with different utility functions. The problem re-appears if you yourself are uncertain between two different moral theories.

For example, suppose you gave $99\%$ credence to average utilitarianism and $1\%$ credence to total utilitarianism. In an otherwise empty universe, you can create one person with $2$ utility, or a thousand with $1$ utility.

If we naively computed the expected utility of both actions, we would get $0.99\left(2\right)+0.01\left(2\right)=2$ for the first choice, and $0.99\left(1\right)+0.01\left(1000\right)=10.99$ for the second. It therefore seems that total utilitarianism wins by default, even though it is very unlikely (for you).

But the situation can be worse. Suppose that there is a third option, which created ten thousand people with each $0.000001$ utility. And you have $99\%$ credence on average utilitarianism, $(1-{10}^{-100})\%$ credence on total utilitarianism, and $\left({10}^{-100}\right)\%$ credence on exponential utilitarianism, where the average utility is multiplied by two to the power of the population. In this case the third option - and the incredibly unlikely exponential utilitarianism - win out massively.

Normalising utilities

To prevent the large-population-loving utilities from winning out by default, it's clear we need to normalise the utilities in some way before adding them together, similarly to how you normalise the utilities of opposing agents.

I'll distinguish two methods here: individual normalisations, and collective normalisations. For individual normalisations, if you have credences of $p_i$ for utilities $u_i\in U$, then $u_i$ is normalised into ${\hat{u}}_i$ using some procedure that is independent of $p_i$, $p_j$, and $u_j$ for $j\ne i$. Then the normalised utilities are added to give your total utility function of:

In collective normalisations, the normalisation of $u_i$ into ${\hat{u}}_i$ is allowed to depend upon the other utilities and the credences. All Pareto outcomes for the utilities are equivalent (modulo resolving ties) with maximising such a $u$.

The Nash Bargaining Equilibrium and the Kalai-Smorodinsky Bargaining Solution are both collective normalisations; the Mutual Worth Bargaining Solution is an individual normalisation iff the choice of the default point is individual (but doing that violates the spirit of what that method is supposed to achieve).

Note that there are no non-dictatorial Pareto normalisations, whether individual or collective, that are independent of irrelevant alternatives, or that are immune to lying.

Individual normalisations

Here I'll present the work that I did with Owen Cotton-Barratt, Toby Ord, and Will MacAskill, in order to try and come up with a principled way of doing individual normalisations. In a certain sense, this work failed: we didn't find any normalisations that were clearly superior in every way to others. But we did find a lot about the properties of the different normalisations; one interesting thing is that the dumbest normalsation - the zero-one, or min-max - has surprisingly good properties.

Let $O$ be the option set for the agent: the choices that it can make (in our full treatment, we considered a larger set $O\supset O$, the normalisation set, but this won't be needed here).

For the purpose of this post, $O$ will be equal to $\Pi =\left\{{\pi }_j\right\}$, the set of deterministic policies the agent can follow; this feels like a natural choice, as it's what the agent really has control over.

For any $u_i\in U$ and ${\pi }_j\in \Pi$, there is the expected utility of $u_i$ conditional on the agent following policy ${\pi }_j$; this will be designated by $u_i\left({\pi }_j\right)$.

We may have a probability distribution $q$ over $O=\Pi$ (maybe defined by the complexity of the policy?). If we don't have such a normalisation, and the set of deterministic policies is finite, then we can set $q$ to be the uniform distribution.

Then, given $q$, each $u_i$ becomes a real-valued random variable, taking value $u_i\left({\pi }_j\right)$ with probability $q\left({\pi }_j\right)$. We'll normalise these $u_i$ by normalising the properties of this random variable.

First of all, let's exclude any $u_i$ that are constant on all of $\Pi$; these utilities cannot be changed, in expectation, by the agent's policies, so should make no difference. Then each $u_i$, seen as a random variable, has the following properties:

• Maximum: $max{u}_i={max}_{{\pi }_j}{u}_i\left({\pi }_j\right)$.
• Minimum: $min{u}_i={min}_{{\pi }_j}{u}_i\left({\pi }_j\right)$.
• Mean: ${\mu }_i={\sum }_{j}q\left({\pi }_j\right)u_i\left({\pi }_j\right)$.
• Variance: ${\sigma }_i^2={\sum }_{j}q\left({\pi }_j\right)\left(u_i\right({\pi }_j){)}^2-{\mu }_i^2$.
• Mean difference: ${\delta }_i={\sum }_{j,k}q\left({\pi }_j\right)q\left({\pi }_k\right)|u_i\left({\pi }_j\right)-u_i\left({\pi }_k\right)|$.

There are five natural normalisation methods that emerge from these properties. The first and most trivial is the min-max or zero-one normalisation: scale and translate $u_i$ so that $min{u}_i$ takes the value $0$ and $max{u}_i$ takes the value $1$ (note that the translation doesn't change the desired policy when summing utilities, so what is actually required is to scale $u_i$ so that $(max{u}_i)-(min{u}_i)=1$).

The second nomalisation, the mean-max, involves setting $(max{u}_i)-{\mu }_i=1$; by symmetry, the min-mean normalisation involves setting ${\mu }_i-(min{u}_i)=1.$

Finally, the last two normalisations involve setting either the variance, or the mean difference, to $1$.

Meaning of the normalisations

What do these normalisations mean? Well, min-max is a normalisation that cares about the difference between perfect utopia and perfect dystopia: between the best possible and the worst possible expected outcome. Conceptually, this seems problematic - it's not clear why the dystopia matters, with seems like something that opens the utility up to extortion - but, as we'll see, the min-max normalisation has the best formal properties.

The mean-max is the normalisation that most appeals to me; the mean is the expected value of random policy, while the max is the expected outcome of the best policy. In a sense, that's the job of an agent with a single utility function: to move the outcome from random to best. Thus the max has a meaning that the min, for instance, lacks.

For this reason, I don't see the min-mean normalisation as being anything meaningful; it's the difference between complete disaster and a random policy.

I don't fully grasp the meaning of the variance normalisation; Owen Cotton-Barratt did the most work on it, and showed that, in a certain sense, it was resistant to lying/strategic distortion in certain circumstances, if a given utility didn't 'know' what the other utilities would be. But I didn't fully understand this point. So bear in mind that this normalisation has positive properties that aren't made clear in this post.

Finally, the mean difference normalisation controls the spread between the utilities of the different policies, in a linear way that may seem to be more natural than the variance.

Properties of the normalisation

So, which normalisation is best? Here's were we look at the properties of the normalisations (they will be summarised in a table at the end). As we've seen, independence of irrelevant alternatives always fails, and there can always be an incentive for a utility to "lie" (as in, there are $U$, $u_i\in U$, $p$, $\Pi$, and $q$, such that $u_i$ would have a higher expected utility under the final $u$ if it was replaced with $u_i^{\prime }\ne u_i$).

What other properties do all the normalisations share? Well, since they normalise independently, $u$ is continuous in $p$. And because the minimum, maximum, variance, etc... are continuous in $q$ and in $u_i\left({\pi }_j\right)$, then $u$ is also continuous in that information.

In contrast, the best policy $\arg {max}_{{\pi }_j}u\left({\pi }_j\right)$ of $u$ is not typically continuous in the data. Imagine that there are two utilities and two policies: $u_0\left({\pi }_0\right)=u_1\left({\pi }_1\right)=1$ and $u_0\left({\pi }_1\right)=u_1\left({\pi }_0\right)=0$. Then for $p_0<1/2$, ${\pi }_1$ is the optimal policy (for all the above normalisations for uniform $q$), while for $p_0>1/2$, ${\pi }_0$ is optimal.

Ok, that's enough of properties that all methods share; what about ones they don't?

First of all, we can look at the negation symmetry between $u_i$ and $-u_i$. Min-max, variance, and mean difference all have the same normalisation for $u_i$ and $-u_i$; mean-max and min-mean do not, since the mean can be closer to the min than that max (or vice versa).

Then we can consider what happens when some policies ${\pi }_j$ and ${\pi }_k$ are clones of each other: imagine that for all $u_i\in U$, $u_i\left({\pi }_j\right)=u_i\left({\pi }_k\right)$. Then what happens if we remove the redundant ${\pi }_k$ and normalise on $U-\left\{{\pi }_k\right\}$? Well, it's clear that the maximum or minimum value of $u_i$ cannot change (since if ${\pi }_k$ was a maximum/minimum, then so is ${\pi }_j$, which remains), so the min-max normalisation is unaffected.

All the other normalisations change, though. This can be seen in the example $U=\{u_0,u_1\}$, $\Pi =\{{\pi }_0,{\pi }_1,{\pi }_2,{\pi }_3\}$, with $u_0\left({\pi }_0\right)=u_1\left({\pi }_0\right)=-1$, $u_1\left({\pi }_0\right)=1$, $u_1\left({\pi }_1\right)=0$, $u_1\left({\pi }_2\right)=u_1\left({\pi }_3\right)=1$, and $u_0\left({\pi }_2\right)=u_1\left({\pi }_3\right)=0$; in terms of sets of expected utilities in terms of policies, ${\pi }_0$ has $(1,0,0,-1)$ while ${\pi }_1$ has $(1,1,0,-1)$. Then for uniform $q$, all other normalisation methods change if we remove ${\pi }_3$ which is identical to ${\pi }_2$ for both utilities.

Thus all the other normalisation change when we add (or remove) clones of existing policies.

Finally, we can consider what happens if we are in one of several worlds, and the policies/utilities are the identical in some of these worlds. This should be treated the same as if those identical worlds were all one.

So, imagine that we are in one of three worlds: $W_0$, $W_1$, and $W_2$, with probabilities ${\rho }_0$, ${\rho }_1$, and ${\rho }_2$, respectively. Before taking any actions, the agent will discover which world it is in. Thus, if ${\Pi }_i$ is the set of policies in $W_i$, the complete set of policies is ${\Pi }_0\times {\Pi }_1\times {\Pi }_2$.

The worlds $W_1$ and $W_2$ are, however, indistinguishable for all utilities in $U$. Thus we can identify $f\left({\Pi }_1\right)\cong {\Pi }_2$, with $u_i\left({\pi }_j\right)=u_i\left(f\right({\pi }_j\left)\right)$ for all $u_i\in U$. Then a normalisation method combines indistinguishable choices property if the normalisation is the same in world ${\rho }_0{W}_0+{\rho }_1{W}_1+{\rho }_2{W}_2$ and ${\rho }_0{W}_0+({\rho }_1+{\rho }_2)W_1$. Then:

• Min-max, mean-max, and min-mean combine indistinguishable choices. Variance and mean difference normalisations do not.

Proof (sketch): Let $u_i^j=u_i{|}_{W_j}$ be the random variable that is $u_i$ on $W_j$ under the assumption that $W_j$ is the true underlying world. Then on ${\rho }_0{W}_0+{\rho }_1{W}_1+{\rho }_2{W}_2$, $u_i$ behaves like the random variable ${\rho }_0{u}_i^0+{\rho }_1{u}_i^1+{\rho }_2{u}_i^2$. (this means that $u_i$ has probability ${\rho }_j$ of being $u_i^j$, not that it adds random variables together). Mean, max, and min all have the property that $f(aX+bY)=af\left(X\right)+bf\left(Y\right)$; variance and mean difference, on the other hand, do not.

Summary of properties

In my view, it is a big worry that the variance and mean difference normalisations fail to combine indistinguishable choices. World $W_1$ and $W_2$ could be strictly identical, except for some irrelevant information that all utility functions agree is irrelevant. We have to worry about whether the light from a distant star is slightly redder or slightly bluer than expected; what colour ink was used in a proposal; the height of the next animal we see, and so on.

This means that we cannot divide the universe into relevant and irrelevant variables, and focus solely on the first.

In table form, the various properties are:

As can be seen, the min-max method, simplistic though it is, has all the possible nice properties.