I'm working towards a toy model that will illustrate all the steps in the research agenda. It will start with some algorithmic stand-in for the "human", and proceed to create the $U_H$, following all the steps in that research agenda. So I'll be posting a series of "toy model pieces", that will then be ultimately combined in a full toy model. Along the way, I hope to get a better understanding of how to do the research agenda in practice, and maybe even modify that agenda based on insights making the toy model.

For this post, I'll look in more detail into how to combine different types of (partial) preferences.

Short-distance, long-distance, and other preferences

I normally use population ethics as my go-to-example for a tension between different types of preferences. You can get a lot of mileage by contrasting the repugnance of the repugnant conclusion with the seeming intuitiveness of the mere addition argument.

However, many people who read this will have strong opinions about population ethics, or at least some opinions. Since I'm not trying to convince anyone of my particular population ethics here, I thought it best to shift to another setting where we could see similar tensions at work, without the baggage.

Living in a world of smiles

Suppose you have three somewhat contradictory ethical intuitions. Or rather, in the formulation of my research agenda, two somewhat contradictory partial preferences.

The second is that any world would be better if people smiled more ($P_1$). The third is that if almost everyone smiles all the time, it gets really creepy ($P_2$).

Now, the proper way of resolving those preferences is to appeal to meta-preferences, or to cut them up into their web of connotations: why do we value smiles? Is it because people are happy? Why do we find universal smiling creepy? Is it because we fear that something unnatural is making them smile that way? That's the proper way of resolving those preferences.

However, let's pretend there are no meta-preferences, and no connotations, and just try to combine the preferences as given.

Smiles and worlds

Fix the population to a hundred people, and let $W$ be the set of worlds. This set will contain one hundred and one different worlds, described by $w\left(n\right)$, where $0\le n\le 100$ is an integer, denoting the number of people smiling in these worlds.

We can formalise the preferences as follows:

• $P_1=\left\{w\right(n\left){\le }_{1}w\right(m)\mid n\le m\}$.
• $P_2=\left\{w\right(n\left){\le }_{2}w\right(m)\mid n\ge 95$ and $n\ge m$ }$.

These give rise to the following utility functions (for simplicity of the formula, I've translated the definition of $U_2$; translations don't matter when combining utilities; I've also written $U_i\left(w\right(n\left)\right)$ as $U_i\left(n\right)$):

• $U_1\left(n\right)=2n-100$.
• $U_2\left(n\right)=2\times min(94-n,0)$.

But before being combined, there preferences have to be normalised. There are multiple ways we could do this, and I'll somewhat arbitrarily choose the "mean-max" method, which normalises the utility difference between the top world and the average world^[1].

Given that normalisation, we have:

• $||U_1|{|}_{mema}=100-0=100$.
• $||U_2|{|}_{mema}=0-(-42/101)=42/101\approx 0.42$.

Thus we send the $U_i$ to their normalised counterparts:

• $U_1\left(n\right)\to {\hat{U}}_1\left(n\right)=n/50-1$.
• $U_2\left(n\right)\to {\hat{U}}_2\left(n\right)=\frac{101}{21}min(94-n,0)$.

Now consider what happens when we do the weighted sum of these utilities, weighted by the intensity of the human feeling on the subject:

• $U=w_1{\hat{U}}_1+w_2{\hat{U}}_2$.

If the weights $w_1$ and $w_2$ are equal, we get the following, where the utility of the world grows slowly with the number of smiles, until it reaches the maximum at $n=94$ and then drops precipitously:

Thus $U_1$ is dominant most of the time when comparing worlds, but $U_2$ is very strong on the few worlds it really wants to avoid.

But what if $U_2$ (a seeming odd choice) is weighted less that $U_1$ (a more "natural" choice)?

Well, setting $w_1=1$ for the moment, if $w_2=21/5050$, then the utility for all worlds with $n\ge 94$ are the same:

.

Thus if $w_2>21/5050$, ${\hat{U}}_2$ will force the optimal $n$ to be $n\le 94$ (and ${\hat{U}}_1$ will select $n=94$ from these options). If $w_2<21/5050$, then ${\hat{U}}_1$ will dominate completely, setting $n=100$.

This seems like it could be extended to solve population ethics considerations in various ways (where $U_1$ might be total utilitarianism, with $U_2$ average utilitarianism or just a dislike of worlds with everyone at very low utility). To go back to my old post about differential versus integral ethics, $U_1$ is a differential constraint, $U_2$ is an integral one, and $n=94$ is the compromise point between them.

Inverting the utilities

If we invert the utilities, things behave differently. If we had $-U_1$ (smiles are bad) and $-U_2$ (only lots of smiles are good) instead, things would be different^[2]. In mean-max, the norm of these would be:

• $||-U_1|{|}_{mema}=100-0=100$.
• $||-U_2|{|}_{mema}=12-\left(42/101\right)=1170/101\approx 11.58$.

So the normalised version of $-U_1$ is just $-{\hat{U}}_1$, but the normalised version of $U_2$ is different from $-{\hat{U}}_2$.

Then, at equal weights, we get the following graph for $U$:

Thus $-U_2$ fails at having any influence, and $n=0$ is optimum.

To get the break-ever point, we need $w_2=585/303$, where $n=0$ and $n=100$ are equally valued:

For $w_2$ greater than that, $-U_2$ dominates completely, and forces $n=100$.

It's clear that $U_1$ and $U_2$ are less "antagonistic" than $-U_1$ and $-U_2$ are (compare the single peak in the graph in the first case, with the two peaks in the second).