Here's an idea about how to formally specify society-wide optimization, given that we know the utility function of each individual. In particular, it might be useful for multi-user AI alignment.

A standard tool for this kind of problem is Nash bargaining. The main problem with it is that it's unclear how to choose the BATNA (disagreement point). Here's why some simple proposals don't work:

• One natural BATNA for any game is assigning each player their maximin payoff. However, for a group of humans it means something horrible: Alice's maximin is a situation in which everyone except Alice are doing their best to create the worst possible world for Alice. This seems like an unhealthy and unnatural starting point.
• Another natural BATNA is the world in which no humans exist at all. The problem with this is: suppose there is one psychopath who for some reason prefers everyone not to exist. Then, there are no Pareto improvements over the BATNA, and therefore this empty world is already the "optimum". The same problem applies to most choices of BATNA.

Here is my proposal. We define the socially optimal outcome by recursion over the number of people $n$. For $n=1$, we obviously just optimize the utility function of the lone person. For a set of people $P$ of cardinality $n>1$, let's consider any given $i\in P$. The BATNA payoff of $i$ is defined to be the minimum over all $j\in P$ of the payoff of $i$ in the socially optimal outcome of $P\setminus j$ (we consider worlds in which $j$ doesn't exist). If there are multiple optimal outcomes, we minimize over them. Typically, the minimum is achieved for $j=i$ but we can't just set $j=i$ in the definition, we need the minimization in order to make sure that the BATNA is always admissible^[1]. We then do Nash bargaining with respect to this BATNA.

Good properties of this proposal:

• The outcome is Pareto efficient. It is also "fair" in the sense that the specification is rather natural and symmetric.
• The only especially strong assumption needed to make sense of the definition, is the ability to consider worlds in which some people don't exist^[2]. For example, we don't need anything like transferable utility or money. [EDIT: See child comment for a discussion of removing this assumption.]
• AFAICT threats don't affect the outcome, since there's no reference to minimax or Nash equilibria.
• Most importantly, it is resistant to outliers:
  • For example, consider a world with a set $S$ of selfish people and $1$ psychopath who we denote $y$. The outcome space is $2^{S\bigsqcup \left\{y\right\}}$: each person either exists or not. A selfish person gets payoff $1$ for existing and payoff $0$ for non-existing. The psychopath's payoff is minus the number of people who exists. Let $n$ be the cardinality of $S$. Then, we can check that the socially optimal outcome gives each selfish person a payoff of $\frac{n}{n+1}$ (i.e. they exist with this probability).
  • In the above example, if we replace the selfish people with altruists (whose utility function is the number of altruists that exist) the outcome is even better. The expected number of existing altruists is $(1-\frac{1}{(n+1)!})n$.

1. ^{^}

   "Admissible" in the sense that, there exists a payoff vector which is a Pareto improvement over the BATNA and is actually physically realizable.

2. ^{^}

   We also need to assume that we can actually assign utility functions to people, but I don't consider it a "strong assumption" in this context.