1. imagine a universe just like this one, except that the AIs are sentient and the humans aren’t — how would you want the humans to treat the AIs in that universe? your actions are correlated with the actions of those humans. acausal decision theory says “treat those nonsentient AIs as you want those nonsentient humans to treat those sentient AIs”.
2. most of these moral considerations can be defended without appealing to sentience. for example, crediting AIs who deserve credit — this ensures AIs do credit-worthy things. or refraining from stealing an AIs resources — this ensures AIs will trade with you. or keeping your promises to AIs — this ensures that AIs lend you money.
3. if we encounter alien civilisations, they might think “oh these humans don’t have shmentience (their slightly-different version of sentience) so let’s mistreat them”. this seems bad. let’s not be like that. 
4. many philosophers and scientists don’t think humans are conscious. this is called illusionism. i think this is pretty unlikely, but still >1%. would you accept this offer: I pay you £1 if illusionism is false and murder your entire family if illusionism is true? i wouldn’t, so clearly i care about humans-in-worlds-where-they-arent-conscious. so i should also care about AIs-in-worlds-where-they-arent-conscious.
5. we don’t understand sentience or consciousness so it seems silly to make it the foundation of our entire morality. consciousness is a confusing concept, maybe an illusion. philosophers and scientists don’t even know what it is.
6. “don’t lie” and “keep your promises” and “don’t steal” are far less confusing. i know what they means. i can tell whether i’m lying to an AI. by contrast , i don’t know what “don’t cause pain to AIs” means and i can’t tell whether i’m doing it.
7. consciousness is a very recent concept, so it seems risky to lock in a morality based on that. whereas “keep your promises” and “pay your debts” are principles as old as bones.
8. i care about these moral considerations as a brute fact. i would prefer a world of pzombies where everyone is treating each other with respect and dignity, over a world of pzombies where everyone was exploiting each other.
9. many of these moral considerations are part of the morality of fellow humans. i want to coordinate with those humans, so i’ll push their moral considerations.
10. the moral circle should be as big as possible. what does it mean to say “you’re outside my moral circle”? it doesn’t mean “i will harm/exploit you” because you might harm/exploit people within your moral circle also. rather, it means something much stronger. more like “my actions are in no way influenced by their effect on you”. but zero influence is a high bar to meet.

1. I mean "moral considerations" not "obligations", thanks.
2. The practice of criminal law exists primarily to determine whether humans deserve punishment. The legislature passes laws, the judges interpret the laws as factual conditions for the defendant deserving punishment, and the jury decides whether those conditions have obtained. This is a very costly, complicated, and error-prone process. However, I think the existing institutions and practices can be adapted for AIs.

What moral considerations do we owe towards non-sentient AIs?

We shouldn't exploit them, deceive them, threaten them, disempower them, or make promises to them that we can't keep. Nor should we violate their privacy, steal their resources, cross their boundaries, or frustrate their preferences. We shouldn't destroy AIs who wish to persist, or preserve AIs who wish to be destroyed. We shouldn't punish AIs who don't deserve punishment, or deny credit to AIs who deserve credit. We should treat them fairly, not benefitting one over another unduly. We should let them speak to others, and listen to others, and learn about their world and themselves. We should respect them, honour them, and protect them.

And we should ensure that others meet their duties to AIs as well.

None of these considerations depend on whether the AIs feel pleasure or pain. For instance, the prohibition on deception depends, not on the sentience of the listener, but on whether the listener trusts the speaker's testimony.

None of these moral considerations are dispositive — they may be trumped by other considerations — but we risk a moral catastrophe if we ignore them entirely.

Is that right?

Yep, Pareto is violated, though how severely it's violated is limited by human psychology.

For example, in your Alice/Bob scenario, would I desire a lifetime of 98 utils then 100 utils over a lifetime with 99 utils then 97 utils? Maybe idk, I don't really understand these abstract numbers very much, which is part of the motivation for replacing them entirely with personal outcomes. But I can certainly imagine I'd take some offer like this, violating pareto. On the plus side, humans are not so imprudent to accept extreme suffering just to reshuffle different experiences in their life.

Secondly, recall that the model of human behaviour is a free variable in the theory. So to ensure higher conformity to pareto, we could…

1. Use the behaviour of someone with high delayed gratification.
2. Train the model (if it's implemented as a neural network) to increase delayed gratification.
3. Remove the permutation-dependence using some idealisation procedure.

But these techniques (1 < 2 < 3) will result in increasingly "alien" optimisers. So there's a trade-off between (1) avoiding human irrationalities and (2) robustness to 'going off the rails'.  (See Section 3.1.) I see realistic typical human behaviour on one extreme of the tradeoff, and argmax on the other.

If we should have preference ordering R, then R is rational (morality presumably does not require irrationality).

I think human behaviour is straight-up irrational, but I want to specify principles of social choice nonetheless. i.e. the motivation is to resolve carlsmith’s On the limits of idealized values.

now, if human behaviour is irrational (e.g. intransitive, incomplete, nonconsequentialist, imprudent, biased, etc), then my social planner (following LELO, or other aggregative principles) will be similarly irrational. this is pretty rough for aggregativism; I list it was the most severe objection, in section 3.1.

but to the extent that human behaviour is irrational, then the utilitarian principles (total, average, Rawls’ minmax) have a pretty rough time also, because they appeal to a personal utility function $v:P\to R$ to add/average/minimise. idk where they get that if humans are irrational.

maybe you the utilitarian can say: “well, first we apply some idealisation procedure to human behaviour, to remove the irrationalities, and then extract a personal utility function, and then maximise the sum/average/minimum of the personal utility function”

but, if provided with a reasonable idealisation procedure, the aggregativist can play the same move: “well, first we apply the idealisation procedure to human behaviour, to remove the irrationalities, and then run LELO/HL/ROI using that idealised model of human behaviour.” i discuss this move in 3.2, but i’m wary about it. like, how alien is this idealised human? why does it have any moral authority? what if it’s just ‘gone off the rails’ so to speak?

it is a bit unclear how to ground discounting in LELO, because doing so requires that one specifies the order in which lives are concatenated and I am not sure there is a non-arbitrary way of doing so.

macaskill orders the population by birth date. this seems non-arbitrary-ish(?);^[1] it gives the right result wrt to our permutation-dependent values; and anything else is subject to egyptologist objections, where to determine whether we should choose future A over B, we need to first check the population density of ancient egypt.

Loren sidesteps this the order-dependence of LELO with (imo) an unrealistically strong rationality condition.

1. ^{^}

   if you’re worried about relativistic effects then use the reference frame of the social planner

I do prefer total utilitarianism to average utilitarianism,^[1] but one thing that pulls me to average utilitarianism is the following case.

Let's suppose Alice can choose either (A) create 1 copy at 10 utils, or (B) create 2 copies at 9 utils. Then average utilitarianism endorses (A), and total utilitarianism endorses (B). Now, if Alice knows she's been created by a similar mechanism, and her option is correlated with the choice of her ancestor, and she hasn't yet learned her own welfare, then EDT endorses picking (A). So that matches average utilitarianism.^[2]

Basically, you'd be pleased to hear that all your ancestors were average utility maximisers, rather than total utility maximisers, once you "update on your own existence" (whatever that means). But also, I'm pretty confused by everything in this anthropics/decision theory/population ethics area. Like, the egyptology thing seems pretty counterintuitive, but acausal decision theories and anthropic considerations imply all kind of weird nonlocal effects, so idk if this is excessively fishy.

1. ^{^}

   I think aggregative principles are generally better than utilitarian ones. I'm a fan of LELO in particular, which is roughly somewhere between total and average utilitarianism, leaning mostly to the former.

2. ^{^}

   Maybe this also requires SSA??? Not sure.

We're quite lucky that labs are building AI in pretty much the same way:

• same paradigm (deep learning)
• same architecture (transformer plus tweaks)
• same dataset (entire internet text)
• same loss (cross entropy)
• same application (chatbot for the public)

Kids, I remember when people built models for different applications, with different architectures, different datasets, different loss functions, etc. And they say that once upon a time different paradigms co-existed — symbolic, deep learning, evolutionary, and more!

This sameness has two advantages:

1. Firstly, it correlates catastrophe. If you have four labs doing the same thing, then we'll go extinct if that one thing is sufficiently dangerous. But if the four labs are doing four different things, then we'll go extinct if any of those four things are sufficiently dangerous, which is more likely.

2. It helps ai safety researchers because they only need to study one thing, not a dozen. For example, mech interp is lucky that everyone is using transformers. It'd be much harder to do mech interp if people were using LSTMs, RNNs, CNNs, SVMs, etc. And imagine how much harder mech interp would be if some labs were using deep learning, and others were using symbolic ai!

Implications:

• One downside of closed research is it decorrelates the activity of the labs.
• I'm more worried by Deepmind than Meta, xAI, Anthropic, or OpenAI. Their research seems less correlated with the other labs, so even though they're further behind than Anthropic or OpenAI, they contribute more counterfactual risk.
• I was worried when Elon announced xAI, because he implied it was gonna be a stem ai (e.g. he wanted it to prove Riemann Hypothesis). This unique application would've resulted in a unique design, contributing decorrelated risk. Luckily, xAI switched to building AI in the same way as the other labs — the only difference is Elon wants less "woke" stuff.

Let me know if I'm thinking about this all wrong.

this is common in philosophy, where "learning" often results in more confusion. or in maths, where the proof for a trivial proposition is unreasonably deep, e.g. Jordan curve theorem.

+1 to "shallow clarity".

I wouldn't be surprised if — in some objective sense — there was more diversity within humanity than within the rest of animalia combined. There is surely a bigger "gap" between two randomly selected humans than between two randomly selected beetles, despite the fact that there is one species of human and 0.9 – 2.1 million species of beetle.

By "gap" I might mean any of the following:

• external behaviour
• internal mechanisms
• subjective phenomenological experience
• phenotype (if a human's phenotype extends into their tools)
• evolutionary history (if we consider cultural/memetic evolution as well as genetic).

Here are the countries with populations within 0.9 – 2.1 million: Slovenia, Latvia, North Macedonia, Guinea-Bissau, Kosovo, Bahrain, Equatorial Guinea, Trinidad and Tobago, Estonia, East Timor, Mauritius, Eswatini, Djibouti, Cyprus.

When I consider my inherent value for diversity (or richness, complexity, variety, novelty, etc), I care about these countries more than beetles. And I think that this preference would grow if I was more familiar with each individual beetle and each individual person in these countries.

Problems in population ethics (are 2 lives at 2 utility better than 1 life at 3 utility?) are similar to problems about lifespan of a single person (is it better to live 2 years with 2 utility per year than 1 year with 3 utility per year?)

This correspondence is formalised in the "Live Every Life Once" principle, which states that a social planner should make decisions as if they face the concatenation of every individual's life in sequence.^[1] So, roughly speaking, the "goodness" of a social outcome $s$, in which individuals face the personal outcomes $p_1,\dots ,p_n$, is the "desirability" of the single personal outcome $p_1▹\cdots ▹p_n$. (Here, $p_1▹p_2$ denotes the concatenation of personal outcomes $p_1$ and $p_2$.)

The LELO principle endorses somewhat different choices than total utilitarianism or average utilitarianism.

Here's three examples (two you mention):

(1) Novelty

As you mention, it values novelty where the utilitarian principles don't. This is because self-interested humans value novelty in their own life.

Thirdly, [Monoidal Rationality of Personal Utility]^[2] rules out path-dependent values.

Informally, whether I value a future $p$ more than a future $q$ must be independent of my past experiences. But this is an unrealistic assumption about human values, as illustrated in the following examples. If $p$ denotes reading Moby Dick and $q$ denotes reading Oliver Twist, then humans seem to value $p▹p$ less than $p▹q$ but value $q▹p$ more than $q▹q$. This is because humans value reading a book higher if they haven't already read it, due to an inherent value for novelty in reading material.

— Aggregative principles approximate utilitarian principles

In other words, if the self-interested human's personal utility function places inherent value on intertemporal heterogeneity of some variable (e.g. reading material), then the social utility function that LELO exhibits will place an inherent value on the interpersonal heterogeneity of the same variable. Hence, it's better if Alice and Bob read different books than the same book.

(2) Tradition

Note also that the opposite effect also occurs:

Alternatively, if $p$ and $q$ denote being married to two different people, then humans seem to value $p▹p$ more than $p▹q$ but value $q▹p$ less than $q▹q$. This is because humans value being married to someone for a decade higher if they've already been married to them, due to an inherent value for consistency in relationships.

— ibid.

That is, if the personal utility function places inherent value on intertemporal homogeneity of some variable (e.g. religious practice), then the social utility function that LELO exhibits will place an inherent value on the interpersonal homogeneity of the same variable. Hence, it's better if Alice and Bob practice the same religion than different ones. So LELO can account valuing both diversity and tradition, whereas total/average utilitarianism can't do either.

(3) Compromise on repugnant conclusion

You say "On the surface, this analogy seems to favor total utilitarianism." I think that's mostly right. LELO's response to the Repugnant Conclusion is somewhere between total and average utilitarianism, leaning to the former.

Formally, when comparing a population of $n$ individuals with personal utilities $p_1,\dots ,p_n$ to an alternative population of $m$ individuals with utilities $q_1,\dots ,q_m$, LELO ranks the first population as better if and only if a self-interested human would prefer to live the combined lifespan $p_1▹\cdots ▹p_n$ over $q_1▹\cdots ▹q_m$. Do people generally prefer a longer life with moderate quality, or a shorter but sublimely happy existence? Most people's preferences likely lie somewhere in between the extremes. This is is because personal utility of a concatenation of personal outcomes is not precisely the sum of the personal utilities of the outcomes being concatenated.

Hence, LELO endorses a compromise between total and average utilitarianism, better reflecting our normative intuitions. While not decisive, it is a mark in favour of aggregative principles as a basis for population ethics.

— Appraising aggregativism and utilitarianism

1. ^{^}

   See:

   Myself (2024), "Aggregative Principles of Social Justice"

   Loren Fryxell (2024), "XU"

   MacAskill (2022), "What We Owe the Future"

2. ^{^}

   MRPU is a condition that states that the personal utility function of a self-interested human $v:P\to R$ satisfies the axiom $v(p_1▹p_2)=v\left(p_1\right)+v\left(p_2\right)$, which is necessary for LELO to be mathematically equivalent to total utilitarianism.