I had some vague thoughts about scaling in utilitarian-ish ethics, noticed I was confused about where it lead, and I thought it might be nice to present them here, hear where I'm wrong, and learn where I'm repeating existing work. The only prior discussion I could immediately find was in this comments section. I don't have any good textbooks on ethics handy to actually review the literature. I look forward to learning what I'm missing. I also think I'm conflating terms like "moral value" and "utility" and so on, sorry about that. 

Presented as a series of intuition pumps, hopefully without much jargon. 

Uniqueness

Utilitarians (whether moral realists, or just trying to fulfill their own values as effectively as possible) often assume that moral value (utility) scales linearly. Twice as much pleasure is twice as good; twice as much suffering is twice as bad. But does that make sense? 

Intuition pump 1: God with a rewind button

God (or whichever grad student runs the simulation I live in) watches me experience the worst 10 minutes of my life. Just awful. They rewind the (deterministic) universe 10 minutes, and watch it over a couple times. 

There's a sense in which there was twice as much suffering. But have I been harmed? Or was my duplicated suffering an indistinguishable state, carrying no additional moral weight? 

Intuition pump 2: Transporter clones

If my transporter clone is deconstructed (a euphemism for 'killed') in the first 0.01 nanoseconds of transport, I feel fine with that. If they're killed after a full day, that feels less-good - they've become a unique person with unique moral value. Individual moral value seems like it might scale super-linearly-ish with the unique experiences of the transporter clone, and that moral value quickly becomes indistinguishable with the moral value of a totally unique person. Meanwhile, the moral weight of each individual clone might scale sublinearly when they're very similar to each-other - between t=0 and t=0.01, there are two of me, but they might only sum to slightly more than one me-util - certainly not two me-utils. 

So uniqueness of moral joy or suffering seems really important, in the extreme edge cases. 

Moral weight

The moral weight of animals is often calculated linearly too - with something like neuron count as a common 'good-enough' proxy measurement. To hear it put in smarter words than I've got: 

For the purposes of aggregating and comparing welfare across species, neuron counts are proposed as multipliers for cross-species comparisons of welfare. In general, the idea goes, as the number of neurons an organism possesses increases, so too does some morally relevant property related to the organism’s welfare. Generally, the morally relevant properties are assumed to increase linearly with an increase in neurons, though other scaling functions are possible.

- Adam Shriver, rethinkpriorities.org

Let's take all but the last sentence as assumptions, for now. Why linear scaling? Does that make sense? 

Intuition pump 3: Neuron subdivision

Chop my brain up (all 8.6e10 neurons, or if you're a synapse guy, all 1.5e14 of those), and put some minimal set of neurons/synapses into each of a billion or trillion microscopic bio-robots. Each bio-robot uses those neurons for control, is taught to like moving in the direction of a chemical gradient, and let loose to have very fulfilling (if nearly identical) lives. (Fulfilling for microbes, of course). 

Does that colony of microbes have equivalent moral weight to the person I used to be? Does your utility function say that no net harm has been done? 

What about 500k-ish bee-level bio-robots? Is there any subdivision that carries equivalent moral weight to the person I once was? (aside: maybe splitting in two, across the corpus callosum (split-brain syndrome), counts as two independent people, under egalitarianism as a normative value?) 

Now, what the heck is going on? 

If I accept the output of the uniqueness intuition pumps ("it's okay for god to rewind the deterministic universe" and "it's mostly okay for transporters to kill the original after small t"), then I'd have to believe that N exactly identical moral patients have a constant moral weight. And more generally, that the total utility of N moral patients scales as a constant if they're identical, a linear function of N if they're totally unique, and a sublinear function of N if they're very similar. 

If that's true, what the heck am I doing when computing utility under probability distributions?? 

Scenario: In 99 out of 100 parallel worlds, I gain a dollar; and in 1 out of 100 worlds I pay a dollar. Aren't the 99 copies of me identical? Didn't I just agree that, therefore, they have constant utility, no matter how many of them there are? I was just trying to be a more consistent utilitarian - did I give up the ability to perform utility calculations at all? 

Okay, maybe the 99 copies aren't identical. After all, for it to be a physical probability distribution, there would have to be some states in the system that aren't identical (copy 33 gets to be a little unique if they get to see number '33' on a hundred-sided die). But I can imagine arbitrary degrees of similarity between those 99 copies, in the same way that the transporter clone can achieve arbitrary degrees of similarity with the original by being vaporized closer and closer to t=0. Just lock the die (and die-reading computer) in a box. The sturdier the box, the more similar the 99 copies. 

But maybe parallel worlds don't exist, and let's say nothing about branch-counting and the Born rule. So maybe 'probability' (P) and 'count' (N) aren't interchangeable. But isn't that the frequentist assumption? The law of large numbers says probability-over-one-agent (P) and count-over-many-agents (N) converges. P and N are interchangeable. 

So it seems like I've lost - if I want to have a sub-linear total moral value for N very similar moral patients, then I can't have a utility that scales linearly with probability P over similar universes, and vice versa. 

I don't see how to keep all three of (utilitarianism, probability, and moral weight as a function of uniqueness). 

So, choose one

1. Expected utility (probability times utility) doesn't work to decide moral value under utilitarianism. 
2. 'Probability' and 'count' are not equivalent in the limit. The law of large numbers is wrong, or somehow does not apply here. 
3. The total moral weight of N agents scales linearly, no matter how similar those agents are. Killing your transporter clone is muder. 

Maybe someone smarter than me can figure out how to give up 1., maybe with a risk-aversion-type argument, or an appeal to prioritarianism over egalitarianism. I don't see it. It's also very possible that, if I had a good intuition for SSA/SIA, this confusion would resolve itself. 

But I'm not smarter than me. So, until I become smarter, I'll have to give up 3. The intuition pumps above felt somewhat persuasive when I wrote them, but I must have been wrong. 

I'm ready for my brain to be chopped up into a trillion nearly-identical bio-robots, doctor.