Content warning: Anthropics, Moral Philosophy, and Shrimp

This post isn't trying to be self contained, since I have so many disparate thoughts about this. Instead, I'm trying to put a representative set of ideas forward, and I hope that if people are interested we can discuss this more in the comments. I also plan to turn this into a (probably small) sequence at some point.

I've had a number of conversations about moral philosophy where I make some claim like 

Utility is bounded and asymptotically sublinear in number of human lives, but superlinear or ~linear in the ranges we will ever have to care about.

Common reactions to this include:

• "Wait, what?"
• "Why would that be the case?"
• "This doesn't make any sense relative to my existing conceptions of classical utilitarianism, what is going on here?"

So I have gotten the impression that this is a decently novel position and I should break it down for people. This is the post where I do that breakdown. As far as I know, this has not been written up anywhere else and is primarily my own invention, but I would not be terribly surprised if some commenter comes forward with a link describing an independent invention of the thing I'm pointing at.

I won't spend much of this post defending consequentialist utilitarianism, that is not what I'm here to do. I'm just here to describe a way that values could be that seems academically interesting and personally compelling to me, and that resolves several confusions that I once had about morality.

I'll start out with some motivating thought experiments and math and carefully work my way into the real world I live in, since this view takes a bit of context to explain. Some of these pieces may seem disjoint at first until I bring them together.

Aside: "Utilitarianism"

I'm using "consequentialist utilitarianism" here to refer to the LessWrong thing that Eliezer uses to model ideal agents and also basically the thing that economics uses, not the classical philosophy thing or anything else by that name. 

The More Mathy Pointer

$U:\text{outcomes}\to R$

An outcome is the way that a universe could go over time, like "the big bang happened, (...) there was all sorts of sentient life that emerged, then they all cooperated and did beautiful things" or any other timeline that is possible, among the set of ways the universes could be. This set is very very large. A utility function is a function from the set of possible outcomes to the real numbers, and an ideal consequentialist utilitarian agent has a^[1] utility function such that it can be modeled as maximizing that function. 

Some people ascribe other features to utilitarianism beyond these, but that is not what I'm doing. I'm just looking at the possible timelines a world could go through and giving them each a score. 

Duplicate Simulations

If you run a perfectly accurate and robust simulation of me in some friendly environment on a computer, this is great and I would pay money to have this happen. A place where I find myself disagreeing with others pretty frequently, however, is that I would not pay for a second computer right next to the first running the exact same sequence of computations. Recall that, by assumption, it doesn't add any extra assurance that the simulation keeps going, it's just of the same thing with the same reliability.

My moral intuition says: the simulation is already running once, the computation is already happening, and no parts of the universe are going to get information about my existence from the second marginal computer. Paying for this seems as intuitively absurd to me as doubling up each transistor on an existing computer chip, just to make the computations done on it have more moral weight. 

Another angle is explained at length in one of the classic planecrash lectures (spoilers): it is not meaningful for a universe to become twice as real; realness is a relative metric, not an absolute one. You can have a quantum experiment that outputs one result 2/3 of the time, and that outcome is in some sense twice as real as the 1/3 outcome, but you would still get the exact same results if you made every outcome "twice as real," and so I think it doesn't make sense to think of making something "twice as real" as a meaningful action at all. 

The same thing drives my intuition here: it doesn't really make sense to want to make this simulation twice as real . In fact, the only reason that it is reasonable to pay for even the first simulation is because the people in this universe, where that Me wouldn't otherwise exist, can now observe what that Me is doing. That Me already existed in some sense in the background, but now the utility of being observed is added on. 

If you have both computers broadcasting their identical computations out into space (once again with zero signal noise or lossiness), then I will pay money for both if they are outside each other's lightcones, since this implies that there are Me's making contact with different sets of interesting things, and in expectation this probably aggregates into more things knowing about Me, and that's cool and valuable.

To summarize the things we now have:

• The value in simulations of people and their worlds comes from newly being able to observe those worlds, not from bringing them into existence in the first place.
• Duplicate simulations don't count for any marginal value unless they add to the observability of that world (e.g. by being in substantially different locations).

Slightly Different Simulations

Put the computers back next to each other, and perturb one of the simulations from Me into Me*. Not enough to turn me into a wildly different person, but enough to change some minor personal preference, like reversing my preference ordering between blue and yellow shirts. I will now pay a positive amount of money for this marginal simulation, and I tend to have more willingness to pay the more you perturb the simulation while maintaining some things I care about like the simulatee being human-shaped and generally pleased with their situation.

What is going on with the "I tend to have more willingness to pay the more you perturb the simulation" part? Why would the value of a being^[2] go up the more it's different from what you already have access to? 

I tend to conceptualize what I value about life around me is the amount of cool computations going on: the human brain does so many interestingly varied and wonderful things, and there is a huge amount of individual character and distinction between individuals. However, I've already established that I have pretty limited conditions for valuing sameness of people or worlds that exclude duplicate simulations right next to each other but include things that are differently observable, like real people on Earth who are very similar to each other.

When I imagine a world where everyone is exactly the same person, though, I feel like we are missing out on this big space of possible human experience, instead only filling a small fraction of it:

Going back to simulating slightly different versions of Me on computers, I feel like they are covering meaningfully different parts of the space, but still have some overlap:

Utility Variation with Population

We now have enough pieces to paint a picture of how much I value different numbers of humans (or similar things). When I think about what I value in humanity or in any other animal that is at least epsilon valuable, there are a few factors that I care about that matter:

More is Better

Assuming that you can keep living conditions good, it seems broadly good to have more humans rather than fewer of them in a given world. They cover more of the space of valuable computations.

In Some Domains, More is Superlinearly Better

"Some Domains" being ones like "having fewer than a quintillion^[3] humans" and whatnot. There are things that can only happen once you reach a certain number of humans in a world, and those things have extra value: conversation takes two, doing some projects takes tens to thousands, building some moderately advanced civilizations takes billions, and so does having such rich and diverse culture as we do today. I don't know what exactly can happen with a trillion humans, but probably many cool things that can't be done now. 

But Value Space is Limited

There are only so many possible human shaped computations that are valuable to me, and while I'm pretty confident that we're not anywhere near the limit of that yet, I wouldn't be so confident saying that I'll still believe this into the quintillions or the many-more-than-that-illions. At some point, things fill up, and each marginal person isn't going to be adding particularly new computations to the mix:

I want to emphasize, this is a very very big space of human-shaped valuable things and I don't think we will run into the problem.

What About The Other Animals?

Consider two additional types of animals beyond just humans: shrimp and galaxy-brained post-humans like the ones here. 

I take a pretty cosmopolitan view on one part of this, which is that the shrimp-computations aren't literally the human computations, but I care about those computations similarly much even though there is less moral weight per organism because they have much much smaller brains. There is something in shrimp cognition that I care about, since having more different minds in the universe is cool. Despite having small brains, there is still probably some meaningful inter-shrimp variation that makes things . 

The post-humans, on the other hand, have very large experiences consisting of things that humans can't even dream of, experiencing worlds that cannot sustain humans and generally having vastly more valuable experience. 

Several things happen with my value of these three types of beings:

• The spaces of value for each species is going to be radically different, and thus the horizontal asymptotes, with post-humans>humans>shrimp, the way I've set it up.
• The utility with respect to each species increases at different rates: this makes sense because an individual human is worth more than an individual shrimp.
• The space of value for shrimp is much of the way filled with many fewer individuals than the value space for humans, even though the space for humans is bigger. 

What Does this Mean About Classical EA?

Classical EA philosophy takes the approach of "just multiply the number of things impacted by the unit impact," and this works pretty well for specific charitable goals like saving the greatest number of human lives, since we're basically at linearity in value with respect to human lives, under my values. 

This doesn't seem to work very well for things like shrimp lives. Shrimp brains are small and I imagine are mostly managing muscle outputs like "swim this way because there is food over there" and "eat food" and things like that that are going to be widely shared over the vast majority of the species. I really don't think there is much variation going on there, and I don't think a marginal shrimp killed is going to meaningfully empty the space of shrimp-computation-value that already exists.

I still think that people like the shrimp welfare project are doing something positive, to be clear, they are preventing new kinds of shrimp suffering from coming about, and that seems good. It's not at all an effective cause relative to my values, but it is not nothing.

Other Curiosities

I might explore some of these other topics in future posts, and am happy to try to discuss them in the comments:

• The utility function is bounded
• I'm a positive utilitarian
  • as in, I think quite a lot of outcomes are better than nonexistence (this is more centrally what the term "positive utilitarianism" means.)
  • as in, I like quite a lot of things, and this feels related to why the other meaning is true. 

I will fill in links to these posts as I write them. 

1. ^{^}

   unique up to positive affine transformation

2. ^{^}

   Technically "the value of the marginal observability of this being to a universe that does not already contain them" but this is annoyingly long to say every time, and I think that this isn't actually necessary for the post to be fully clear.

3. ^{^}

   I don't actually have a number for this, it just seems like it is a number that is obviously much bigger than the number of humans who have ever lived but less than the number of humans who could in theory one day exist.