I notice some confusion about the next action here – I'm assuming there are some writeups comparing failure modes of prioritarianism vs other utilitarian-esque philosophies, but I'm imagining them being fairly dense. And even if I made it all the way through them... not being quite sure what the confusions are most important to resolve.
I am unsure whether I get the starting assumtions right but taking them to the extreme it seems contradictory enough that it should give pause for thought.
There are 1,000,000,000,000,000 people. Stick all of the people with needles or give all the people a candy. It would seem the negative gets multiplied by the subject amount but the positive doesn't. Adding another person that receives candy makes you stay at +1 candy but adding another person to be needled would seem to increase suffering.
I can understand if you care about the impact to the world instead of your subjective qualia (ie refuse pill that makes you artificially happy but doesn't improve the world) and I can understand a viewpoint where its all about your subjective wellbeing. But I would think you should use the same model for positive and negative things or that you are not taking into account subjective positivity and objective negativity.
If the Universe is infinite, every positive experience is already instantiated once. This view could then imply that you should only focus on preventing suffering. That depends somewhat on exactly what you mean with "I" and "we", though, and if you think that the boundary between our lightcone and the rest of the Universe has any moral significance.
You apparently believe that being identical (dis)counts for positive experiences only:
By contrast, if there is one person out there experiencing suffering, that is sad. And if there are two it's twice as sad, even if they have identical experiences. And if there are 1,000,000,000,000,000 it's 1,000,000,000,000,000x as sad, even if they're all identical.
That doesn't match my intuition at all. I tend to agree with Scott Aaronson's approach in The Ghost in the Quantum Turing Machine, section 2.13:
I’m against any irreversible destruction of knowledge, thoughts, perspectives, adaptations, or ideas, except possibly by their owner. Such destruction is worse the more valuable the thing destroyed, the longer it took to create, and the harder it is to replace. From this basic revulsion to irreplaceable loss, hatred of murder, genocide,the hunting of endangered species to extinction, and even (say) the burning of the Library of Alexandria can all be derived as consequences.
A corollary for me is that both happiness and suffering are additive in non-identical parts only. This doesn't really matter in the world we live in, because no two people are absolutely identical, and at present there is no easy way to calculate a difference in subjective experiences even if their observed reactions appear the same. But this may change, hypothetically, if truly identical minds can be created. My intuition is that there is no moral weight difference between 1 and 1 million of identical minds.
I've updated a bit over time to "okay, maybe I can just thoroughly accept that identical minds matter identically", but I still feel a lot of resistance to that.
One thing this comment made me realize is that I feel different intuitions re: "there are a million identical people. What are the respective utility of giving one of them a candy bar, vs all of them, vs one of them getting a pinprick, vs all of them?"
vs
"You have spare compute lying around, which you might use to run a one person getting a candy, or a million, or one person getting a pinprick, or a million. How do these compare?"
What do you think about the argument that the Universe might well be infinite, and if so, your view means that nothing we do matters, since every brainstate is already instantiated somewhere? (Taken from Bostrom's paper on the subject.)
That's a good question, and a good example of how decompartmentalizing can lead one astray. I warmly recommend the rest of the section 2 in the above paper. Scott points out in “Could a Quantum Computer Have Subjective Experience?” that the argument can be made without any physics involved as every brain state has a finite description, and those descriptions can be enumerated, and every number can already be found in the decimal expansion of Pi, so what's the point of anything? One objection is "but it's not real unless it happens somewhere", also addressed there.
In contrast I personally lean towards non-negative individual utility function - ignore all suffering and count only good things.
Someone had to experience the worst possible timeline that is still worth simulating. Thankfully, you volunteered. The rest of the Raemon-verse considers you a hero.
Romeo and I ended up chatting. Some takeaways I could remember a few days after the fact:
There are a few levels of abstraction that I could have engaged with, when I was thinking about this question
One is the 'object-level' of 'is this particular variant of utilitarianism cogent?'
A higher level question is 'notice that I am the sort of process that generated this question. Why did I do that? What elements of the ancestral environment generated these intuitions?"
In the ancestral environment, there *is* something of an upper bound of how good things can possibly get – I can be prosperous and have lots of children. But my adaptation-executor-process sees that as way-less-good than getting killed forever is bad. Some of my intuitions with the OP might be a carryover of that. How should I think about that?
Text is a bit laborious. Happy to chat about various simulation/multiverse parameters some time if it seems alive.
You phrased your thought experiments in terms of good and bad lives. But what about the good and bad parts of a life? When you imagine a lot of good lives, you say that the value doesn't add up. But what about the bad parts of those lives? Do you imagine that still accumulates and wipes out the value in the good parts?
First a couple background frame things:
1.
One important caveat I didn't specify is that I'm seeing this entire frame through the lens of "this is what we worry about after we've won. At the very least, once there's been a major transition to 'digital minds', (where identical lives start to matter), and possibly, once we generally control as much of the lightcone as we were going to control in a stable fashion.
And the question is less formed around "how do we optimize individual lives", and more formed around "in the limit, what do we do when it comes to what-sort-of-lives we create, and what sort of societies-that-allow-for-and-create-lives do we create. And possibly 'if we have the ability to intervene in other places where minds exist, either causally or through acausal trade, how do we want to influence that?"
2.
I'm happy with weird hacks that say 'don't take any of this too seriously and if you're considering either destroying the universe or doing weird goodharty things to solve the problem Repugnant Solution style, please stop, build yourself a Jupiter Brain, and think for a thousand years first (obviously spending the first year or so thinking about how to remain sane while being a Jupiter Brain for 1000 years). Meanwhile, try not to take drastic actions.
...
With that in mind, I think my answer to your question is "I think [very weakly] that it makes more sense to check whether a being endorses having lived", rather than total experiences (with some caveats of I can imagine beings designed in a sadistic fashion to endorse having lived through torture and I might have complicated opinions about how to treat those being's ability to endorse things)
An alternate hypothetical scheme might be, for a given life, to treat each moment as a "life" that ranges from birth-to-present-moment. Say a being is born, has a good early life, then experiences some hardship that makes them suicidal, then later thinks that hardship was worthwhile in some way. You might treat them as an 1000s of lives rather than one, and ask each one if they endorse having been born. You might add an extra step of, for the instance where they're suicidal, communicate* to them that in the future they endorsed having lived through the hardship.
If they *still* endorse suicide, I'd probably weight that 'life' as a negative value in the scale. (i.e. if of 100 time slices of a person, 70 of those time slices endorse having lived and 30 don't, even after simulating them with some time for meta reflection on the whole thing and thinking about the game theory of what they endorse, then that counts as 70 good things and 30 bad things on the cosmic scale)
Important clarification: The point of this question is not "if these premises are true, what actions should we take?", but "These premises seem suspicious, and I'd like good critiques of why they don't hold up, or help deconfusing them, or... something. I'm not sure."
The text I added to my post was:
This seems surprising and confusing and probably paradoxical. But I've reflected on it for a month and the intuition seems reasonably stable.
I can't tell if it's more surprising and paradoxical than other various flavors of utilitarianism, or other moral frameworks. Sometimes intuitions are just wrong, and sometimes they're wrong but pointing at something useful, and it's hard to know in advance.
I'm looking to get a better sense of where these intuitions come from and why. My goal with this question is to basically get good critiques or examinations of "what ramifications would this worldview have", which can help me figure out whether and how this outlook is confused. So far I haven't found a single moral framework that seems to capture all my moral intuitions, and in this question I'm asking for help sorting through some related philosophical confusions.
I'm especially skeptical about intuitions very different from past or current situations. In me, there's an availability heuristic where I incorrectly visualize some examples that are too close to reality, use my intuitions for those, then try to apply a corrective factor.
I don't have a good model for moral value of near-identical copies. My current belief is that quantum-level duplication makes them one being, and any other level of similarity diverges fast enough to make them distinct entities, each with full weight. I definitely don't think that positive and negative experiences are different on this topic.
This goes directly to the idea of "done it all". There are a (literally) infinite number of variations of experience, and we can never run out. There'll always be good experiences that we don't have enough consciousnesses to have. There is no maximum high score (except possibly maximum temporal density of reachable experiences until the end of reality). You can never step in the same river twice, and entities can never have the same experience twice.
I MAY share your intuition (or I may be misreading) that there is more negative than positive in entity-experience space. My take from that is that we need to be somewhat careful in creating more experiences, but not that it needs to be perfect, nor that prioritizing elimination of negative overwhelms creation of positive.
Edit: I do think I understand where the "repeated negative is bad, repeated positive is neutral" intuition comes from. In my personal life, I prefer variety of positive experiences over repetition. My hedonic adaptation seems to be faster (and perhaps more complete a regression to the norm) for good things than for pain. I don't think that reasoning applies to cases of distinct-but-similar individuals in the same way it does for distinct-but-similar experiences in an individual.
This assumes you've read some stuff on acausal trade, and various philosophical stuff on what is valuable from the sequences and elsewhere. If this post seems fundamentally confusing it's probably not asking for your help at this moment. If it seems fundamentally *confused* and you have a decent sense of why, it *is* asking for your help to deconfuse it.
Also, a bit rambly. Sorry.
Recently, I had a realization that my intuition says something like:
[edited to add]
This seems surprising and confusing and probably paradoxical. But I've reflected on it for a month and the intuition seems reasonably stable.
I can't tell if it's more surprising and paradoxical than other various flavors of utilitarianism, or other moral frameworks. Sometimes intuitions are just wrong, and sometimes they're wrong but pointing at something useful, and it's hard to know in advance.
I'm looking to get a better sense of where these intuitions come from and why. My goal with this question is to basically get good critiques or examinations of "what ramifications would this worldview have", which can help me figure out whether and how this outlook is confused. So far I haven't found a single moral framework that seems to capture all my moral intuitions, and in this question I'm asking for help sorting through some related philosophical confusions.
[/edit]
[1] or, positive experiences might be infinite, but a smaller infinity than the negative ones?
Basically, when I ask myself:
Once we've done literally all the things – there are as many humans or human like things that could possibly exist, having all the experiences they could possibly have...
...and we've created all the mind-designs that seem possibly cogent and good, that can have positive, non-human-like experiences...
...and we've created all the non-sentient universes that seem plausibly good from some sort of weird aesthetic artistic standpoint, i.e. maybe there's a universe of elegant beautiful math forms where nobody gets to directly experience it but it's sort of beautiful that it exists in an abstract way...
...and then maybe we've duplicated each of these a couple times (or a couple million times, just to be sure)...
...I feel like that's it. We won. You can't get a higher score than that.
By contrast, if there is one person out there experiencing suffering, that is sad. And if there are two it's twice as sad, even if they have identical experiences. And if there are 1,000,000,000,000,000 it's 1,000,000,000,000,000x as sad, even if they're all identical.
Querying myself
This comes from asking myself: "do I want to have all the possible good experiences I could have?" I think the answer is probably yes. And when I ask "do I want to have all the possible good experiences that are somewhat contradictory, such that I'd need to clone myself and experience them separately" the answer is still probably yes.
And when I ask "once I have all that, would it be useful to duplicate myself?" And... I'm not sure. Maybe? I'm not very excited about it. Seems like maybe nice to do, just in as a hedge against weird philosophical confusion. But when I imagine doing that the millionth time, I don't think I've gotten anything extra.
But when I imagine the millionth copy of Raemon-experiencing-hell, it still seems pretty bad.
Clarification on humancentricness
Unlike some other LessWrong folk, I'm only medium enthusiastic about the singularity, and not all that enthusiastic about exponential growth. I care about things that human-Ray cares about. I care about Weird Future Ray's preferences in roughly the same way I care about other people's preferences, and other Weird Future People's preferences. (Which is a fair bit, but more as a "it seems nice to help them out if I have the resources, and in particular if they are suffering.")
Counterargument – Measure/Magical Reality Fluid
The main counterargument is that maybe you need to dedicate all of the multiverse to positive experiences to give the positive experiences more Magical Reality Fluid (i.e. something like "more chance at existing", but try not to trick yourself into thinking you understand that concept if you don't).
I sort of might begrudgingly accept this, but this feels something like "the values of weird future Being That Shares a Causal Link With Me", rather than "my values."
Why is this relevant?
If there's a finite number of good experiences to have, then it's an empirical question of "how much computation or other resources does it take to cause them?"
I'd... feel somewhat (although not majorly) surprised, if it turned out that you needed more than our light cone's worth of resources to do that.
But then there's the question of acausal trade, or trying to communicate with simulators, or "being the sort of people such that whether we're in a simulation or not, we adopt policies such that alternate versions of us with the same policies who are getting simulated are getting a good outcome."
And... that *only* seems relevant to my values if either this universe isn't big enough to satisfy my human-values, or my human values care about things outside of this universe.
And basically, it seems to me the only reason I care about other universes is that I think Hell Exists Out There Somewhere and Must Be Destroyed.
(Where "hell" is most likely to exist in the form AIs running incidental thought experiments, committing mind-crime in the process).
I expect to change my mind on this a bunch, and I don't think it's necessary (or even positive EV) for me to try to come to a firm opinion on this sort of thing before the singularity.
But it seems potentially important to have *meta* policies such that someone simulating me can easily tell (at lower resolutions of simulation) whether I'm the sort of agent who'd unfold into an agent-with-good-policies if they gave me more compute.
tl;dr – what are the implications of the outlook listed above? What ramifications might I not be considering?