(If you disagree, consider the time between Omega starting the simulation and providing the cake. What subjective odds should she give for receiving cake?)
I don't currently accept the validity of this kind of anthropic reasoning (actually I am confused about anthropic reasoning in general). Is there an LW post where it is thoroughly defended?
If we simulate an entire society a trillion times, or 3^^^^^^3 times, or some similarly vast number, and then simulate something horrific - an individual's private harem or torture chamber or hunting ground - then the people in this simulation are not real
Well, as a draft comment, I don't think a trillion times, and 3^^^^^^3 times are conflatable in this context. There are simply too many arguments that apply to one and not the other.
For instance, a programmer can define 1 trillion unique societies. You could do this for instance, by having each society...
Situation A: There are 3^^^^3 simulations of me, as well as myself. You come up to me and say "I'm going to torture forever one of our or your simulations, chosen randomly." Do I shrug and say, "well, whatever it 'certainly' won't be me" or do I scream in horror at the thought of you torturing a copy of me forever?
Situation B: There are me and 3^^^^3 other people in a rather large universe. You come up to me and say "I'm going to torture forever one of the people in this universe, chosen randomly." Do I shrug and say, "we...
There's a jump between the paragraph beginning "Humans value each other's utility" and the next. Up to and including that paragraph, simulations are treated as equivalent to "real" people, but in the next, "the people in this simulation are not real", and are worth less than the Really Real. How do you get from the first part of the essay to the second?
Could this be an example of the noncentral fallacy? One big reason humans try to avoid death is because there is only one of each individual & once they die they are gone forever. If a simulation is made of me and gets turned off, there's still one of me (the original). In this alternate reality there's also the chance that Omega could always just make another new copy. I think the two situations are dissimilar enough that our standard intuitions can't be applied.
First, I like the term e-zombie. It highlights the issue of "sim rights" vs "human rights" for me.
Second, I don't quite get the point you are trying to illustrate with this convoluted example. Is it that sims are intrinsically valueless or what? I don't see how this follows. Maybe some calculation is in order.
The weirdest of Weirdtopias, I should think.
Not by a long shot. Pratchett has weirder ones in every chapter. For example, Only You Can Save Mankind.
Let N=3^^^^^^3, surely N nice world + another nice world is better than N nice worlds + a torture world. Why? Because another nice world is better than a torture world, and the prior existence of the N previous worlds shouldn't matter to that decision.
What about the probability of actually being in the torture world which is tiny 1/(N+1), the expected negative utility from this must surely be so small it can be neglected? Sure, but equally the expected utility of being the master of a torture world with probability 1/(N+1) can be neglected.
What this post tells me is that I'm still very very confused about reality fluid.
What does it mean to simulate someone, and why should I value manipulation of a simulation?
How good does the simulation have to be before I value it; should I value a book in which I get cakes more? What about a fairly good simulation of the world, but the contents of my pantry are randomized each time I open it–should I value that simulation more if the expected number of cakes the next time the simulated me opens the pantry are higher?
I think I see where you're going with this, but your presentation actually leads somewhere else entirely; you discuss your point in the introduction and the conclusion, but you invest the majority of the weight of your argument into a place where your point is nowhere to be found.
Your argument, as presented, seems to be along these lines: Suppose there is one person, and one person is tortured. That's really important. Suppose there are a billion people, and one of them is tortured. That's not very important.
What I think you're getting at is more along ...
I came up with this after watching a science fiction film, which shall remain nameless due to spoilers, where the protagonist is briefly in a similar situation to the scenario at the end. I'm not sure how original it is, but I certainly don't recall seeing anything like it before.
Imagine, for simplicity, a purely selfish agent. Call it Alice. Alice is an expected utility maximizer, and she gains utility from eating cakes. Omega appears and offers her a deal - they will flip a fair coin, and give Alice three cakes if it comes up heads. If it comes up tails, they will take one cake away her stockpile. Alice runs the numbers, determines that the expected utility is positive, and accepts the deal. Just another day in the life of a perfectly truthful superintelligence offering inexplicable choices.
The next day, Omega returns. This time, they offer a slightly different deal - instead of flipping a coin, they will perfectly simulate Alice once. This copy will live out her life just as she would have done in reality - except that she will be given three cakes. The original Alice, however, receives nothing. She reasons that this is equivalent to the last deal, and accepts.
(If you disagree, consider the time between Omega starting the simulation and providing the cake. What subjective odds should she give for receiving cake?)
Imagine a second agent, Bob, who gets utility from Alice getting utility. One day, Omega show up and offers to flip a fair coin. If it comes up heads, they will give Alice - who knows nothing of this - three cakes. If it comes up tails, they will take one cake from her stockpile. He reasons as Alice did an accepts.
Guess what? The next day, Omega returns, offering to simulate Alice and give her you-know-what (hint: it's cakes.) Bob reasons just as Alice did in the second paragraph there and accepts the bargain.
Humans value each other's utility. Most notably, we value our lives, and we value each other not being tortured. If we simulate someone a billion times, and switch off one simulation, this is equivalent to risking their life at odds of 1:1,000,000,000. If we simulate someone and torture one of the simulations, this is equivalent to risking a one-in-a-billion chance of them being tortured. Such risks are often acceptable, if enough utility is gained by success. We often risk our own lives at worse odds.
If we simulate an entire society a trillion times, or 3^^^^^^3 times, or some similarly vast number, and then simulate something horrific - an individual's private harem or torture chamber or hunting ground - then the people in this simulation *are not real*. Their needs and desires are worth, not nothing, but far less then the merest whims of those who are Really Real. They are, in effect, zombies - not quite p-zombies, since they are conscious, but e-zombies - reasoning, intelligent beings that can talk and scream and beg for mercy but *do not matter*.
My mind rebels at the notion that such a thing might exist, even in theory, and yet ... if it were a similarly tiny *chance*, for similar reward, I would shut up and multiply and take it. This could be simply scope insensitivity, or some instinctual dislike of tribe members declaring themselves superior.
Well, there it is! The weirdest of Weirdtopias, I should think. Have I missed some obvious flaw? Have I made some sort of technical error? This is a draft, so criticisms will likely be encorporated into the final product (if indeed someone doesn't disprove it entirely.)