I agree with everyone else: superposition with rocks does not produce "garbage" in any sense that would make a person-simulation sufferingless if it wasn't before.
In particular, note that for the particular case of quantum superposition, what you are describing is basically the same thing that in other contexts goes by the name of "many worlds": the person-simulation and the rocks-simulation evolve independently enough that it's at least defensible to think of them as happening in separate universes.
If "the operator of calculating future-time snapshots is linear", it'd be very hard for me to consider X = A(S) + R "garbage".
From the point of view of physics, it contains garbage,
But a miracle occurs, and your physics simulation still works accurately for the individual components...?
I get that your assumption of "linear physics" gives you this. But I don't see any reason to believe that physics is "linear" in this very weird sense. In general, when you do calculations with garbage, you get garbage. If I time-evolve a simulation of (my house plus a bomb) for an hour, then remove all the bomb components at the end, I definitely do not get the same result as running a simulation with no bomb.
Well, actually, physics appears to be perfectly linear... if you work purely quantum level. In which case adding R is just simulating R, and also simulating you, pretty much independently. In which case no, it isn't garbage. It's two worlds being simulated in parallel.
It seems to me you are taking my assumption of linearity on the wrong level. To be exact, I need the assumption of linearity of the operator of calculating future-time snapshots (fixed in the article).
This is entirely different from your example.
Imagine for example how the Fourier Transform is linear as an operation.
OK. I think I see what you are getting at.
First, one could simply reject your conclusion:
However at no point did I do anything that could be described as "simulating you".
The argument here is something like "just because you did the calculations differently doesn't mean your calculations failed to simulate a consciousness". Without a real model of how computation gives rise to consciousness (assuming it does), this is hard to resolve.
Second, one could simply accept it: there are some ways to do a given calculation which are ethical, and some ways that aren't.
I don't particularly endorse either of these, by the way (I hold no strong position on simulation ethics in general). I just don't see how your argument establishes that simulation morality is incoherent.
http://lesswrong.com/lw/jcj/an_ethical_puzzle_about_brain_emulation/
Yes. Thanks. I can tell that guy was thinking the same thing as me.
Sadly, I found no answers in the other thread that succeed at resolving the paradoxes.
You've encrypted a brain, and maybe salted it a bit to boot. You're still running the brain's "consciousness" program, it's just encrypted, and the brain is still experiencing exactly the same things, on account of it is running exactly the same program it would otherwise. The fact that the brain is cryptographically entangled with other data doesn't make the brain not exist.
I think this is probably the correct answer. If a simulation obtains the correct result, it is simulating the mind in some form, even while shuffling computations between obfuscatory boxes. The notion that adding physical states somehow changes this is a red herring. If I translate you 10 m to the left, you don't stop being a mind.
If a simulation obtains the correct result, it is simulating the mind in some form
However, this conclusion doesn't have implications for morality: if I (as a human) think about how'd feel if I torture you, I get the correct answer. It doesn't make my thinking a condemnable act.
Except you don't get the correct answer when you think about it. Your simulation is incredibly abstracted, and your model of the victim does not have much, if any, moral weight. If you had the ability to calculate a precise mindstate, that's the level of simulation that would require moral calculus.
I may be in the minority, but I reject "wrong" and "immoral" as boolean values. You have to quantify the effects on your world of knowing the results of the weighted range of experiments and compare to the "moral cost" of causing pain for a being (again, the net value of that hour of created life, weighted by probability). But say I take your point and agree that the net value of the range of your experiments is negative, and you "should" not perform them.
The computations in the second situation have the same fidelity and granularity as the first, and are morally equivalent. Your X is bigger (and therefore takes more to compute) than A(S), exactly because it contains both worlds. At any point in the physics simulation, you can subtract out R_t and get A(S)_t.
The second is morally reprehensible to exactly the same degree as the first.
The computations in the second situation have the same fidelity and granularity as the first, and are morally equivalent. Your X is bigger (and therefore takes more to compute) than A(S), exactly because it contains both worlds. At any point in the physics simulation, you can subtract out Rt and get A(S)t.
OK, but this only moves the problem to a different place. Say I can find an operation that let's me transform the data irreversibly, in such a way that I can still get my computations and answers done, but no way of getting the original states back. What would you say then?
No part of his objection hinged on reversibility, only the same linearity assumption you rely on to get a result at all.
My feeling about this is that it's okay to have some degree of arbitrariness in our preferences - our preferences do not have a solid external foundation, they're human things, and like basically all human things will run into weird boundary cases when you let philosophers poke at them.
The good news is that I also think that hard-to-decide boundary cases are the ones that matter least, because I agree with others that moral uncertainty should behave a lot like regular uncertainty in this case (though I disagree with certain other applications of moral uncertainty).
The unfortunate thing about simulation as a 'hard to decide boundary case' is that, if we start doing it, we will probably do a LOT of it, which is a reason that its moral implications are likely to matter.
If we start doing it, we'll have an actual case to look at, instead of handwaving about coulda/woulda/shoulda of entangling people and rocks.
Have you read Stross's /Permutation City/? A part of the book focuses on a thought experiment basically just like this one. (The place the book then takes the thought experiment is one I find weird and implausible, but the question is the interesting part.)
Actually, I would also recommend the essay "What Color are Your Bits?" as relevant: http://ansuz.sooke.bc.ca/lawpoli/colour/2004061001.php
It's about the assignment of properties like copyright to strings of bits, where the property is not a computable function (or indeed, a function at all) of the bits themselves, but still has meaning. I think it's an interesting perspective relevant to the question of assigning the property of consciousness to strings of bits, or physical processes computing on strings of bits. (I don't think it's an answer to the question of whether one can or should do so! It just brings up interesting points.)
Actually, I would also recommend the essay "What Color are Your Bits?" as relevant:
Thanks! That was a new perspective. I feel like claiming that the consciousness is in the "color" would be avoiding an answer... but it's something to think about.
Even if your other assumptions work, I dispute the claim that value only depends on final state. If you reach the same outcome in two different paths, but one involved torture and the other didn't, they aren't valued equally.
Therefore, if you didn't simulate the torture, you can't get a value for how bad it is.
Sorry, I don't understand what you mean: "final state" - state of what? "how bad it is" - what is?
"final state" - state of what?
The world. You're assuming that the value of the world only depends on its state at the end of your simulation. But it doesn't: events that happen between now and then also matter. So if you want to check how bad the world will be if you do action X, you can't just use your trick to find out how the world will be after doing action X, because you also need to know what happened in between.
If you don't agree that states in between matter, consider whether torturing someone and then erasing their memory is morally problematic.
Oh. In this case, I agree with you. I never intended to claim otherwise, or even, the whole original point doesn't make sense without this.
My current view is that it's not possible to check how the world looks after torture without generating information that approximates simulation of torture itself; however this information can be arbitrarily diluted, and diluting it discounts the moral weight by an appropriate factor. We count this quantitatively.
Under this view, the "final state" automatically counts with the "in between" part.
I never intended to claim otherwise, or even, the whole original point doesn't make sense without this.
I'm not sure how the original post makes sense if you agree. I understood the original point as:
This seems to imply that our evaluations don't need to take into account middle states. Value is definitely not linear, so you can't do subtraction of the trick states.
This is a problem even if your skip turns out to be possible.
I understood the original point as:
More or less, and as you said the original point was about the possibility of circumventing the "instantiation of middle states". But if I assumed from the beginning the middle states are not important, it would make no sense to argue that such possibility exists. I saw this as a paradox in which on one hand intuitively, the middle matters, but on the other we can reduce it to something that intuitively seems morally OK (i.e. some unrelated abstract computation).
Your intuition that the middle matters seems to match my current information-theoretic understanding, even if you disagree on what exactly makes it so.
Even if your other assumptions work, I dispute the claim that value only depends on final state. If you reach the same outcome in two different paths, but one involved torture and the other didn't, they aren't valued equally.
Therefore, if you didn't simulate the torture, you can't get a value for how bad it is.
The same can be done with addition of complex probability amplitudes, which is more exactly true in the sense that we are sure it is properly linear,
QM is linear, but you cannot take a snapshot of probability amplitudes.
Where is the contradiction supposed to lie? You're implying the second set of simulations does not contain consciousness. If that's right, then physics gave you a way to cheat and compute the answer in an unexpected way that cannot occur in nature. Since physics was not set up to require torture, this seems entirely possible, if suspicious.
You are not noticing that the "simulate physics" action already includes lots of technical details you don't know about, which already effectively introduce a gap between what happens in the simulation (e.g. your feelings) and what I actually do (e.g. manipulate some matrices of numbers).
All that changes between the two versions is some details of how I do the computations, and if your position changes at all depending on how I do the computations, it is already incoherent.
It's an easy exercise to find a continuous interpolation between any two versions, and ask "so where exactly is the boundary?".
On the other hand, if you try to argue that the more possibility of doing computations I'm not actually doing is immoral, you are essentially arguing that the static snapshot is immoral too. And if that's the case, than so is looking at you carefully.
I deny the parent's second paragraph. (In a nitpicky sense this is trivially false; you could do simulations by kidnapping people and taking over their brains, or by writing calculations on them using knives.)
And of course I did notice your first point, but I deny that we should leap from there to the conclusion of no consciousness. I don't know that I believe it in your scenario; I'm only going along with it because you wanted to assume it with effective-certainty, and you (implausibly) claimed that part wasn't essential to your argument.
Pick some simulation that you are sure definitely has the attribute of "no consciousness" you want to defend (e.g. fill the whole data with a regular checked pattern).
Then pick some simulation that I'm doing that you don't like.
Interpolate all situation between them with a continuous parameter from 0 to 1.
At which point does the "no consciousness" attribute disappear?
The fact that consciousness admits of degrees is verified whenever you go to sleep, or at least when you incorporate external stimuli into a dream. Highway hypnosis seems like another good example.
You say you're already doing the simulation. Interpolations couldn't tell you anything new about what happens within said simulation. Now, there are nevertheless reasons to wonder if they contain consciousness or otherwise add to the problem. But what we know is that you already simulated torture. I need not have a full answer to every philosophical question in order to object.
Thank you!
Indeed my line of thought was incorrect.
The information-theoretic approach + admitting fractional levels of consciousness neatly solves what seemed paradoxical to me.
I edited the article to reflect this update.
Edit:
I consider this solved and no longer stand behind this or similar arguments. The answer given by hairyfigment (thank you!) is simple: combine the information-theoretic approach (as advocated by Eliezer) with admitting degrees of consciousness to be real values (not just is or isn't).
Thank you again hairyfigment for dispelling my confusion.
Original post:
Consider the following scenario:
1. I have technical capabilities that allow me to simulate you and your surroundings with high enough accuracy, that the "you" inside my simulation behaves as the real "you" would (at least for some time).
1b. I take an accurate snapshot "S" of your current state t=0 (with enough surroundings to permit simulation).
2. I want to simulate your behavior in various situations as a part of my decision process. Some of my possible actions involve very unpleasant consequences for you (such as torturing you for 1 hour), but I'm extremely unlikely to choose them.
3. For each action A from the set of my possible actions I do the following:
3a. Update S with my action A at t=0. Let's call this data A(S).
3b. Simulate physics in the world represented by A(S), until I reach a time t=+1 hour. Denote this by A(S)_1.
3c. Evaluate result of the simulation by computing value(A(S)_1), which is a single 32-bit floating point number. Discard all the other data.
A large portion of transhumanists might stand behind the following statement:
What I do in step 1 is acceptable, but step 3 (or in particular, step 3b) is "immoral" or "wrong". You feel that the simulated you's suffering matters, and you'd act to stop me from doing simulations of torture etc.
If you disagree with the above statement, what I write below doesn't apply to you. Congratulations.
However if you are in the group (probably a majority) that agrees with the "wrongness" of my doing the simulation, consider changing my actions to the version described below.
(Note that for simplicity of presentation, I assume that the operator of calculating future-time snapshots is linear, and therefore I can use addition to combine two snapshots, and later subtract to get one of the components back. If you think the operation of directly adding snapshots is not plausible, feel free to substitute another one - attacking this particular detail does not weaken the reasoning. The same can be done with addition of complex probability amplitudes, which is more exactly true in the sense that we are sure it is properly linear, but than we couldn't avoid a much more sophisticated mechanism that ensures that two initial snapshots are sufficiently entangled to make the computation on the sum not be trivially splittable along some dimension.)
(Edit: the operation used in the argument can be improved in a number of other ways, including clever ideas like guessing the result and verifying if it is correct only in small parts, or with tests such that each of them only gives a correct answer with a small probablility p etc. The point being, we can make the computation seem arbitrarily innocuous, and still have it arrive at a correct answer.)
1, 1b & 2. Same as before.
2b. I take an accurate snapshot "R" of a section of the world (of the same size as the one with you) that contains only inanimate objects (such as rocks).
3. For each action A from the set of my possible actions I do the following:
3a. Compute A(S): update S with my action A at t=0.
3b. Compute X = A(S) + R. This is component addition, and the state X no longer represents anything even remotely possible to construct in the physical world. From the point of view of physics, it contains garbage, and no recognizable version of you that could feel or think anything.
3b. Simulate physics in the world represented by R (snapshot of some rocks), until I reach a time t=+1 hour. Denote result by R_1.
3c. Run the physics simulation on X as if moving the time forward by 1 hour, obtain X_1.
3c. Compute value(X_1 - R_1). Discard all the other data.
However at no point did I do anything that could be described as "simulating you".
I'll leave you to ruminate on this.