We've been thinking about moral status of identical copies. Some people value them, some people don't, Nesov says we should ask a FAI because our moral intuitions are inadequate for such problems. Here's a new intuition pump:
Wolfram Research has discovered a cellular automaton that, when run for enough cycles, produces a singleton creature named Bob. From what we can see, Bob is conscious, sentient and pretty damn happy in his swamp. But we can't tweak Bob to create other creatures like him, because the automaton's rules are too fragile and poorly understood, and finding another ruleset with sentient beings seems very difficult as well. My question is, how many computers must we allocate to running identical copies of Bob and his world to make our moral sense happy? Assume computing power is pretty cheap.
I completely lack the moral intuition that one should create new conscious beings if one knows that they will be happy. Instead, my ethics apply only to existing people. I am actually completely baffled that so many people seem to have this intuition.
Thus, there is no reason to copy Bob. (Moreover, I avoid the repugnant condition.)
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.
Part 2