Suppose you're rushing an urgent message back to the general of your army, and you fall into a deep hole. Down here, conveniently, there's a lever that can create a duplicate of you outside the hole. You can also break open the lever and use the wiring as ropes to climb to the top. You estimate that the second course of action has a 50% chance of success. What do you do?
Obviously, if the message is your top priority, you pull the lever, and your duplicate will deliver that message. This succeeds all the time, while the wire-rope only has 50% chance of working.
The point of this is that duplication is not necessarily like probability splitting. A 50% chance of being either inside or outside the hole, is not the same thing as a certainty of being both.
Now, some selfish, indexical utility functions will treat those two cases as the same, but as we've shown above, most non-indexical utilities will not.
But where will you end up, really?
This is the question that it feels we must answer: after pulling the lever, do you expect to be the copy on the top or the copy on the bottom?
But that's a question without meaning. There is no stochasticity here, and no uncertainty. There will be one copy at the bottom of the hole, maybe asking themselves "I wonder where I will end up", and, after pulling the lever, there will be two copies, one at the top and on at the bottom, both of them remembering falling into the hole, thinking "I wonder where I will end up", and pulling the lever. Everything is deterministic and known.
What if you close your eyes for ten seconds after pulling the lever? You could argue that both copies will now face genuine uncertainty during those ten seconds, as they don't know whether they are on the top or on the bottom.
But both copies are thinking identical thoughts; their mental processes are the same. "Am I at the top or at the bottom?" will be thought by both copies; "if I open my eyes, do I expect to see dirt or clear sky?". Your two copies cannot distinguish themselves in any way: they think the same, reason the same, have the same evidence. As long as they can't distinguish their position, they are to all extents and purposes the same agent.
You may still be tempted to assign probabilities to being below or above, but what if the lever initially creates one copy - and then, five seconds later, creates a million? How do you update your probabilities during these ten seconds? These kind of questions illustrate the problem that seems to bedevil any theory of probability of "who you are" among identical or similar copies.
The Many Worlds of Quantum Mechanics
So, what does this argument that duplication-is-not-probability imply for many worlds quantum mechanics?
For me, this still remains a great puzzle. Superficially, it just seems like a mass of duplications; the fact that some of these duplications have greater "quantum measure" than others doesn't seem to be relevant: we can't observe uniform increases or decreases in the quantum measure. Nothing in the quantum measure seems to imply that the "thread of conscious experience" should preferentially flow through larger measure branches.
However, observing the universe we are in, we see everything being very consistent with treating the quantum measure as a probability. Chairs are stable, people don't tunnel through walls, everything doesn't go instantly crazy at every moment, etc.
This puzzles me, I confess. It has caused me to update away from many worlds and more towards alternative interpretations of quantum mechanics, such as the transactional interpretation.