"What's the worst that can happen?" goes the optimistic saying. It's probably a bad question to ask anyone with a creative imagination. Let's consider the problem on an individual level: it's not really the worst that can happen, but would nonetheless be fairly bad, if you were horribly tortured for a number of years. This is one of the worse things that can realistically happen to one person in today's world.
What's the least bad, bad thing that can happen? Well, suppose a dust speck floated into your eye and irritated it just a little, for a fraction of a second, barely enough to make you notice before you blink and wipe away the dust speck.
For our next ingredient, we need a large number. Let's use 3^^^3, written in Knuth's up-arrow notation:
- 3^3 = 27.
- 3^^3 = (3^(3^3)) = 3^27 = 7625597484987.
- 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = (3^(3^(3^(... 7625597484987 times ...)))).
3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know.
Now here's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:
Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 people get dust specks in their eyes?
I think the answer is obvious. How about you?
I think I've seen some other comments bring it up, but I'll say it again. I think people who go for the torture are working off a model of linear discomfort addition, in which case the badness of the torture would have to be as bad as 3^^^3 dust particles in the eye to justify taking the dust. However, I'd argue that it's not linear. Two specs of dust is worse than twice as bad as one spec. 3^^^3 people getting specs in their eyes is unimaginably less bad than one person getting 3^^^3 specs (a ridiculous image considering that's throwing universes into a dude's eye). So the spec very well may be less than 1/(3^^^3) as bad as torture.
Even so, I doubt it. So purely utilitarian probably does suggest torture the one guy.
It has to be more than just not linear for that to solve it, it has to be so nonlinear that no finite number of specks at all can add up to the torture, since otherwise we could just ask the same question using the new number instead of 3^^^3.
If it's so nonlinear that no finite number of specks can add up to torture, then you find the maximum amount that a finite number of specks can add up to. Then there are two amounts (one slightly more than that and one slightly less) where one amount cannot be balanced by dust particles and one amount can, which doesn't really make any sense.