"What's the worst that can happen?" goes the optimistic saying. It's probably a bad question to ask anyone with a creative imagination. Let's consider the problem on an individual level: it's not really the worst that can happen, but would nonetheless be fairly bad, if you were horribly tortured for a number of years. This is one of the worse things that can realistically happen to one person in today's world.
What's the least bad, bad thing that can happen? Well, suppose a dust speck floated into your eye and irritated it just a little, for a fraction of a second, barely enough to make you notice before you blink and wipe away the dust speck.
For our next ingredient, we need a large number. Let's use 3^^^3, written in Knuth's up-arrow notation:
- 3^3 = 27.
- 3^^3 = (3^(3^3)) = 3^27 = 7625597484987.
- 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = (3^(3^(3^(... 7625597484987 times ...)))).
3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know.
Now here's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:
Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 people get dust specks in their eyes?
I think the answer is obvious. How about you?
I definitely think it is obvious what Eliezer is going for: 3^^^3 people getting dusk specks in their eyes being the favorable outcome. I understand his reasonig, but I'm not sure I agree with the simple Benthamite way of calculating utility. Popular among modern philosophers is preference utilitarianism, where the preferences of the people involved are what constitute utility. Now consider that each of those 3^^^3 people has a preference that people not be tortured. Assuming that the negative utility each individual computes for someone being tortured is larger in value than the negative utility of a speck of dust in their eyes, then even discounting the person being tortured (which of course you might as well given the disparity in magnitude, which is more or less Eliezer's point) you would have higher utility with the flecks of dust.
There are in fact numerous other ways to calculate the utility so such that 3^^^3 people with dust flecks in their eyes is preferable to one person undergoing fifty years of torture while still preserving the essential consequentialist nature of the argument. John Stuart Mill might argue there is a qualitative difference between torture and dust flecks in your eyes that keeps you from adding them in this way while an existentialist might argue that pain and pleasure aren't what should be computing with the utility function but something closer to "human flourishing" or "eudaimonia" and that in this calculation any number of dust flecks has zero utility while torture has a large negative utility. It all depends on how you define your utility function.