"What's the worst that can happen?" goes the optimistic saying. It's probably a bad question to ask anyone with a creative imagination. Let's consider the problem on an individual level: it's not really the worst that can happen, but would nonetheless be fairly bad, if you were horribly tortured for a number of years. This is one of the worse things that can realistically happen to one person in today's world.
What's the least bad, bad thing that can happen? Well, suppose a dust speck floated into your eye and irritated it just a little, for a fraction of a second, barely enough to make you notice before you blink and wipe away the dust speck.
For our next ingredient, we need a large number. Let's use 3^^^3, written in Knuth's up-arrow notation:
- 3^3 = 27.
- 3^^3 = (3^(3^3)) = 3^27 = 7625597484987.
- 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = (3^(3^(3^(... 7625597484987 times ...)))).
3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know.
Now here's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:
Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 people get dust specks in their eyes?
I think the answer is obvious. How about you?
I spent quite a while thinking about this one, and here is my "answer".
My first line of questioning is "can we just multiply and compare the sufferings ?" Well, no. Our utility functions are complicated. We don't even fully know them. We don't exactly know what are terminal values, and what are intermediate values in them. But it's not just "maximize total happiness" (with suffering being negative happiness). My utility function also values things like fairness (it may be because I'm a primate, but still, I value it). The "happiness" part of my utility function will be higher for torture, the "fairness" part of it, lower. Since I don't know the exact coefficient of those two parts, I can't really "shut up and multiply".
But... well... 3^^^3 is well... really a lot. I can't get out this way, even adding correcting terms, even if it's not totally linear, even taking into account fairness, well, 3^^^3 is still going to trump over 1.
So for any realistic computation I would make of my utility function, it seems that "torture" will score higher than "dust speck". So I should chose torture ? Well, not sure yet. For I've ethical rules. What's an ethical rule ? It's an internal law (somehow, a cached thought, from my own computation, or coming from outside) that says "don't ever do that". It includes "do not torture !" it includes "nothing can ever justify torturing someone for 50 years". Why are those rules for ? There are here to protect myself from doing mistakes. Because I can't trust myself fully. I've biases. I don't have full knowledge. I've limited amount of time to take decisions, and I only run at 100Hz. So I need safeguards. I need rules, that I'll follow even when my computation tells me I shouldn't. Those rules can be overridden. But they need to be overridden by something with almost absolute certitude, and to be of the same (or higher) level. No amount of dust speck can trigger an override of the "no torture" rule. I know my history well enough to know that when you allow yourself to torture, because you're "sure" that if you don't something worse will happen, well, you end up becoming the worse. I've high ideals, I've the will to change the world for better - therefore I need rules to prevent me from becoming Stalin or the Holy Inquisition. And that's typically the case. 3^^^3 persons will receive dust speck ? Well, too bad. Sure, it'll be a less optimal utility function than allowing just one person to be tortured. But I don't trust myself to sentence that person to be tortured. So I'll chose dust specks for me and everyone.
If you allow me to argue by fictional evidence, well, that reminds me to the end of Asimov Robot cycle (Robots and Empire, mostly). Warning: spoilers coming. If you didn't read it, go read it, and skip the rest of that paragraph ;) So... when the two robots, Daneel and Giskard, realize the limitations of the First Law: « A robot may not injure a human being or, through inaction, allow a human being to come to harm. », and try to craft the Law Zero: « A robot may not harm humanity, or, by inaction, allow humanity to come to harm. », they end up facing a very difficult problem - one for which they'll need Psychohistory to solve, and even then, only partially. It's relatively easy to know that a human being is in danger,or suffering, and how to help me. It's much, much harder to know that humanity is in danger and how to help it. That's a deep reason behind ethical rules : torture someone is just plain wrong. I may think it's good in that situation, because it'll prevent a Terror Attack, or help me win the war against that horrible Enemy, or because it'll deter crime, or because it'll save 3^^^3 persons from a dust speck. But I just don't trust myself enough to go as far as to torture someone because I computed it would do good overall.
And the last important point on the issue is social rules. There is, in XXIest century western societies at least, a strong taboo on torture. That taboo is a shield. It means than when a president of the USA uses torture, he loses the elections (of course, it's much more complicated, but I think it did play a role). It makes using torture a very, very costly strategy. We have the same with political violence. When the cops attacked a anti-war protest at the Charonne metro station on Feb 8, 1962, killing 9 demonstrators including a 16 years old boy, that was the end of Algeria war. Of course, it wasn't just that. De Gaulle was already trying to stop war, it was lost. But the uproar (nearly half a million of people attended their burial) was so high that the political cost of still supporting the war was made much bigger, so that the end of the war was hastened.
I won't take the responsibility of weakening those taboo (against torture, against political violence, ...) by breaking them myself. The consequences on society, on further people using more torture later on, are too scary.
So, to conclude, I'll chose dust specks. Not because my utility function scores higher on dust speck. But because I can't trust myself enough to wield something as horrible as torture (I've ethical rules, and I'll follow them, even when my computations tell me to do otherwise, for it's the only safeguard I know against becoming Stalin) and because I value way too much the societal taboo against torture to take the responsibility of lowering it.
Now... I've a feeling of discontent for reaching that conclusion, because it coincide with my initial gut-level reaction to the post. It somehow feel like I wrote the bottom line, and then the rationalization. But... I did my best, I did overcome the first "excuse" (non-linearity and valuing fairness) my mind gave me. But I don't find flaws in the two others. And well, reversed stupidity is not intelligence. Reaching the same conclusion that I had intuitively doesn't always mean it's a wrong conclusion.