Ben, you are right. Two people with dusty eyes is worse than one. But it isn't twice as worse. It's not even nearly twice as worse. On the other hand I would say that two people being tortured is almost twice as bad as one, but not quite. I'm sure I can't write down a formula for my utility function in terms of number of deaths, or dusty eyes, or tortures, but I know one thing: it is not linear. There's nothing inherently irrational about choosing a nonlinear utility function. So I will continue to prefer any number of dusty eyes to even one torture. I would also prefer a very large number of 1-day tortures to a singe 50-year one. (far far more than 365 * 50). Am I being irrational? How?
My utility function doesn't add the way you seem to think it does. A googolplex of dusty eyes has the same tiny negative utility as one dusty eye as far as I'm concerned. Honestly. How could anyone possibly care how many people's eyes get dusty. It doesn't matter. Torture matters a lot. But that's not really even the point. The point is that a bad thing happening to n people isn't n times worse than a bad thing happening to one person.
Eliezer: it doesn't matter how big of a number you can write down. You are dealing with an asymptote. There is a limit to how bad momentary eye-irritation can be, no matter how many people it happens to. no matter how many people. That limit is far less than how bad a 50 year torture is.
let f(x) = (5x - 1)/x what is f(3^^^3)? It's 5, or close enough that it doesn't matter.