If a model assigns high utility to things that are not actually good, that's a problem with the model
But the question at hand is: which things are good and which are bad? Is one happy and one unhappy person better or worse than two persons going "meh"? Is one person being tortured better or worse than a large number of people suffering dustspecks?
And in particular, are there any way to argue that one set of answers to these questions is objectively less "rational" than another, or is it just a matter of preferences that you happen to have and have to take as axiomatic?
But the question at hand is: which things are good and which are bad? Is one happy and one unhappy person better or worse than two persons going "meh"? Is one person being tortured better or worse than a large number of people suffering dustspecks?
Depends how happy and unhappy, and how much torture vs. how many dustspecks.
One set of answers is only objectively more or less rational according to a particular utility function, and we can only do our best to work out what our utility functions actually are. So I certainly can't say "everyone...
The device allows for certain issues like slavery and income distribution to be determined beforehand. Would one vote for a society in which there is a chance of severe misfortune, but greater total utility? e.g, a world where 1% earn $1 a day and 99% earn $1,000,000 vs. a world where everyone earns $900,000 a day. Assume that dollars are utilons and they are linear (2 dollars indeed gives twice as much utility). What is the obvious answer? Bob chooses $900,000 a day for everyone.
But Bob is clever and he does not trust himself that his choice is the rational choice, so he goes into self-dialogue to investigate:
In the insurance industry, purchasing insurance comes with a price. Perhaps 1.5% premium of the cost of reimbursing you for your house that may burn down. The actuaries have run the probabilities and determine that you have a 1% chance that your house will burn down. Assume that all dollar amounts are utilons across all assets. Bob once again is a rational man. Every year Bob is chooses to pay 1.5% in premium even though his average risk is technically a 1% loss, because Bob is risk adverse. So risk adverse that he prefers a world in which he has less wealth, the .5% went to the insurance companies making a profit. Once again Bob questions his rationality on purchasing insurance:
Eliezer's post on Torture vs. Dust Specks [Here] has generated lots of discussion as well as what Eliezer describes as interesting [ways] of [avoiding] the question. We will do no sort of thing in this post, we will answer the question as intended; I will interpret that eye specks is cumulatively greater suffering than the suffering of 50 years.
My PVG tells me that I would rather have a speck in my eye, as well as the eye's of 3^^^3 people, than to risk to have one (perhaps me) suffer torture for 50 years, even though my risk is only 1/(3^^^3) which is a lot less than 50 years (Veil of ignorance). My PVG is what I will maximize, and doing so is the definition of instrumental rationality.
In short, the rational answer is not TORTURE or SPECKS, but depends on what your preference, values and goals are. You may be one of those whose preference is to let that one person feel torture for 50 years, as long as your actions that steer the future toward outcomes ranked higher in your preferences, you are right too.
Correct me if I am wrong but I thought rationality did not imply that there were absolute rational preferences, but rather rational ways to achieve your preferences...
I want to emphasize that in no way did I intend for this post to declare anything. And want to thank everyone in advance for picking apart every single word I have written. Being wrong is like winning the lottery. I do not claim to know anything, the assertive manner in which I wrote this post was merely a way to convey my ideas, of which, I am not sure off.