The "people are wonderful" bias is so pernicious and widespread I've never actually seen it articulated in detail or argued for. I think most people greatly underestimate the size of this bias, and assume opinions either way are a form of mind-projection fallacy on the part of nice/evil people. In fact, it looks to me like this skew is the deeper origin of a lot of other biases, including the just-world fallacy, and the cause of a lot of default contentment with a lot of our institutions of science, government, etc. You could call it a meta-bias that causes the Hansonian stuff to go largely unnoticed.
I would be willing to pay someone to help draft a LessWrong post for me about this; I think it's important but my writing skills are lacking.
It's worth separating what people actually believe about how altruistic other people are from what they pretend to believe about the altruism of other people.
If you ask someone whether they believe that there's a chance that their partner would cheat on them, they are most likely to tell you that their partner would cheat on them. The same person might take a few signs that point in the direction of their partner cheating as a huge problem.
I would also expect that beliefs differ a lot between people.
My model of utility (and the standard one, as far as I can tell) doesn't work that way. No rational agent ever gives up a utilon - that is the thing they are maximizing. I think of it as "how many utilons do you get from thinking about John Doe's increased satisfaction (not utilons, as you have no access to his, though you could say "inferred utilons") compared to the direct utilons you would otherwise get".
Those moral weights are "just" terms in your utility function.
And, since humans aren't actually rational, and don't have consistent utility functions, actions that imply moral weights are highly variable and contextual.
2Biff Wiff
Ah yeah, that makes sense. I guess utility isn't really the right term to use here.