Note: edited to replace 'value' with 'preference' as suggested by orthonormal.
Imagine you overheard two children having an argument over whether vanilla ice cream was better than chocolate ice cream. To you as an observer, it would be obvious that this kind of argument has no content. The children aren’t disputing anything measurable in the exterior world; they would agree with each other than chocolate ice cream contains elements from cocoa beans, and vanilla contains the extract from vanilla beans. Most adults wouldn’t have this argument at all, because it’s self-evident that if Mary says, truthfully, that she likes vanilla better than chocolate ice cream, and her husband Albert confesses that he prefers chocolate, then both of them are right. There is no contradiction; vanilla and chocolate are both neutral items until they come into contact with human tastebuds and human brains, at which point their positive or negative weighting is a fact about those brains, not about the substances themselves.
I think that this concept generalizes. Imagine that Mary and Albert are having an argument. Mary hates how Albert leaves papers spread across the kitchen table with empty coffee mugs on top. She wishes he would remember to put his clothes in the laundry basket instead of leaving them on the floor. She nags about it. Albert is helplessly baffled at why she thinks it’s such a big deal. He accuses her of being a nitpicker and a perfectionist.1
It’s hard to say that both of them are right, if each is hurting the other’s feelings. Again, though, their argument isn’t about anything factual. They both agree that there are papers on the desk and clothes on the floor, and that Albert is the one responsible. Where they diverge is the preference they place on this world-state.
Mary is a bit of a neat freak. She likes shiny floors and spotless counters, and she finds cleaning pleasant and relaxing. Seeing a cluttered countertop causes her a small amount of psychological pain. It doesn’t matter where she acquired this attitude; it’s so deeply entrenched in her mind that clean is good that she doesn’t even realize it’s a preference, instead of a fact.
Albert, in turn, has no particular opinion about a clean living space. It doesn’t bother him if the floors are clean enough to eat off, but it doesn’t bring him any particular happiness either. He just doesn’t notice his environment as much. However, he finds cleaning tedious to the point of pain. Left to his own devices, he would live contentedly in squalor.
To Mary, Albert seems incredibly lazy. After all, if cleanliness is so pleasant, and the act of cleaning really isn’t so bad, what except for laziness could keep her husband from holding up his end of the housework? It’s a health hazard, leaving all those dishes everywhere. Doesn’t it bother him?
To Albert, Mary seems obsessive and unreasonable. Why hold yourself to such high standards when it could be so much more rewarding to relax once in a while? It’s one thing for her to work herself to the bone keeping the house that clean, but she doesn’t have to, and she has no right to ask him to, that’s for sure.
These viewpoints are not contradictory. In a universe that contained no minds, a clean table and a cluttered table would both be neutral objects, but in the world-simulation that Mary’s brain builds, a cluttered table is obviously bad and cleaning is neutral. In Albert’s world, a cluttered table is neutral and cleaning is bad. Since they live together and each presumably wants the other to be happy.
If Mary and Albert were rationalists, Mary might say the following:
“Albert, you know that keeping a clean house has a large positive weighting in my utility function, but I know that stuff like vacuuming and scrubbing the bathroom is a big negative in your utility function. I want you to be happy; that means that my utility function includes a miniature copy of yours. Part of me wishes we could share the chores equally because that would feel fairer, but I know it wouldn’t be fair; I enjoy cleaning as a way to wind down at the end of a stressful day, but you don’t, and I don’t have the right to ask you to self-modify so that you would enjoy it. Since I’m too busy to do all of the cleaning by myself, would it be too much to ask for you to do 10% of it? That should balance out my desire for fairness with your lack of enjoyment for cleaning. Also, I know you like to spread out and making a mess when you’re working, so why don’t we say that the study on the second floor belongs to you and you can keep it as messy as you want as long as it doesn’t start to smell. In return, I want you to keep your clothes off the floor and please don’t leave your papers or your coffee-cups anywhere except the study. That shouldn’t be too much to ask?”
Albert has a few minor suggestions to make. Maybe he’s willing to do up to 15% of the cleaning, maybe he doesn’t mind mopping the floor and just hates vacuuming because it makes his ears hurt, maybe he wants to clarify that he doesn’t leave coffee mugs everyone on purpose, he’s just really forgetful and may need to be reminded once in a while. But because Albert’s a rationalist, too, the discussion goes smoothly. Maybe he even offers to try hacking his attitudes to cleaning so he doesn’t find it so unpleasant.
If Mary and Albert aren’t rationalists, though, they have a long way to go to reach that conversation. To each of them, it seems like the other is being pig-headed and deliberately blind about something obvious. Albert doesn’t realize that he’s making a preference judgement about tidiness versus messiness, let alone that he could change how he feels about it.
In the past year, I’ve started to realize how many of the arguments I have, or hear others having, are not about anything that can be measured. (Maybe this was obvious all along to most people; it wasn’t obvious to me.) Different minds work differently. I wish some of my friends would work out more and be fitter, because to me it’s really obvious that fitness is incredibly important and exercise is really not all that bad and you feel better afterwards and you focus better and sleep better…but that’s a preference judgement. I made it with my brain, with all its particularities, and even if it’s verifiably true for me (and I don’t know if I actually focus better after exercise or whether that’s just a biased perception) it may be verifiably false for other people. Maybe there are people who just feel tired and sweaty and grumpy after they work out. They might agree that it would be nice to be fitter, but they’re not making the same trade-off that I am, because the act of exercising, as separate from its long-term benefits, has a negative rather than a positive weight to them.
I don’t exercise because I judged it a rational thing to do; my mother put me on a swim team when I was eight years old and my body is so used to being worked hard in the pool every day that I get crabby if I don’t swim. That a particularity about my childhood and the way it affected the wiring in my brain. It might take someone else orders of magnitude more motivation to follow the same routine that I do, but that doesn’t mean they’re just being lazy and irrational when they don’t work out every day.
Ultimately, for me this means that if I want to go out and help someone because I think they’re making sub-optimal decisions, first I have to make sure my mental model of them is accurate, right down to their preferences and where they differ from mine. What might seem like ‘helping’ to me could be annoying interference to them, and it’s useful to know that in advance. If someone I know spends all their time writing poetry instead of going to the gym, and they fully enjoy that and would consider trading poetry-writing-time for gym-time to be an unreasonable and unnecessary drop in their quality of life, then that’s a true fact about how their brain works. It doesn’t have to be, and maybe their total happiness would be higher once they got used to getting out of the house; maybe they would find that exercise brought a wave of inspiration; but telling someone they’re wrong about their own preferences is not going to help them make that change.2
This applies to the arguments I choose to have, too. If I notice that a pet peeve I have is really a more complicated equivalent of vanilla vs. chocolate, it helps me to find it less irritating and bring it up less often. I don’t apply this method often enough in my daily life, but when I do, it snips a lot of needless conflict in the bud, leaving time for discussions and yes, sometimes conflict, over facts or over preferences that I really want to change people’s minds about.
1. Albert and Mary are loosely based on the dynamic between myself and my former landlady. She was the tidy one, as were most of my roommates; I was the messy one who apparently drove everyone crazy. It took me most of the year to realize how irrational I was being in the way I treated the whole issue. I did, and still do, value sleeping over cleaning, and my schedule is busy enough that there is a trade-off between the two, but I did sign the contract and agree to live in the house, and there was a certain implicit assumption that I would hold up the same standards as everyone else … and if I could do the year over, I would either live somewhere else or do a better job of cleaning, because according to my value system it isn't okay to treat other people's values as less worthwhile than your own.
2. They might not be right about their values; some people have remarkably poor models of what really makes them happy and what they find unpleasant, and another, more rational person might have a better model of them than they do of themselves.