# 60

Professor Quirrell didn't care what your expression looked like, he cared which states of mind made it likely.

- HPMOR, Ch. 26

Words should not always be taken at face value. Presumably you know this. You probably have some heuristics about specific situations or claims in which a person’s words should not be taken literally. But I think most peoples’ heuristics here are far too narrow - that is, most people take words literally far too often.

The sequences talk about habitually asking, in everyday life, “What do I think I know, and how do I think I know it? What physical process produced this belief?”. I suggest a similar habit for words in everyday life: “What is being said, and why is it being said? What physical process produced these words?”.

This post is a bunch of examples, in an attempt to goad your system-1 into looking past surface-level meanings more often.

## Dishes

Once or twice a week, I’ll hear my girlfriend yell from the kitchen “Joooooohn! Why are there so many dirty dishes in the sink?”. Going to wash the dishes is not the correct response to this.

If I go wash the dishes, then she will quite consistently find something else to complain about in the meantime - floor needs sweeping, nothing to eat, neighbors are noisy, etc. Usually multiple other things. It was never really about the dishes in the first place, after all. Really, she’s stressed and looking for an outlet.

A hug fixes the problem much more effectively than washing the dishes would.

The general mental motions required to notice this are something like:

• Stop. Don’t just go wash the dishes.
• Ask why this is coming up, and in particular why it’s coming up right now specifically. Is there any particular reason the dishes are relevant right now? (Sometimes the answer is “yes”, and then it’s time to go do the dishes.)
• If there isn’t a reason why the dishes are relevant right now, then I need to figure out the actual reason for the complaint.

This scene from the movie Limitless is a similar example. It’s a bit over-the-top, but it’s one of the few examples of a supposedly-intelligent person in a Hollywood movie actually doing something intelligent (as opposed to technobabble).

## Designers and Engineers

If you work in software, one problem you’ve probably encountered on the job is “the thing they literally ask for is only very loosely correlated with the thing they actually want”.

A designer or product manager comes to a software engineer with some crazy request. They want to redesign a particular button, move it to a different place on the page, and change what it does, but still keep it the same button. A very confused engineer asks “What on earth does that even mean? How is it supposed to be the same button when everything has changed?”. After far too many questions, it turns out that the product team just wanted to re-use the tracking from the old button, because adding new columns to their data is annoying. Main point: the thing they literally ask for is only very loosely correlated with the thing they actually want.

Meanwhile, that same product team is testing out a prototype with potential users and collecting feedback. The users have all sorts of crazy requests. One of them wants a summary page with a bunch of app-internal numbers on it. After asking far too many questions, the product team figures out that what this user actually wants is a way to generate receipts for their customers. Once again, the thing they literally ask for is only very loosely correlated with the thing they actually want.

Down the hall, a manager asks an analyst for the click-through rate on the checkout screen. What the manager actually wants to know is whether lowering prices would lead to more sales. Whether that click-through rate is a good proxy for customers’ price sensitivity is the sort of question the analyst needs to answer, which means the analyst needs to figure out that that’s the real question in the first place. The thing they literally ask for is only very loosely correlated with the thing they actually want.

In another office, the COO has found a used conveyor system on sale and wants to buy it for the warehouse. They ask a lawyer to write up the contract for the purchase, and to keep it simple - just a straightforward asset purchase. The COO probably hasn’t even thought about what happens if the conveyor is defective; the lawyer needs to realize that the COO probably wants the contract to cover any potential problems, even though the COO has not thought about it. The thing they literally ask for...

We could go on all day.

In the information/knowledge economy, a key part of most jobs is realizing that what someone literally asks for is only very loosely correlated with what they actually want. The mental motions required to handle such problems effectively are much the same as the previous section:

• Stop. Don’t just immediately do what was literally requested.
• Ask why this particular request was made. Is there an obvious goal, and is this clearly the best way to achieve that goal?
• If not, what’s the real goal, and what’s the best way to achieve that goal?

## Salespeople

I was on vacation in Mexico with my parents and siblings. We were headed to a touristy beach-park, and took a cab. The cab driver “helpfully” suggested an alternative touristy beach-park. This sounded like the sort of “helpful” suggestion which would earn the driver a kickback, and this hypothesis was promptly confirmed when she handed my mother a laminated advertisement for the place.

… at which point my mother leaned over and said “Hey this looks pretty nice! And it’s even pretty cheap.”

Unrolled into a dialogue, my reaction to this was something like…

Inner voice 1: “Huh??? There isn’t any information about niceness or price on that piece of paper.”

Inner voice 2: “It’s the literal content of the words. There’s a price written on there, and it’s a bit lower than the place we're going.”

Inner voice 1: “Ok, but what does the number on that piece of paper have to do with the amount of money which would change hands at the gate to this place? It’s an ad aimed at tourists, it’s almost certainly misleading. And same with the pictures.”

Inner voice 2: “I don’t think your mother realizes that.”

Inner voice 1: <exasperated sigh>

Point of the story: obviously do not trust information from advertisements or salespeople.

The one exception to this is information which does not seem tailored to make the sale happen. That said, be careful - salespeople can get kickbacks in nonobvious ways. Today’s car dealers, for instance, make most of their money on kickbacks from financing and warranty deals rather than the car itself. (I know this from firsthand experience - I worked at an online car dealership a few years back.)

I won’t do a political example here, but this also includes politicians. It especially includes politicians from your own preferred party. Also note that politicians tend to rely more on bullshit than lies, relative to salespeople - it's not just a question of whether their words are "trustworthy", but of whether they have any correspondence to the real world at all. Ask what physical process resulted in these particular words, and often the answer will be "signalling group loyalty" or "polled well with constituents", with physical reality playing no significant role.

## The Parable Of The Dagger

People tend to generalize “don’t trust ads/salespeople” to an heuristic like “be suspicious of the incentives behind information-sharing”. This isn’t a bad heuristic, but it’s the sort of heuristic which makes it a little too easy to miss the more general rule. The taxi driver trying to sell us on a beachpark is highly salient, but the key underlying factor is that the letters and numbers on the piece of paper do not necessarily have anything to do with the amount of money changing hands at the gate. “Be suspicious of incentives” is less general than “ask what causal process resulted in these words”.

The parable of the dagger makes this point more directly. A jester has angered the king (with a tricky logic puzzle) and been thrown in the dungeon. The king sets up a puzzle for him...

The jester was brought before the king in chains, and shown two boxes.

"One box contains a key," said the king, "to unlock your chains; and if you find the key you are free.  But the other box contains a dagger for your heart, if you fail."

And the first box was inscribed:

"Either both inscriptions are true, or both inscriptions are false."

And the second box was inscribed:

"This box contains the key."

The jester correctly reasons through the puzzle, and picks the second box, only to find that it contains the dagger.

"How?!" cried the jester in horror, as he was dragged away.  "It's logically impossible!"

"It is entirely possible," replied the king.  "I merely wrote those inscriptions on two boxes, and then I put the dagger in the second one."

The steps the jester would need to take to avoid his death are quite similar to the mental motions from earlier:

• Stop. Don’t just take the words written on the boxes at face value.
• Is there an obvious reason the words on the boxes would accurately predict my fate?
• If not, then what is the king really up to?

Finally, a more complex example.

An evo-devo class I was sitting in on assigned this paper. The experimenters were interested in the evolution of Hox genes - genes typically used in animals to establish different roles for different body segments along the head-to-tail axis (e.g. the segments of an ant or bee, the sections of a human spine, etc). They found Hox-analogues in a sea anemone - rather odd, since the anemone doesn’t have the sort of specialized head-to-tail segments with which Hox are usually associated. So, the experimenters investigated the role of those genes in anemone specifically.

The highlight of the paper was this image, which tells most of the story on its own:

Those colors each represent the activity of one Hox protein. They behave exactly like they do in other animals, with each protein lighting up one segment further than the preceding protein… except rather than lighting up head-to-tail along the length of the animal, they’re ordered axially around the animal. (The order in which they light up is determined by the order in which their genes appear in the genome - the genes are in a line, so that a repressor/promoter targeting one will also repress/promote the Hox genes after it.)

The experimenters then use both RNA interference and CRISPR (in separate experiments) to suppress/knock out specific Hox genes, and show that this results in some of the segments “merging” - which in turn gives the anemone merged tentacles from those segments.

Here’s the weird thing: the results from the RNAi experiments were much more impressive, and much more detailed, than the results from the CRISPR experiments. (You can see that visually in the figures above: the tentacles merge much more dramatically in the RNAi examples on the top than the CRISPR examples on the bottom.) Why?

You might guess that there’s some biological weirdness going on, random things interfering with other random things, as sometimes happens in the messiness of biological systems. But my guess is that it’s mostly not about the underlying biology. Reading between the lines of the paper, it sounds like the lab has lots of expertise and experience with RNA interference methods. But CRISPR was the hot new thing, so probably some grad student or reviewer suggested that the paper would be sexier if they threw in a quick CRISPR experiment. The lab lacked experience with this sort of genetic engineering, so probably they just didn’t do it in the most effective way, and ended up with less-impressive results for that experiment. (The class professor, who was familiar with past work from this lab, confirmed that this sounded likely.)

As always, it’s the same basic steps:

• Stop. Don’t just assume that the words and figures in the paper are directly representative of the system under study.
• Ask where these words and figures came from. What did the experimenters actually do, what thoughts went through their heads?
• To the extent that the words and figures reflect something other than the system under study, what can we deduce from them?

One particularly common case is researchers claiming implications which their data do not establish - especially “X causes Y”. (I won’t provide an example, partly because I don’t usually save those papers.) As always, we need to look at the actual process which generated the words: what experiments were actually run, what were the results, and do they actually establish the causal claim? Are the results not just necessary but sufficient to establish causality? If not, what information can we glean from the results?

## Conclusion

Most people have a variety of heuristics about when (not) to take words at face value: beware of ulterior motives, beware of people asking for things they don’t understand, check whether claims in a paper’s abstract are actually established by the results. These are good heuristics to have, but they make it easy to overlook the more general technique:  “What is being said, and why is it being said? What physical process produced these words?”.

The basic mental steps:

• Stop. Don’t just automatically take the words at face value.
• Ask what physical process generated the words. Where did they come from? Why these particular words at this particular time?
• What can we deduce from the fact that the words were spoken, other than the literal content?

# 60

Mentioned in
New Comment

This largely rings true to me but is missing one (in my opinion) absolutely crucial caveat/complication:

Most people (including, as experience has repeatedly confirmed, the vast majority of rationalists/LWers) will do "Ask what physical process generated the words. Where did they come from? Why these particular words at this particular time?" wrong, by virtue of being far too confident in the first answer that their stereotype centers generate, and not accounting for other-people's-minds-and-culture-being-quite-different-from-their-own.

I think most people do an ok job of this most of the time, as long they stop to think about it. The sorts of situations you seem to be pointing to are mainly about politics, which kills the mind when applying "why these words?" just as it does in so many other ways.

More generally, the skill of asking "what physical process produced these words?" is limited by one's ability to accurately model the physical processes which produce words. If someone produces systematically terrible models (as most people do when discussing politicized subjects), then obviously it's not going to work very well, but I don't think that has much to do with this skill specifically. It's more about the general skill of producing good models.

tangential but disposable dishes/flatware cost ~$10/mo/person. With the amount of time people spend interacting with dishes (even if the dishwasher works) the$/hr makes no sense.

I think this is the sort of advice that sounds efficient and clever but really isn't.

Members of my family (of three) spend maybe thirty seconds per meal transferring plates and cutlery to the dishwasher, and I guess the same amount of time getting them out of the dishwasher and into where they are stored. Throwing them away would take maybe 20s per meal. So using not-disposable stuff costs us 40 person-seconds per meal, two meals a day, plus a bit for breakfast (but that, when we bother at all, uses less cutlery and crockery), for let's say 90 seconds a day or 2700 seconds a month. That's 45 minutes per month, versus (taking your figure as correct) $30/month for disposable stuff. Assuming for the moment that this is the only consideration, using disposable stuff is better if whoever's doing the clearing away could use that time to earn money at an after-tax rate of$30 per 45 minutes or \$40 per hour. I can, but my wife and teenage daughter can't and they do just as much table-cleaning and dishwasher-emptying as I do. Further, I am on a salary rather than an hourly wage, and spending an extra 45 minutes per month at work is unlikely to make much difference to my prospects of promotion, bonuses, getting fired, etc.

"Real" cutlery and crockery are much, much nicer to use than disposable. (Disposable plates are almost all too small. Disposable knives don't cut. Disposable plastic forks have bendy tines. Disposable wooden forks have super-blunt tines and cost more. Disposable cutlery of any sort is liable to break. Disposable cutlery of any sort is less comfortable on the hands.)

And there are externalities. Disposable plastic products are made using oil, which is a somewhat scarce resource. I'm pretty sure making them consumes more energy (much of it made by burning fossil fuels -- scarce resource and bad for the environment and for people's health) than cleaning them. They probably end up in landfill, which also isn't great for the environment.

Oh, and while we're adding up the seconds of time it takes to do these things, if we used disposable cutlery and crockery we would need to empty the garbage more often, which also takes time and is a more unpleasant job than loading the dishwasher.

Hm, that also makes sense. I think I overestimated how much time I spend doing those kinds of dishes versus doing "pots and pans" dishes.

Shrug. Many of the things you mentioned don't bother me. Maybe overall what you say is true for a well functioning household. Dishes are often cited as one of the top sources of fights between couples though.

I think there's a worthwhile distinction between "thing that couples fight about" and "thing that causes couples to fight more". In general, something in one of those categories will be in the other, but how strongly a thing is in one isn't a great guide to how strongly it's in the other. My guess is that if people are having big arguments about dishes then it's probably more symptom than cause. (And if they're just occasionally getting annoyed about dishes, it probably doesn't matter much.)

I should add that, although there are a bunch of numbers in my comment above, I haven't actually done a time-and-motion study and figured out exact timings, and my estimates absolutely could be wrong. For me they'd have to be a long way out before it became worth switching from "permanent" to disposable eating-tools.

(Another factor that might be relevant to the decision: maybe the amount and type of storage you need in your kitchen or, if you have and use one, your dining room, is different in the two cases. E.g., if you get disposable things in plastic wrappers, maybe you just leave them sat on the kitchen table all the time, whereas if you have china plates and steel cutlery you probably keep it in cupboards and drawers. That could have implications in cost, convenience, clutter, etc.

[EDITED to add:] Point taken about "well functioning household"; more broadly, the right tradeoffs can change a lot if e.g. some or all of the people in the house suffer from depression or other things that mess with motivation, or have much worse than average mobility, or hate dish-washing / dishwasher-loading a great deal, etc. I certainly don't want to claim that no one would do well to use disposable cutlery and crockery. I'm just not convinced by any general claim that most people, most LW readers, most prosperous people in the US, etc., would have better lives overall if they used disposables.

Dishes are often cited as one of the top sources of fights between couples though.

They would do better to solve that problem than have a substandard dining experience dripping on them every day.

I actually went that route for a while. Switched back mainly because disposable dishes weren't tough enough for many foods - e.g. if I want to cut a tough piece of meat, then plastic forks break, plastic knives won't cut it, and paper plates quickly get destroyed enough that sauce leaks onto the table. Have you had similar issues and found solutions?

The only meat I ever cook is hamburger. Can still mostly use disposables and use plates sparingly?

Mixing disposables and reusables was worse than either. It meant that I didn't need to wash dishes habitually, so dishes would either demand immediate attention when I used them, or sit in the sink until they got really gross. When it's a habit to wash them, it's less costly, because I do it in spare moments when I want to keep my hands busy while thinking.

Disposables do play well with frequently eating out, though. Back when I was eating out for ~2/3 of my meals, disposables were good enough that I never used dishes (at home) at all, and then the benefits were substantial. Going from "occasional dishes" to "zero dishes" gave a lot of value.

for me the nice thing about disposables is that if I wash off any prep items/surfaces before eating there is zero to do after eating when motivation is low.

I took this approach for a while. As others mention there are some externalities you may or may not care about, but for me it was a reasonable tradeoff to make until so long as I couldn't treat my preferences as object. Now that I can just make myself want to do dishes without paying willpower penalties, I just do them and in fact enjoy it!

Wow, that makes sense and I can't believe I never realized it.

After reflecting a little bit, I think that I do in fact take words at face value far too often. My next thought was that it takes effort to read between the lines, and I'm not sure whether that effort is usually worth it. It definitely is in some situations, such as the examples in this post, but I think that there are a lot more minute situations that occur in daily life where it wouldn't be worth it to constantly pause and try to read between the lines, so I'm not sure if it would make sense to up your default level of reading between the lines.

Here's an example. I wanted to bet on the Miami Heat to win the championship and asked a friend to place the bet for me. He responded with this link: https://www.google.com/amp/s/www.actionnetwork.com/news/legal-sports-betting-united-states-projections. I clicked it. He lives in Arkansas, and I saw that it's legal in Arkansas. I responded to his text by joking around about how long it took me to find Arkansas on the map. We went back and forth joking about that. Then I asked him to let me know when he places the bet so I could Venmo him. He said, "Did you read the article?" Then I read it more closely and saw that in Arkansas you can only bet in person, and we previously had talked about how that wouldn't be worth the time.

In retrospect, I do feel a little silly about it. He wouldn't have texted me that link just to show me that sports betting is legal in Arkansas. I mean it's possible, but probably pretty unlikely. So if I stopped and thought about it I would have been able to figure out what he was really telling me: that he wasn't able to place the bet. But as a general rule I don't think that it would be worth my effort to try to read between the lines in similar situations, and nothing really stood out to me about this situation in particular to indicate that I should violate that general rule, so I don't think I have any regrets.

One thing that makes me hesitate is that perhaps reading between the lines is a skill that can be developed, and so after doing it a lot it wouldn't take much effort to apply. Which would probably make the long term calculus of reading between the lines worthwhile even if the immediate calculus isn't. Something similar is the skill to think about the incentives that someone faces. I've slowly built that up and I'm at the point where it pops out to me pretty quickly.

One thing that does seem very clear to me is that it'd make sense to learn to notice situations where you should spend the effort trying to read between the lines. The dishes situation was a good example of that. The more general situation would be "when someone expresses anger towards me". I think in that general situation it makes sense to pause and spend time trying to read between the lines.

The post is sensible for responding to most forms of incompetent/sloppy communication (a benign example is small talk). It's bad advice when there is intent and potential for eventual clear communication.

This needs a distinction between understanding and truth. Understanding what is being said is a separate activity from assigning credence to a claim. The distinction breaks down when the speaker doesn't know what they are talking about, and won't be able to get to know it during the conversation. But if it's possible, it's useful to first learn what the proposed construction is, even if you have no idea how relevant it is (for example, how well it describes/models the world), or if you are certain that it's not relevant. When you can agree on what exactly is being referenced, while ignoring its relevance or truth, that is a valuable milestone in moving the discussion forward.

Discussing the truth of claims is often superfluous. Once you know what is being asserted, you either verify (evaluate) it on your own, or you would require an argument, which is again a construction that can just be explained in the same way without any appeals to its relevance or truth. At no point do you need to know which states of mind make communication of the construction (or the argument) likely. That is, if the object of the discussion is whatever is actually being discussed, not the speaker's state of mind.

I think the language of terminal vs instrumental values fits in to the first two sections. Eg the (more) terminal value of the product team in the button example was to be able to re-use the tracking from the old button, and the redesign was just a means to that end.

Please always dismiss the literal meaning of the words I say (or type) and substitute your personal probability distribution over why I said or typed those words instead.

(upvoted)

(but also)

My model suggests that this was sarcastic, and not meant to be taken literally.

"Be not too quick to blame those who misunderstand your perfectly clear sentences, spoken or written. Chances are, your words are more ambiguous than you think." -- Illusion of Transparency

They are good questions in some cases when you can answer them, bad questions in all cases when you can't.

When a customer says he wants a book to give to a friend's kid, and the only thing he knows about the kid is how old she is, you the seller have to gamble. The customer might refuse, and in the process some of the "physical reality" might come through ("I heard she likes to cook"), but that doesn't mean you should just guess what he wants on the outset or hurry him in any way.

When a customer says she wants something cheap for her husband's son, the best thing to do is to pick something cheap yet not what you personally think would make a bad gift, not argue with the woman or second-guess her motivations.

People aren't good at saying what they want, but they rely on conversation, and making it one-sided... doesn't work.

To be clear, nothing in the OP should be interpreted to mean things should be one-sided. If you're trying to build a model of where words came from (or, more specific to your example, a customer's intent), one of the most useful things you can do is gather more evidence. The person standing there talking to you is a readily-available source of such evidence, and you gather it by continuing to go back-and-forth with them.

(it's just that gathering evidence of what the other person wants and asking yourself in what particular way the other person is trying to use you seem to be two different things.)

I hate the Jester parable.  Why should one take the lesson that written statements are different than spoken ones from a king?  The objection "you lied" applies whether the king put misleading statements on boxes, or whether he just ignored the boxes and stabbed the criminal for fun.

“I merely wrote those inscriptions on two boxes, and then I put the dagger in the second one.”

Statements can have inconsistent truth values. The logical analysis done by the jester is wrong because the jester is assuming that the statements are either true or false. This assumption is unwarranted, and given the actual box contents, the statements aren't true or false.

In other words, the jester didn't correctly analyze the logic of the inscriptions, but mess up because the result has no connection to the real world. The jester incorrectly analyzed the logic of the inscriptions. If he had done so correctly, he would have figured out that the contents of the boxes could be anything.

[+][comment deleted]2y 2