Multiple Choice

by Alicorn 9y17th May 201035 comments


When we choose behavior, including verbal behavior, it's sometimes tempting to do what is most likely to be right without paying attention to how costly it is to be wrong in various ways or looking for a safer alternative.

If you've taken a lot of standardized tests, you know that some of them penalize guessing and some don't.  That is, leaving a question blank might be better than getting a wrong answer, or they might have the same result.  If they're the same, of course you guess, because it can't hurt and may help.  If they take off points for wrong answers, then there's some optimal threshold at which a well-calibrated test-taker will answer.  For instance, the ability to rule out one of four choices on a one-point question where a wrong answer costs a quarter point means that you should guess from the remaining three - the expected point value of this guess is positive.  If you can rule out one of four choices and a wrong answer costs half a point, leave it blank.

If you have ever asked a woman who wasn't pregnant when the baby was due, you might have noticed that life penalizes guessing.

If you're risk-neutral, you still can't just do whatever has the highest chance of being right; you must also consider the cost of being wrong.  You will probably win a bet that says a fair six-sided die will come up on a number greater than 2.  But you shouldn't buy this bet for a dollar if the payoff is only $1.10, even though that purchase can be summarized as "you will probably gain ten cents".  That bet is better than a similarly-priced, similarly-paid bet on the opposite outcome; but it's not good.

There's a few factors at work to make guessing tempting anyway:

  • You can't always leave "questions" blank.
  • Guessing is inconsistently penalized, and the penalties are often hidden.
  • Inaction sometimes has negative consequences too.
  • An increased ability to rule things out relative to the general population causes overconfidence.
  • We are biased against rating the value of information highly, and prefer to act on what we already know.

However, I maintain that we should refrain from guessing at a higher rate than baseline.  This is especially relevant when choosing verbal behavior - we may remember to act according to expected utility, but rarely think to speak according to expected utility, and this is especially significant around sensitive topics where being careless with words can cause harm.

Taking on each reason to guess one at a time:

You can't always leave questions blank, and unlike on a a written test, the "blank" condition is not always obvious.  The fact that sometimes there is no sane null action - it's typically not a good idea to stare vacantly at someone when they ask you a question, for instance - doesn't mean, however, that there is never a sane null action.  You can be pretty well immune to Dutch books simply by refusing to bet - this might cost gains when you don't have Dutch-bookable beliefs, but it will prevent loss.  It is worthwhile to train yourself to notice when it is possible to simply do nothing, especially in cases where you have a history of performing worse-than-null actions.  For instance, I react with panic when someone explains something to me and I miss a step in their inference.  I find I get better results if, instead of interrupting them and demanding clarification, I wait five seconds.  This period of time is often enough for the conversation to provide the context for me to understand on my own, and if it doesn't, at least it's not enough of a lag that either of us will have forgotten what was said.  But remembering that I can wait five seconds is hard.

Guessing is inconsistently penalized, with sometimes hidden costs, so it's hard to extinguish the guessing response.  If you're prone to doing something in a certain situation, and doing that something doesn't immediately sock you in the face every single time, it will take far longer for you to learn not to do it - this goes for people as well as pets.  Both the immediate response and the subjective consistency are important, and a hidden cost contributes to neither.  However, smart people can rise to the challenge of reacting to inconsistent, non-immediate, concealed costs to their actions.  For instance, I'd be willing to bet that Less Wrong readers smoke less than the general population.  Observe the relative frequency with which guessing hurts or may hurt you, compared to not guessing, and make your plans based on that.

Inaction can be harmful too, and there's a psychological sting to something bad happening because you stood there and did nothing, especially when you're familiar with the hazards of the bystander effect.  I do not advocate spending all day sitting on the sofa for fear of acting wrongly while your house collapses around your ears.  My point is that there are many situations where we guess and shouldn't, not that we should never guess, and that there is low-hanging fruit in reducing bad guessing.

You're right more often than a regular person, but that doesn't mean you are right enough: life's not graded on a curve.  The question isn't, Do I have a better shot at getting this one right than the neighbor across the street? but, Do I have a good enough shot at getting this one right?  You can press your relative advantages when your opponents are people, but not when you're playing against the house.  (Additionally, you might have specific areas of comparative disadvantage even if you're overall better than the folks down the road.)

Information is more valuable than it seems, and there is often a chance to try to improve one's odds rather than settling for what one starts with.  Laziness and impatience are both factors here: checking all the sources you could takes a long time and is harder than guessing.  But in high stakes situations, it's at least worth Googling, probably worth asking a couple people who might have expertise, maybe worth looking a bit harder for data on similar scenarios and recommended strategies for them.  Temporal urgency is more rarely the factor in discounting the value of information-gathering than is simply wanting the problem to be over with; but problems are not really over with if they are guessed at wrongly, only if you get them right.