Teachable Rationality Skills

Recent brainstorming sessions at SIAI (with participants including Anna, Carl, Jasen, Divia, Will, Amy Willey, and Andrew Critch) have started to produce lists of rationality skills that we could potentially try to teach (at Rationality Boot Camp, at Less Wrong meetups, or similar venues).  We've also been trying to break those skills down to the 5-second level (step 2) and come up with ideas for exercises that might teach them (step 3) although we haven't actually composed those exercises yet (step 4, where the actual work takes place).

The bulk of this post will mainly go into the comments, which I'll try to keep to the following format:  A top-level comment is a major or minor skill to teach; upvote this comment if you think this skill should get priority in teaching.  Sub-level comments describe 5-second subskills that go into this skill, and then third-level comments are ideas for exercises which could potentially train that 5-second skill.  If anyone actually went to the work of composing a specific exercise people could run through, that would go to the fourth-level of commenting, I guess.  For some major practicable arts with a known standard learning format like "Improv" or "Acting", I'll put the exercise at the top and guesses at which skills it might teach below.  (And any plain old replies can go at any level.)

I probably won't be able to get to all of what we brainstormed today, so here's a PNG of the Freemind map that I generated during our session.

260 comments, sorted by
magical algorithm
Highlighting new comments since Today at 3:43 PM
Select new highlight date
Moderation Guidelines: Reign of Terror - I delete anything I judge to be annoying or counterproductiveexpand_more

Skill: Being strategic. I.e., life-consequentialism, where you actually do things based on their expected future consequences, as opposed to drifting into a PhD program because your friends are doing it.

Exercise materials: We either need to develop efficient probes for getting people to list out major life choices that they could actually remake (are under serious reconsideration) or we need to develop hypothetical life stories and policy decisions to use in exercises.

Subskill: Avoid pitfalls of verbal strategic reasoning.

  • List consequences without shifting from intuitive-sum to verbal-justification mode.
  • Don't exacerbate scope insensitivity or attending to rare events.

There are studies showing that people who consider their decisions more make worse decisions. As I understand it, the main explanation for this is that people shift from an intuitive sum of costs and benefits, to seeking verbally justifiable decisions, which in turn might lead them to one-reason-decisionmaking, ignoring some of their costs and benefits which are important to them but seem less "sensible". I also suspect it may exacerbate other biases like scope insensitivity or rare events - thinking about cases which are rare or short in duration.

The classic case being "Let's get a bigger house, further away from work, so it has an extra bedroom in case Grandma comes over", which she does once a year, but the 20 minutes of extra commute time happen every day and are not acclimated-to.

Exercise A: Give somebody two hypothetical package deals to choose from. First, have them choose quickly and intuitively. Then, have them think about consequences and list out desiderata and alternatives... but at the end of that, have them do the intuitive sum and state a preference, rather than coming up with a verbal reason for the decision.

Exercise B: Have some of the desiderata be rare cases or cases of short duration. Detect these, cross them out with a black marker.

"Let's get a bigger house, further away from work, so it has an extra bedroom in case Grandma comes over"

Not saying this is a bad example, but it COULD be the case that grandma never being able to come over is totally unacceptable. Which is also a pitfall - something can seem trivial until it goes away.

Only if there's really no other way for Grandma to come over - not even for example sleeping in the living room so she can have the sole bedroom.

Subskill: Maximize on big things, satisfice on small things.

Subskill: Unbundling; optimize separate things separately.

Example 1: Optimize fuzzies and utilons separately.

Example 2: Optimize grades and learning separately (instead of just optimizing grades, or haphazardly optimizing both at the same time).

5SL: Notice when you're optimizing two things at once. (Maybe because you (a) have a sense of awkwardness or of not getting enough done, and when you list out the desirable consequences, there's more than one?) Then, find two different things you can do which optimizes each one individually and without worrying about the other one.

If you need to work on several projects at once (as is often necessary in the real world), then do it by creating and maintaining a clear separation between them. A separation both of time and attention.

Besides scheduling different projects at different times, do things related to neither project in between so your attention to the earlier project doesn't carry over. This is an ideal time for routine maintenance/chores. Check email, do the dishes, go grocery shopping, take lunch; to the extent that you need to think about something other than the immediate task, think about the upcoming project, not the one you just left. Then go to work on the second project with your attention on the first broken.

Yet I've done better doing the opposite. When faced with incompatible courses of action that optimize different things, look for a third alternative that gets both. The choice doesn't have to be hard - even if the optimizing targets are "save the world" and "talk to cool people", frustration with the obviously right choice triggers a search for a third alternative as well.

I'd conclude that the most important skill is to stop, notice you're confused, and work out that it's because you're trying to optimize two goals. Whether you then optimize them separately, or find a third alternative, you'll probably do better than if you conflate "grades = learning" or "utilons = fuzzies" and try to optimize that non-existent conflation.

Sense of unproductivity is a good flag for unbundling goals. I recently tried to figure out why I haven't finished as many free-time programming projects as I used to, and realized that I had at least 4 goals for free-time programming: learn a new language, build something personally useful, build something other people will use, and apply techniques from a textbook I've been working through. I couldn't find a project that satisfied all my goals, so I was skipping back and forth and not finishing anything.

Subskill: Notice foreign goals. Are your parents making you do it? Are you doing it because you read it in a book? Did you just drift into doing that?

Converse subskill: Notice feeling of actually caring about something. Notice whether that caring is giving energy to what you're doing. Notice its absence.

Subskill: Detach from sunk costs.

Material for exercises: Requires a case where the person has sunk costs and the option of continuing or not continuing. Might want to choose cases slightly less fraught than a marriage or a PhD program - you want to work up to those gradually.

Exercise A: For a case where sunk costs exist, imagine that you are a new person who was just now teleported into this person's life. Variant: Imagine you were just teleported into this person's life, and everyone else knows this, and they all expect you to execute sudden changes of course. See what this changes about your thinking, regardless of whether it changes your decision.

Exercise B: For whatever sunk costs you're in the middle of expending, look at the same scenario and rephrase it as a sunk benefit, the purchased option to finish a task more quickly than before. E.g., if you already paid $100 on a $150 item, change from "I paid $100" to "I now have the option of purchasing this item for $50". Again, the stated objective of the exercise will be, "Notice the difference in your thinking", not, "try to change your decision".

Exercise C: As above, but imagine that you bought the option for a penny on eBay. E.g. if you're one year into a four-year PhD program, imagine that you paid a penny on eBay to purchase an option to get a PhD in three years rather than the usual four years. Would you exercise that option if you paid a penny for it?

That last exercise seems to run afoul of some value-related heuristics. Price is so common a proxy for utility that imagining that you paid a penny for the option on eBay might irrationally devalue whatever you're looking at.

Of course, looking at the contrast might still give you some useful insights -- provided you can untangle them.

Subskill: Use fungibility. There are different ways to achieve many goals. E.g. instead of wasting an evening with a relative in a way you resent, send them a postcard.

Subskill: Don't express emotions as policy decisions. (Anna.) Find some other outlet for the emotion besides the policy decision, i.e., screaming. (Eliezer.)

Keep in mind that the current psychology literature indicates that cathartic release is actually undesirable-- while catharsis might improve affect in the short term, this is largely because it represents giving in to one's anger, and in the long run it actually makes you more likely to experience anger in the future.

I therefore advise that, in the event that one needs to make a particularly important decision and is currently angry, cathartic screaming may be effective, but its use should be restricted only for cases where it is strictly necessary.

(and this is coming from the ex-leader of a cathartic screaming club :P)

Or, more generally, what you do reinforces your mental state at the time. Acting in anger (even just a cathartic release) reinforces your tendency toward (the likelihood of future) anger; the same with acting from fear and most other emotions. Also, whatever the cause of your procrastination, acting on it, letting yourself get away with procrastinating, increases your problems with it in the future. As a poster I made for my study wall many years ago said: "Commit through Action: Do It".

Exercise: Recall acts you've done while your emotions were running high, in cases where it seems like something that might be worth optimizing. Of those acts, ask whether the action/policy/response can best be interpreted as maximizing a worthwhile criterion, or as direct expressions of the emotion.

Subskill: Chain from feelings of angst and frustration into saying, "I need to be strategic!" and the other skills listed.

Subskill: Before the final moment of doing something that has any sort of cost or downside, ask whether you're doing it because of its consequence or merely because you previously decided to do it.

Exercise: Before ordering food in a restaurant, check whether you want the food, or just have a cached belief that you like it.

Not just in a restaurant. I am trying to lose weight, and one of the more effective strategies is to make myself stop and ask myself if I really want to fix and eat something right now, or if I am thinking about eating for some other reason. It helps that I have gotten rid of most of the ready to eat food in my house and have to take the time to actually fix something, which slows me down enough to ask the question.

Failure mode: comfort food. Thinking about eating for non-hunger, non-appetite reasons, but persistent and inducing a real desire for food.

Like many people, I have the opposite problem. (Not trying to gain weight exactly, but to keep from becoming weak and sick.) Even readily available ice cream gets put off for hours, and then some more hours. The limiting cached thought in my case seems to be "it's not worth the effort". Might be self-fulfilling.

Even ice-cream has to be actually pulled out of the fridge, scooped out of the punnet into a bowl. This may seem like a small hurdle - but it's obviously enough to keep you from eating.

Instead - try getting a bunch of non-perishable food (eg mixed nuts or dried fruit/trail mix) and leaving it in an open jar somewhere easily accessible (eg on the bench). Every time you walk past it - grab a small handful and nibble on it.

From there, you can progress to a bowl of pre-cut-up fresh fruit or veg (keep a lid on it so it doesn't gather flies) for a healthier option. (cut it up all at once at the beginning of the week - leave out a day's worth each morning).

...and that should get the ball rolling.

Anna's subskill list from "Humans Are Not Automatically Strategic":

  • Ask ourselves what we’re trying to achieve;
  • Ask ourselves how we could tell if we achieved it (“what does it look like to be a good comedian?”) and how we can track progress;
  • Find ourselves strongly, intrinsically curious about information that would help us achieve our goal;
  • Gather that information (e.g., by asking as how folks commonly achieve our goal, or similar goals, or by tallying which strategies have and haven’t worked for us in the past);
  • Systematically test many different conjectures for how to achieve the goals, including methods that aren’t habitual for us, while tracking which ones do and don’t work;
  • Focus most of the energy that isn’t going into systematic exploration, on the methods that work best;
  • Make sure that our "goal" is really our goal, that we coherently want it and are not constrained by fears or by uncertainty as to whether it is worth the effort, and that we have thought through any questions and decisions in advance so they won't continually sap our energies;
  • Use environmental cues and social contexts to bolster our motivation, so we can keep working effectively in the face of intermittent frustrations, or temptations based in hyperbolic discounting.

Subskill: Musashi's "cut through in the same motion".

Since LW lore has grown wide, can you please at least point to the reference for the uninitiated?

From Musashi:

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.”

Pretty sure it's mentioned in Twelve Virtues of Rationality, which provides a decent summary in context -- although you should read Musashi's Book of Five Rings if you really want to absorb the concept. He's a lucid (if sometimes infuriatingly vague) writer, and there are several good translations floating around.

Exercise: take a class in historical fencing techniques. :)

Doesn't have to be Japanese style. Italian or Spanish schools teach this too. Avoid the modern sport/olympics style classes.

Probably not great advice if you're looking specifically for a practice that'll quickly teach the habit of carrying tactical decisions through into strategic goals -- in this context that's something you only get from lots of blade-to-blade practice, and the koryu arts are almost universally very heavy on kata. In a typical dojo you might not get to freestyle sparring for a year or more. Western reconstructionist fencing tends to be less so, but there's still a pretty serious ramp-up period in every salle I've ever been exposed to.

On the other hand, if you can get past that period, just about any martial art which involves partnered practice is remarkably good at developing the skill of instrumentalizing strategic thinking (though it still needs to be generalized to the rest of life, a difficult trick which probably qualifies as a virtue of rationality in its own right). Weapon arts (and aikido, but it's unique in this respect among the empty-hand arts I've studied) are also good for developing a habit which is difficult to put into words, but which might be approximated as "presence" or "mindfulness".

Subskill: "What is the consequence, what is the goal?"

Exercise A: For some policy that someone is carrying out today, starting as close to the object level as possible (answering an email, making a phone call, buying something at the store), ask about the consequences, and the consequences of the consequences. Identify the consequences that are desirable or that the action is being carried out for-the-sake-of. State the goal in abstract terms. Ask whether achieving the goal has further consequences - even things terminally desirable often have other, instrumentally desirable or undesirable consequences. Trace out the chain of specific events and the abstract instrumental and terminal goals they correspond to.

Exercise B: For each goal node, find some other policy - not necessarily a superior policy, but some other policy - that would be helpful for the same goal, not necessarily in the same way. (The point being to unanchor your concept of that goal from the exact, specific means of achieving it. This also obviously starts on the habit of searching for superior alternatives.)

Rationality skill: Recognize rationality skill in others.

There is a strong tendency for people to use the heuristic, "P(Person X is right) = # of times person X has been observed to be right / # of statements person X has made", or worse, "(# statements by X - # of times X has admitted to making a mistake) / # statements by X".

This encourages people who want to be respected to do 4 bad things:

  • Make many pronouncements on things that are obviously right, perhaps in a manner suggesting they are controversial claims.
  • Avoid saying anything unless they are certain they are correct.
  • Avoid saying anything concrete enough to possibly be proven wrong.
  • Never, ever admit to having made a mistake.

Unfortunately, these methods are extremely effective.

Avoid saying anything unless they are certain they are correct.

I don't think this is a pernicious behavior at all. I suspect that this is actually a sign of rationality.

http://www.paulgraham.com/heroes.html (See Robert Morris)

I think the key here is qualification - Robert Morris avoided being wrong by not stating things unqualified unless he was sure of them, whereas the failure mode for rationalists is not expressing an idea at all unless fairly sure of it.

We want ideas to be shared before they're well-supported, because discussion is generally the best way for them to find support (or disproof) - we just need to signal the uncertainty when we introduce an idea.

It's much like what I've been taught in analytical chemistry - every number has a stated uncertainty associated with it.

It can easily be both; and, while not speaking unless very confident may not be terribly harmful by itself, the perfectionism that it's an example of is a very characteristic failure mode of rationalists.

To some extent, those are also signs of someone low on the social hierarchy (or someone who feels low on the social hierarchy), and a punitive culture that punishes individuals for brainstorming or otherwise being visibly wrong.

Skill: Anti-rationalization (1): Prevent your mind from selectively searching for support of only one side of an argument.

Subskill: Analyze the underlying reasons why you're trying to rationalize for or against something - why a conclusion feels required, or disallowed.

Important subskill: Notice when a candidate "the reason I'm trying to rationalize something" is a poor guess or itself a rationalization - when it's a guess that sounds plausible about someone like you in your position, but doesn't seem to ring true.

Exercise: Pretend you are your evil alter ego when analyzing the reasons for your rationalization. What would your alter ego say about your rationalization? Your alter ego will probably come up with some selfish, lazy or just plain silly reasons for your rationalization. Once you have this list see the section on how to accept the truth.

Subskill: Notice the process of selectively searching for support, and halt it. Chains into "Ask whether, not why", or into trying to unbind the need to rationalize.

Level 1, notice the active process after performing a conscious check whether it's running; Level 2, detect it perceptually and automatically; Level 3, never run this process.

Exercise material: List of suggested conclusions to rationalize. I'm not quite sure what the prerequisites here would be; one of my own childhood epiphanies was finding that I could argue reasons why there ought to be a teapot in the asteroid belt, and thinking to myself, "Well, I better not do that, then." I suspect the primary desideratum would be more that the topic offers plenty of opportunity to come up with clever rationalizations, rather than that the topic offers motivation to come up with clever rationalizations.

Exercise A: Pick a conclusion from the list. Come up with a clever argument why it is true. Notice what it feels like to do this. Then don't do it.

(This is distinct from noticing the feeling of being tempted to rationalize, a separate subskill.)

Exercise B: In the middle of coming up with clever arguments why something is true, stop and chain into the Litany of Tarski, or some other remedy. Variant: Say out loud "Ew!" or "Oops!" during the stop part.

Exercise C: For a conclusion on the list, argue that it is true. Then "Ask whether, not why" - try to figure out whether it is true. Notice the difference between these two processes. Requires a question whose answer is less than immediately obvious, but which someone can find info about by searching their memory and/or the Internet.

Hmmm this just made me think that debate clubs deliberately teach the opposite of this skill.

True, but the successful debaters in my experience are the ones who can construct both sides of an argument in order to pre-empt and account for possible responses.

This isn't necessarily just a binary for/against disinction but considering better ways to acheive the same goal and possible unintended consequence.

[Speaking as an active participant of the UK universities competitive debating circuit, though I acknowledge that makes me prone to rationalise in its favour]

And at least in Britain and Ireland they provide disproportionate numbers of future lawyers and politicians.

argue reasons why there ought to be a teapot in the asteroid belt

It's not clear to me how this is different from fabricating evidence. Is it?

Exercise material (prerequisite for multiple exercises below): Have a hot-topic list such that incoming students at the expected level (e.g. level = typical LW reader) would be tempted to rationalize at least some of them. This requires both that someone care about the topic, and that the topic isn't so cut-and-dry that there's no temptation to distort anything. E.g., I care about atheism but I don't have any emotional fear of that argument coming out "the wrong way" - on the other hand, putting me in an actual argument with, say, my parents, or someone who was a really clever theistic arguer in front of an audience, might generate the emotional temptation to cheat to ensure winning on every single point.

We can probably generate a good hot topic list from the past discussions on LW. Here's some suggestions based on a very quick attempt at recalling past debates.

  • atheism
  • the benefits of rationality
  • the importance of researching friendly AI
  • is cryonics worthwhile
  • the morality of pick-up artistry

Subskill: Be able to identify the internal feeling of having a required conclusion, of an argument only having one allowed answer, and (a slightly different internal sensation) of other answers being disallowed.

Level one: After this skill is explicitly mentioned and invoked by some other process, be able to notice this internal sense of required-conclusion-ness (disallowed-ness) when you consciously focus on it.

Level two: Have a constant perceptual eye out for feelings like this, notice automatically without needing to be "on guard", chain into applying other anti-rationalization skills (e.g. Litany of Tarski).

Exercise idea: For items on the hot-topic list, identify ones that you care about, and:

Exercise A: Try to identify directly, just from looking at the issue and imagining the potential answers to it, the emotional sense that there's only one allowed answer to it, and the emotional sense that a different answer is not allowed.

Exercise B: Imagine being in the process of losing an argument about that issue. Identify the drive (desperation, need) to regain the lost territory and win. Then imagine being in the process of winning an argument about that issue. Identify the sense of triumph and the prior commitment which makes that particular conclusion "winning".

Exercise C: Get into a simulated argument about the issue with someone taking the opposite side from the one you care about. Maintain awareness of your overall emotional state, try to be aware of the internal drive to produce a particular answer, be aware of the sense of revulsion or flinch-away that associates with other answers.

Get into a simulated argument about the issue with someone taking the opposite side from the one you care about.

I think it is important to integrate this with searching for a third alternative.

1) Establish, on paper and before doing anything else, what you think alternative positions are.

2) Have a third party who does not look at your list identify third alternatives.

3) Determine which good ideas you didn't think of. This can only be an approximation, as you and the other person have used different words to describe alternatives. Notice the emotional urge to rationalize and conclude you didn't leave out important third alternatives, and the urge to rationalize that the alternatives you didn't think of aren't compelling. (This is an artificial hot-button issue, but that is only a side-benefit of this step.)

4) The list that you have constructed probably avoids your belief's real weak points. Alternatives identified by the other person, but not you, are best for pitting against your cherished idea, as in exercise C above.

5) Notice the relative strength of the best arguments not on your list as against the ones on your list. If they are strong, consider whether you have been failing to consider third alternatives for intellectual reasons, such as not holding off on proposing solutions, or avoiding your belief's weak points for emotional reasons.

This is a similar list from Robert Anton Wilson's Prometheus Rising (there is a list of lists of such exercises through the book, 12 or 15 total but this may be the best one).


1.) If you are a liberal, subscribe to the National Review, the country's most intelligent (and witty) conservative magazine, for a year. Each month try to enter their reality tunnel for a few hours while reading their articles.

2.) If you are a conservative, subscribe to the New York Review of Books for a year and try to get into their head-space for a few hours a month.

3.) If you are a rationalist, subscribe to Fate magazine for a year.

4.) If you are an occultist, join the Committee for the Scientific Investigation of Claims of the Paranormal and read their journal, The Skeptical Inquirer, for a year.

5.) Buy a copy of the Scientific American and read any article in it. Ask the following questions: Why do they sound so sure? Does the data support dogmatism at this point, or is dogma a primate habit (defending head-space)? Will these theories still be believed in 2011*? In 2593?

[* - my copy from 2001]

6.) Get into a discussion of philosophy with an educated Marxist, an intelligent Muslim, and a Japanese businessman at the first opportunity.

7.) Buy some ZOOM or LIFT (two names for the same caffeine-high stimulant) at a health food store. (This gives a close approximation of the effects of illegal cocaine). When you are zooming or lifting and your mind is racing, find a victim and explain the universe to them until they are able to escape you. What you experience in this speed rap is what the head of the compulsive rationalist is always like. This is the verbal circuit gone wild and totally oblivious to information coming in on any other circuit. It explains why most people cannot stand rationalists. Speed drugs apparently trigger neurotransmitters characteristic of the verbal centers of the left cortex.

The "hot-topic list" goes to what appears to be one of your drafts, which we normal people can't read. [EDIT: It doesn't any more.]

What do you do instead, once you've stopped the process?

Skill: negotiation - deliberately reaching a mutually beneficial trade or agreement, even in situations of slight power imbalance. Important for rationalists who aim at earning money as an instrumental goal.

(At the 5-second level a key component of this is learning to say "no", being able to overcome one's agreeableness as the default decision.)

(For some reason negotiation in situations of extreme power imbalance seems like it should have a different name, and I don't know what that should be.)

(For some reason negotiation in situations of extreme power imbalance seems like it should have a different name, and I don't know what that should be.)

Dominance or Authority spring to mind. In this video Steven Pinker argues that there are three basic relationship types, authority, reciprocity and communality, and negotiation in extreme power imbalance sounds like it uses the social rules for authority rather than reciprocity.

For some reason negotiation in situations of extreme power imbalance seems like it should have a different name, and I don't know what that should be.

I think that is called 'losing' or 'winning'. :)

Because normal comments will inevitably get mixed up with those that satisfy the requested sort of post, I recommend you bold the words "Skill", "Subskill" and "Exercise".