(The Exercise Prize series of posts is the Center for Applied Rationality asking for help inventing exercises that can teach cognitive skills.  The difficulty is coming up with exercises interesting enough, with a high enough hedonic return, that people actually do them and remember them; this often involves standing up and performing actions, or interacting with other people, not just working alone with an exercise booklet and a pencil.  We offer prizes of $50 for any suggestion we decide to test, and $500 for any suggestion we decide to adopt.  This prize also extends to LW meetup activities and good ideas for verifying that a skill has been acquired.  See here for details.)


Exercise Prize:  Check Consequentialism

In philosophy, "consequentialism" is the belief that doing the right thing makes the world a better place, i.e., that actions should be chosen on the basis of their probable outcomes.  It seems like the mental habit of checking consequentialism, asking "What positive future events does this action cause?", would catch numerous cognitive fallacies.

For example, the mental habit of consequentialism would counter the sunk cost fallacy - if a PhD wouldn't really lead to much in the way of desirable job opportunities or a higher income, and the only reason you're still pursuing your PhD is that otherwise all your previous years of work will have been wasted, you will find yourself encountering a blank screen at the point where you try to imagine a positive future outcome of spending another two years working toward your PhD - you will not be able to state what good future events happen as a result.

Or consider the problem of living in the should-universe; if you're thinking, I'm not going to talk to my boyfriend about X because he should know it already, you might be able to spot this as an instance of should-universe thinking (planning/choosing/acting/feeling as though within / by-comparison-to an image of an ideal perfect universe) by having done exercises specifically to sensitize you to should-ness.  Or, if you've practiced the more general skill of Checking Consequentialism, you might notice a problem on asking "What happens if I talk / don't talk to my boyfriend?" - providing that you're sufficiently adept to constrain your consequentialist visualization to what actually happens as opposed to what should happen.

Discussion:

The skill of Checking Consequentialism isn't quite as simple as telling people to ask, "What positive result do I get?"  By itself, this mental query is probably going to return any apparent justification - for example, in the sunk-cost-PhD example, asking "What good thing happens as a result?" will just return, "All my years of work won't have been wasted!  That's good!"  Any choice people are tempted by seems good for some reason, and executing a query about "good reasons" will just return this.

The novel part of Checking Consequentialism is the ability to discriminate "consequentialist reasons" from "non-consequentialist reasons" - being able to distinguish that "Because a PhD gets me a 50% higher salary" talks about future positive consequences, while "Because I don't want my years of work to have been wasted" doesn't.

It's possible that asking "At what time does the consequence occur and how long does it last?" would be useful for distinguishing future-consequences from non-future-consequences - if you take a bad-thing like "I don't want my work to have been wasted" and ask "When does it occur, where does it occur, and how long does it last?", you will with luck notice the error.

Learning to draw cause-and-effect directed graphs, a la Judea Pearl and Bayes nets, seems like it might be helpful - at least, Geoff was doing this while trying to teach strategicness and the class seemed to like it.

Sometimes non-consequentialist reasons can be rescued as consequentialist ones.  "You shouldn't kill because it's the wrong thing to do" can be rescued as "Because then a person will transition from 'alive' to 'dead' in the future, and this is a bad event" or "Because the interval between Outcome A and Outcome B includes the interval from Fred alive to Fred dead."

On a five-second level, the skill would have to include:

  • Being cued by some problem to try looking at the consequences;
  • Either directly having a mental procedure that only turns up consequences, like trying to visualize events out into the future, or
  • First asking 'Why am I doing this?' and then looking at the justifications to check if they're consequentialist, perhaps using techniques like asking 'How long does it last?', 'When does it happen?', or 'Where does it happen?'.
  • Expending a small amount of effort to see if a non-consequentialist reason can easily translate into a consequentialist one in a realistic way.
  • Making the decision whether or not to change your mind.
  • If necessary, detaching from the thing you were doing for non-consequentialist reasons.

In practice, it may be obvious that you're making a mistake as soon as you think to check consequences.  I have 'living in the should-universe' or 'sunk cost fallacy' cached to the point where as soon as I spot an error of that pattern, it's usually pretty obvious (without further deliberative thought) what the residual reasons are and whether I was doing it wrong.

Pain points & Pluses:

(When generating a candidate kata, almost the first question we ask - directly after the selection of a topic, like 'consequentialism' - is, "What are the pain points?  Or pleasure points?"  This can be errors you've made yourself and noticed afterward, or even cases where you've noticed someone else doing it wrong, but ideally cases where you use the skill in real life.  Since a lot of rationality is in fact about not screwing up, there may not always be pleasure points where the skill is used in a non-error-correcting, strictly positive context; but it's still worth asking each time.  We ask this question right at the beginning because it (a) checks to see how often the skill is actually important in real life and (b) provides concrete use-cases to focus discussion of the skill.)

Pain points:

Checking Consequentialism looks like it should be useful for countering:

  • Living in the should-universe (taking actions because of the consequences they ought to have, rather than the consequences they probably will have).  E.g., "I'm not going to talk to my girlfriend because she should already know X" or "I'm going to become a theoretical physicist because I ought to enjoy theoretical physics."
  • The sunk cost fallacy (choosing to prevent previously expended, non-recoverable resources from having been wasted in retrospect - i.e., avoiding the mental pain of reclassifying a past investment as a loss - rather than acting for the sake of future considerations).  E.g., "If I give up on my PhD, I'll have wasted the last three years."
  • Cached thoughts and habits; "But I usually shop at Whole Foods" or "I don't know, I've never tried an electric toothbrush before."  (These might have rescuable consequences, but as stated, they aren't talking about future events.)
  • Acting-out an emotion - one of the most useful pieces of advice I got from Anna Salamon was to find other ways to act out an emotion than strategic choices.  If you're feeling frustrated with a coworker, you might still want to Check Consequentialism on "Buy them dead flowers for their going-away party" even though it seems to express your frustration.
  • Indignation / acting-out of morals - "Drugs are bad, so drug use ought to be illegal", where it's much harder to make the case that countries which decriminalized marijuana experienced worse net outcomes.  (Though it should be noted that you also have to Use Empiricism to ask the question 'What happened to other countries that decriminalized marijuana?' instead of making up a gloomy consequentialist prediction to express your moral disapproval.)
  • Identity - "I'm the sort of person who belongs in academia."
  • "Trying to do things" for simply no reason at all, while your brain still generates activities and actions, because nobody ever told you that behaviors ought to have a purpose or that lack of purpose is a warning sign.  This habit can be inculcated by schoolwork, wanting to put in 8 hours before going home, etc.  E.g. you "try to write an essay", and you know that an essay has paragraphs; so you try to write a bunch of paragraphs but you don't have any functional role in mind for each paragraph.  "What is the positive consequence of this paragraph?" might come in handy here.

(This list is not intended to be exhaustive.)

Pleasure points:

  • Being able to state and then focus on a positive outcome seems like it should improve motivation, at least in cases where the positive outcome is realistically attainable to a non-frustrating degree and has not yet been subject to hedonic adaptation.  E.g., a $600 job may be more motivating if you visualize the $600 laptop you're going to buy with the proceeds.

Also, consequentialism is the foundation of expected utility, which is the foundation of instrumental rationality - this is why we're considering it as an early unit.  (This is not directly listed as a "pleasure point" because it is not directly a use-case.)

Constantly asking about consequences seems likely to improve overall strategicness - not just lead to the better of two choices being taken from a fixed decision-set, but also having goals in mind that can generate new perceived choices, i.e., improve the overall degree to which people do things for reasons, as opposed to not doing things or not having reasons.  (But this is a hopeful eventual positive consequence of practicing the skill, not a use-case where the skill is directly being applied.)

Teaching & exercises:

This is the part that's being thrown open to Less Wrong generally.  Hopefully I've described the skill in enough detail to convey what it is.  Now, how would you practice it?  How would you have an audience practice it, hopefully in activities carried out with each other?

The dumb thing I tried to do previously was to have exercises along the lines of, "Print up a booklet with little snippets of scenarios in them, and ask people to circle non-consequentialist reasoning, then try to either translate it to consequentialist reasons or say that no consequentialist reasons could be found."  I didn't do that for this exact session, but if you look at what I did with the sunk cost fallacy, it's the same sort of silly thing I tried to do.

This didn't work very well - maybe the exercises were too easy, or maybe it was that people were doing it alone, or maybe we did something else wrong, but the audience appeared to experience insufficient hedonic return.  They were, in lay terms, unenthusiastic.

At this point I should like to pause, and tell a recent and important story.  On Saturday I taught an 80-minute unit on Bayes's Rule to an audience of non-Sequence-reading experimental subjects, who were mostly either programmers or in other technical subjects, so I could go through the math fairly fast.  Afterward, though, I was worried that they hadn't really learned to apply Bayes's Rule and wished I had a small little pamphlet of practice problems to hand out.  I still think this would've been a good idea, but...

On Wednesday, I attended Andrew Critch's course at Berkeley, which was roughly mostly-instrumental LW-style cognitive-improvement material aimed at math students; and in this particular session, Critch introduced Bayes's Theorem, not as advanced math, but with the aim of getting them to apply it to life.

Critch demonstrated using what he called the Really Getting Bayes game.  He had Nisan (a local LWer) touch an object to the back of Critch's neck, a cellphone as it happened, while Critch faced in the other direction; this was "prior experience".  Nisan said that the object was either a cellphone or a pen.  Critch gave prior odds of 60% : 40% that the object was a cellphone vs. pen, based on his prior experience.  Nisan then asked Critch how likely he thought it was that a cellphone or a pen would be RGB-colored, i.e., colored red, green, or blue.  Critch didn't give exact numbers here, but said he thought a cellphone was more likely to be primary-colored, and drew some rectangles on the blackboard to illustrate the likelihood ratio.  After being told that the object was in fact primary-colored (the cellphone was metallic blue), Critch gave posterior odds of 75% : 25% in favor of the cellphone, and then turned around to look.

Then Critch broke up the class into pairs and asked each pair to carry out a similar operation on each other:  Pick two plausible objects and make sure you're holding at least one of them, touch it to the other person while they face the other way, prior odds, additional fact, likelihood ratio, posterior odds.

This is the sort of in-person, hands-on, real-life, and social exercise that didn't occur to me, or Anna, or anyone else helping, while we were trying to design the Bayes's Theorem unit.  Our brains just didn't go in that direction, though we recognized it as embarrassingly obvious in retrospect.

So... how would you design an exercise to teach Checking Consequentialism?

SotW: Check Consequentialism
New Comment
309 comments, sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings
[-]Axel340

So... how would you design an exercise to teach Checking Consequentialism?

I would check to see if such a thing already exists or if there are people who have experience designing such things. I know of a Belgian non-profit 'Center for Informative Games' that not only rents games designed to teach certain skills but will also help you create your own.

From their site: On request C.I.S. develops games for others. The applicant provides the content of the game, while C.I.S. develops the conceptual and game technical part to perfection. The applicant has the opportunity to attend some game tests and to redirect when necessary.

They also offer coaching if you want to work on your own: Do you want to create a game concept on your own, but you don't know where to start? No worries C.I.S. can give you a hand. During a number of concrete working sessions C.I.S. facilitates your development. In between sessions the applicant continues the work to, finally, end up with a solid game.

I have enjoyed their games in the past and can attest to their quality. The obvious problem is that it's a purely Belgian organization and the 'search' function only works with Dutch words. However if you want to check them out I'd be happy to act as a go-between. Since a couple of months there is even a Brussels LW meetup, I'm certain I could get a couple of members to help in the production process (again, if this seems interesting)

In a group, with a leader who knows the exercise:

Get a volunteer to act as a judge (or a few to act as a jury, in a large group). Have her leave the room. The leader presents the rest with a short set of Contrived Hypothetical Situations, each with finite options and either clearly-defined outcomes for each option, or a probabilistic distribution of outcomes for each option. The leader says, "Please write down your choice for each problem, sign your paper, and turn it in to me. Then I'll call in the judge, and have her decide on each problem. You get a point wherever her decision agrees with yours. The winner is the one with the most points." When the judge is called in, however, the leader doesn't tell them the actual problems. Rather, the leader just reports the outcomes (or distributions), and asks them to choose which outcome or distribution is best. The winners are announced based on that.

Example: One of the situations given is some variant of the trolley problem. When the judge comes in, she is just asked whether she'd prefer one person to get hit by a trolley, or five. Everybody laughs as she replies "...one?"

Example: The problem given to the g... (read more)

3JKR
I think this is a fantastic idea, with a patch that is much easier than those suggested by the other responses. Simply tell everyone that for the purposes of this exercise, only that information directly presented in the example is to be considered. People sometimes overlook relevant information or clever third options, and these situations are to be judged only based on the data being considered by the hypothetical person in the given scenario. If there is any concern about this set up encouraging people to think about things with an insufficient amount of thoroughness, you can save some time at the end for a just-for-fun period where everyone gets to offer their clever workarounds and extra information that would have changed what the proper decision was, had it been considered.
1A1987dM
Two details the judge isn't told about are 1) you would have to pay for the former meal, but not for the latter, and 2) if you stay in the restaurant, you gain useful information you'll be able to take in account the next time you might want to eat there.
1HonoreDB
1) is patchable by specifying that the leftovers are non-perishable, so eating them is equivalent to buying a meal. 2) Either the judge is told that the variable meal is repeatable if it's good, or we specify in the group problem that you're not going back there no matter what.
0AspiringKnitter
Couldn't the problems others have brought up regarding this scenario be fixed by specifying that this is your last meal ever before the world ends tomorrow morning before breakfast? Then neither information nor money is valuable anymore.
0dlthomas
I think I'd make a decision other than "try that new restaurant on the outskirts of town" for the evening before the world ends. If I don't know the world is going to end, then my decision now mightn't be optimal in light of that additional information (maybe that still tests something interesting, but it isn't quite the same thing).
0AspiringKnitter
Hmm. That could be a good point. If the world were ending, I probably wouldn't waste time on a sit-down meal. How about if it's your last day in the country and you'll be fleeing to escape religious persecution tomorrow, taking nothing with you?
0Zvi
If you stay, you gain information about the restaurant. There's the dollar cost of dining out. It's actually not as easy as it looks to generate a "clean" example. How much need we worry about excluding consequences we can't consciously list and/or quantify?
2Raemon
I patched this example by saying "you're on vacation in another city", so the value of information is mostly negligible. But yeah, it's still pretty hard. Also, ideally not all of our examples end up being instances of sunk-cost-fallacy.

The novel part of Checking Consequentialism is the ability to discriminate "consequentialist reasons" from "non-consequentialist reasons" - being able to distinguish that "Because a PhD gets me a 50% higher salary" talks about future positive consequences, while "Because I don't want my years of work to have been wasted" doesn't.

It's possible that asking "At what time does the consequence occur and how long does it last?" would be useful for distinguishing future-consequences from non-future-consequences - if you take a bad-thing like "I don't want my work to have been wasted" and ask "When does it occur, where does it occur, and how long does it last?", you will with luck notice the error.

"Intense, long, certain, speedy, fruitful, pure—
Such marks in pleasures and in pains endure.

— Jeremy Bentham's mnemonic for the signs of "consequentialist reasons"

EDIT: It occurs to me that I should explain this more. Bentham was trying to popularize consequentialism and remind his readers of what sorts of things count as consequentialist reasons to prefer a particular outcome. Eliezer suggests that we sho... (read more)

Cleverness-related failure mode (that actually came up in the trial unit):

One shouldn't try too hard to rescue non-consequentialist reasons. This probably has to be emphasized especially with new audiences who associate "rationality" to Spock and university professors, or audiences who've studied pre-behavioral economics, and who think they score extra points if they come up with amazingly clever ways to rescue bad ideas.

Any decision-making algorithm, no matter how stupid, can be made to look like expected utility maximization through the transform "Assign infinite negative utility to departing from decision algorithm X". This in essence is what somebody is doing when they say, "Aha! But if I stop my PhD program now, I'll have the negative consequence of having abandoned a sunk cost!" (Sometimes I feel like hitting people with a wooden stick when they do this, but that act just expresses an emotion rather than having any discernible positive consequences.) This is Cleverly Failing to Get the Point if "not wanting to abandon a sunk cost", i.e., the counterintuitive feel of departing from the brain's previous decision algorithm, is treated ... (read more)

This probably has to be emphasized especially with new audiences who associate "rationality" to Spock and university professors, or audiences who've studied pre-behavioral economics, and who think they score extra points if they come up with amazingly clever ways to rescue bad ideas.

One of the other models people have for the rationalizing sort of "rationality" is that of lawyers.

Lawyers are very good at logic — the LSAT, the entrance examination for U.S. law schools, leans heavily on logic puzzles — but the whole point of being a trial or appeals lawyer is to come up with clever (and socially respectable) arguments for whatever position your client may have at the moment.

This extends past real-world lawyerhood. The tabletop role-playing game crowd have the expression "rules lawyer" for a person who comes up with clever arguments for why their character should get away with whatever they want to at the moment.

2pnrjulius
Indeed I think this is the central problem with the way most people use their powers of reasoning. (It even has a name: "the argumentative theory of reason".) They start with a conclusion, and work backwards to find rational (or at least rational-sounding) ways of supporting that conclusion. We all do this automatically; it may be the very thing our brains evolved to do. We have to work very hard to get ourselves to do the opposite, start with evidence and use reasoning based on the evidence to decide on our conclusion. I'd say most scientists manage to do this right maybe half the time, and most laypeople almost never manage it.
[-]pjeby130

Sometimes I feel like hitting people with a wooden stick when they do this, but that act just expresses an emotion rather than having any discernible positive consequences.)

My normal response is, "so what's bad about that?" and go a few rounds until the person has to struggle for an answer... the teachable moment where I can say, "you see what you're doing? you're just making stuff up. What's actually going to happen?"

(That being said, it would definitely have been helpful for me in the past if I had thought to confine questions of consequences to things happening at a point-in-time. I eventually figured out that I needed to ask that for things people were thinking about or remembering, but there was a long time where I also had the hit-them-with-a-stick frustration to this kind of response.)

The only suggestion I have for exercises is to make people write down their own thinking (or state their thinking out loud), and then read it back as a kind of grammar-checking exercise. Are these abstract nouns or concrete nouns? Do they describe a point in time or some sort of vague non-timey thing?

I've done some similar things with small groups, though, and one ... (read more)

9[anonymous]
This is perhaps ironic because I have been going through precisely this PhD sunk-cost problem for the past few months, but regret bias is a serious part of behavior psychology. I've been dissatisfied with the direction that publication standards are moving in my current field (computer vision) for a while, and as a result have had a tough time finding an adviser/project match that would let me do things at a more abstract mathematical level. No one is very interested in those papers. Ultimately, over a two-year period, I reasoned that it was better for me to leave the PhD program, find a job that allowed me to pursue certain goals, and to leave research ideas to my own spare time. The single most difficult hurdle in reaching this decision was feeling very worried that I would regret leaving my institution (Harvard) because everyone tells me that a PhD from Harvard "opens lots of doors" and lots of people who I trust and think are non-trivially intelligent have insisted that unpleasantly sticking it out in the PhD program just to obtain the credential is absolutely the best thing. My own assessment is that I will do just fine without that particular credential and that being able to use personal time to pursue the research I care about, even if I ultimately am not talented enough to publish any of it on my own, will be more fulfilling. But this was a damn hard conclusion to come by. I felt stressed and nervous, concerned that I will hate my future job's working conditions and beat myself up over not sticking it out at Harvard. I largely made it into Harvard through sheer, stupid ability to work unreasonably long hours to self-teach. That is, by stubbornly never quitting; it's not easy, however rational I wish to be, to feel free of these kinds of self-identity stigmas (e.g. don't be a quitter). I guess what I'm trying to say is that perceived future pain of regretting a decision is a legitimate consequence to consider. And sometimes that is absolutely a consequence
8GDC3
I think it's important to try to convert the reason to a consequentialist reason every time actually; it's just that one isn't done at that point, you have to step back and decide if the reason is enough. Like the murder example one needs to avoid dismissing reasons for being in the wrong format. "I don't want to tell my boyfriend because he should already know" translates to: in the universe in which I tell my boyfriend he learns to rely on me to tell him these things a little more and his chance of doing this sort of thing without my asking decreases in the future. You then have to ask if this supposed effect is really true and if the negative consequence is strong enough, which depends on things like the chances that he'll eventually figure it out. But converting the reason gets you answering the right questions. Sunk cost fallacy could be a sign that you don't trust your present judgement compared to when you made the original decision to put the resources in. The right question is to ask why you changed your mind so strongly that the degree isn't worth it even at significantly less additional cost. It is because of new information, new values, new rationality skills or just being in a bad mood right now. An advantage is that you feel just as clever for coming up with the right questions whatever you decide, which out to make this a bit easy to motivate yourself to implement.
1handoflixue
Definitely useful. I personally find the two have a very different emotional/internal "flavor" - I can tell when I want to avoid a sunk cost vs when I'm in a bad mood and just don't want to deal with a cost - but that's not necessarily always true of me, much less anyone else.
7jimmy
I wouldn't even allow that. I much prefer to treat such a sense as a (misguided) signal about the map, rather than a piece of territory that I intrinsically care about. Seeing things with this framing allows you to explore the signals with less distortion, and allows them to go away more easily once you take them into account. If you start treating them as things to worry about, then you get sadness about sadness, fear about fear, and other information cascades that can be quite destructive. Additionally, on the cases where the irrational discomfort actually sways your decision over the threshold, you're training yourself to listen to things that should not exist in the first place, which just reinforces the problem.
7handoflixue
This strikes me as a perfect lead-in to Spock style "Bah, my emotions SHOULDN'T exist, therefor I will just IGNORE them". This does not work well. If we ignore a REAL negative consequence in our planning, we're going to get frustrated when the consequence happens anyway, because now it's an UNEXPECTED negative consequence of our decision. If we further decide that we're not REALLY having that negative consequence, then it will get further exacerbated by our unwillingness to accept the situation, and therefor our inability to actually do anything to fix the situation. It's entirely possible that we're now miserable for two weeks instead of three days. Heck, It's entirely possible the whole thing could have been fixed by thinking about it and saying "I would normally feel bad, but since I'm aware of this, I can instead just remind myself of the awesome rational decision I'm making, and how cool my life is because of this Rationality thing!", possibly supplemented by a celebratory slice of cake to reinforce that this is a positive, not a negative, event. (And cake makes everyone happy!)
4jimmy
No no no, not that. That's terrible! "listen" is ambiguous - oops. You want to acknowledge the feeling, but not act on it. Once you can acknowledge it, you can realize that it doesn't make sense, and then release that feeling and be done with it.
5handoflixue
If I'm hungry, I can't just ignore that and continue to function at 100%. I can go eat and restore my blood sugar, or I can delay that hunger and function at less-than-peak efficiency because my body does not have all the resources it needs. Emotions are the same way - if I feel upset or a sense of loss, I have to address that emotion. This is not always a simple "acknowledge and release" 5 minute process. Believing otherwise will screw me up just as badly as believing I can cure hunger by "acknowledging and releasing" it instead of eating lunch.
4jimmy
I think we agree a lot more than you realize. Pretending that you aren't feeling emotion that you are feeling is a recipe for disaster. In your analogy, I recommend the equivalent of eating. However, this doesn't mean that you yield to the emotions when they're pushing you towards bad decisions. It also doesn't mean you pretend that it has to be some big ordeal to fix the problem right. Those are both very bad ideas for more reasons than are obvious. "eating" can be anywhere from a split second automatic response to a extended ordeal. If you know what you're doing, the Phd example is not more than a 5 minute process - I've walked people through worse things in about that time.
4Blueberry
Please elaborate!
4jimmy
I "cheated" a bit, in that I had them spend ~15-20 minutes with a chat bot that taught them some skills for getting in touch with those parts of their mind. Actually working through the problem was a few minutes of text chat that basically pointed out that there was no magic option and that they needed to let go of the problem emotions. All the real magic was in putting them in the state of mind to shut up and listen. I talk about it a bit here
0handoflixue
I suppose the best analogy I could offer here is getting robbed. It takes maybe 5 minutes to get robbed. There's (usually) nothing you can do to fix the situation or recoup the cost. But people still feel bad about it for a while. Your link seems to suggest, more or less, using hypnosis to just wipe out this guilt - except the examples you give don't really seem to address that emotional side at all. You're focusing on the intellectual acceptance of "yes, I should drop the PhD", which isn't what I'm talking about. I'm talking about the emotional baggage that comes with that, the sense that you've wasted 2 years as a sunk cost that isn't even doing you any good at this point. If you're really hypnotizing way that guilt, that emotional response, then I guess I am misunderstanding you. Is that the case? Because I would say that, based on my personal experience, that is seriously dangerous territory. Not to say you shouldn't trust yourself with it - I do it myself. But it is a technique I have seen cause a lot of people serious problems, and definitely not one I'd teach casually.
0jimmy
Haha! YES! Yep. They're doing it wrong. No no no no no! I'm talking about the emotional acceptance. It is a very different thing than intellectual acceptance, but that does not mean they can't track each other. If your mind is organized well, they do track. Have you read Kaj's post on overcoming suffering and suffering as attention allocation conflicts? This is basically what I'm talking about There is a very important distinction between suppressing emotion (perhaps successfully) and eliminating the cause of the emotion directly by coming up with a better way of handling the conflict. The latter is healthy and quite low risk compared to the null option. This is what I do - with or without "hypnosis". Suppressing emotion is a recipe for disaster.
1Strange7
It would have the consequence of conditioning in the subject's mind an association between a particular thought process and being hit with a stick. Most people don't like being hit with sticks, so the association is likely to make them avoid that particular thought process. Do you not consider "teaching people to avoid a dangerously stupid thought process" a positive consequence?
4Bluehawk
Actually they would associate the stick with a number of things, including but not limited to the stupid thought process. They would be quite likely to associate the stick with their encounter with Eliezer, and to their (failed) attempt to converse with and/or follow his thought processes. Mind: They associate the stick with all aspects of the attempt, not only with the failure. It might work in a Master/Apprentice scenario where the stick-hitting-victim is bindingly pre-committed to a year of solitude with Stick-Happy!Eliezer in order to learn from him the art of Cognitive Kung Fu. This is the only scenario I can immediately visualize in which the stick-hitting victim would not immediately decide that Stick-Happy!Eliezer is a person they can get away with avoiding, and possibly with reporting to the police for assault. EDIT01: This is assuming that the experiential sample size is 1.
1Strange7
I was only pointing out that arguably-positive consequences would be present. I agree that they most likely would not predominate outside controlled conditions, and the overall decision not to engage in spontaneous armed assault was a wise one.
1Will_Newsome
Rationalization is an important skill and should be rewarded, not punished. If you never try to rationalize others' decisions then you won't notice when they actually do have a good justification, and if you never practice rationalization then you'll never get good enough at it to find their justifications when they exist. The result is gross overconfidence in the stupidity of the opposing side and thus gross overconfidence in one's own rationality. That leads to tragedies and atrocities, both personal and societal.
8Alicorn
Perspective-taking is a separate "skill" from rationalizing one's own behavior.
5Will_Newsome
Hm, is perspective-taking the same skill that I was talking about? I can't tell. Also I thought that Eliezer's examples were phrased in the hypothetical, and thus it'd be rationalizing others' beliefs/behavior, not one's own. I'm not sure to what extent rationalizing a conclusion and rationalizing one's own behavior are related. Introspectively, the defensiveness and self-justifying-ness inherent to the latter makes it a rather different animal.
6handoflixue
"Coming up with explanations" is a good skill. "Coming up with a single, stupid explanation, failing to realize it is stupid, and then using it as an excuse to cease all further thought" is a very, very bad skill. Thinking "well, but abandoning a sunk cost actually IS a negative future event" is smart IFF you then go "I'd be miserable for three days. How does that weigh against years spent in the program?" It's very, very bad, however, if you stop there and continue to spend 2 years on a PhD just because you don't want to even THINK about those three days of misery. I think understanding this dichotomy is critical. If you stop even thinking "well, but abandoning a sunk cost IS a negative future event" because you're afraid of falling in to the trap of then avoiding all sunk costs, then you're ignoring real negative consequences to your decisions.

Not attempting to answer the question, but I've been nursing a thought about the rationality org that might be useful: The nearby Stanford University has a world-renown program in "decision sciences" http://decision.stanford.edu/ which is basically "how to make decisions scientifically"; they virtually invented influence diagrams, they teach biases as a part of the program, etc. The head of the program, Ronald Howard, also co-founded http://www.decisioneducation.org/ , his teen-oriented "rationality org".

  • there is probably things to learn from both

  • if "rationality org" has a value proposition to these organizations they can be useful in teaching opportunities and for credibility building

3Vladimir_Nesov
The reading list (books) from decisioneducation.org.
3Eliezer Yudkowsky
These do indeed sound like people to talk to, and local too - thanks!
0Dr_Manhattan
The other guy to talk to (unless you decide to strategically approach younger faculty) is Ross Shachter, inventor of bayes-ball algorithm and some other interesting AI papers.
0MichaelVassar
Please tell me how that goes. Also, you planning to look into this, or Anna et al?
2Incorrect
A comma broke one of the links in your post.

I am reminded of a game we played in elementary school:

There are 100 pieces of candy in a jar, and 20 students. Each student gets to vote "share" or "selfish". If EVERYONE votes to share, the candy is split evenly. If ONE person votes "selfish", they get all the candy. If MORE than one person votes "selfish", no one gets candy, and the experiment is repeated until either the candy is distributed, or X iterations (3-5 seems normal) have passed.

Before each iteration, the students are allowed to discuss strategy. The solution, of course, is for a single trustworthy person to make a binding commitment to vote "selfish" and then evenly distribute the candy. By pre-commiting to vote "selfish", everyone else knows that voting "selfish" themselves will result in no candy - unlike a commitment to have everybody share.

I've always considered it a decent test of group rationality and social skills whether or not this consensus actually gets reached before the final iteration. I've seen groups that hit on this, had a single iteration with a few outliers testing it just to be sure that the person would really vote "selfish" like they said, and then implementing the strategy. I've seen others where 10-20% of the audience simply would not believe the person who made the pre-commitment, and so there was never a successful iteration

0TheOtherDave
This all hinges on trusting the supposedly trustworthy person, of course.
0handoflixue
Yes, but this being a school setting, "I made a promise to the teacher" is usually considered sufficiently binding. For a more advanced level of rationality, I'd say that coming up with a plausibly binding commitment strategy would be part of the challenge :)
-1AspiringKnitter
Doesn't that rely on everyone eating candy? One person who doesn't eat candy and therefore isn't invested in the outcome could wreck that. Also: theoretically, a student could win hundreds of pieces of candy? I'm sure the parents were very happy about that.
0handoflixue
Anyone can ruin it deliberately, true - it works best with a cooperative group, not a competitive one. Modifying it for a competitive group would definitely remove it from the real of "useful introductory ideas", but would probably still be a useful exercise for more advanced classes. Candy is also concrete and engaging - most people don't respond as enthusiastically to raffle tickets or $0.25. As part of a larger set of challenges, using play money with some sort of modest exchange of play money -> small prizes at the end of the session might work.
0Nornagest
Doesn't need to be candy, necessarily. Money's the first thing that comes to mind, but if that's prohibitively costly to maintain while keeping the prizes attractive, you could play with probabilities: build a set of diverse prizes of approximately equal value, say, and instead of money or candy distribute tickets to a raffle where the winner'd be able to choose one prize. That might have some funny consequences in this experiment, though, depending on the quirks of how people think about probabilities.

An important question to ask that you are leaving out is "What are my alternatives to this course of action?"

The comparison of consequences requires an alternative set of consequences to compare to. Considering the question "Should I be in graduate school?" The answer may well be different if the alternative is getting a decent job than if the alternative is starving unemployment.

The listing of alternatives also helps catch cheating. If the alternative is implausible and disastrous (Stay in grad school or be a hobo) then it is likely that Checking Consequentialism isn't being done seriously.. The alternative compared needs to be a serious answer to the question "What would I do if I couldn't/wouldn't do this?"

7daenerys
I think an "Improv to Rationality" class would be very fun and interesting. I don't know how WELL the games would work to teach the desired skills, but I definitely think it would make the lessons more memorable, AND you'd be able to add the phrase "Team-Building" to your class description. You can sell anything if you promote it as "Team-Building." :P A game that would teach how to look for alternative options (see parent) could be "New Choice". In this game, a scene is played out, with the moderator occasionally demanding a new choice(s) from the players. (Video Example (includes 15 sec commercial)). Not only would it teach about not becoming too set on a current course of action, but could have the lesson of: "When thinking up alternative actions, don't just stop at one or two. Instead imagine a moderator yelling "New Choice!" at you. Keep thinking up new alternatives until you get to crazy-land (i.e. "Here's a cat from my pants") Another sub-skill mentioned, was the ability to recognize bad rationalizations (i.e. "I will feel bad if I quit something I put so much effort into."). Perhaps one way to learn to recognize these, is to do bad rationalization ON PURPOSE, to see what it feels like. A good game for that is "Challenge in a Minute". In this game, a two-sided silly debate is chosen, such as pirates v. ninjas, or Coke v. Pepsi. All the players line up and challenge each other's arguments. Arguments are supposed to start somewhat seriously ("Ninjas are sneakier") and devolve over the course of the game ("I don't like his pants"). (It's easier to just watch a video, than explain the rules.) Participants could then brainstorm what it felt like to come up with the bad rationalizations. I would expect answers like: Grasping at straws, Searching your brain for things to support your position, Being proud of clever retorts, etc. Participants could then ACTUALLY try to answer the debate question (ninja v. pirates, or whatever) in teams, and then discuss what it
-6wedrifid

This seems like a comparatively reliable procedure: imagine a collection of possible worlds generated by possible actions; explain what future events distinguish between these worlds in a way that makes one of them preferable to the others; then choose the action that leads there.

Paying attention to events that distinguish these possible futures from each other guards against errors such as comparing to status quo or should-world (neither of which is among the possible futures), or worse comparing an arbitrarily picked-out event (in one of the possible futures) to an arbitrary anchor (i.e. a cost that feels excessive in some absolute way, and not by comparison to alternatives). Focusing on future events and not past events or actions themselves guards against deontological and identity-based arguments, such as "this is a proper action", "he shouted first" or "because I'm a human being".

Saying "positive consequence" sounds like a bad habit to me: positive compared to what? The comparison should be among the alternatives, without anchoring on some baseline of neutrality such that some consequences are more "positive" than that.

Concreteness Game The object of this game is to train players to generate examples for explaining concepts quickly. The game requires at least two people, but may work better with groups of three or four.

To play, one of the players (Asker) names a concept to be explained, such as "How do you traverse a linked nodal network", "Explain the law of Conservation of Energy", or "What constitutes good financial advice?"

The other player (Generator) then tries to explain the concept/skill/other by using a nearby object to assist. The ... (read more)

You could use mazes where your score is -(total distance traveled) . First, give a simple maze with two obvious paths, A and B but Path A is much shorter. Then give a second maze identical to the first but you are taking over from another player who has already gone down Path B but the shortest way to the exit is to double back and then go down Path A. Then give the same maze but now there is an obstacle on Path A that you must go around if you take this path and so it's now optimal to go on Path B. The obstacle was placed there for some unfair reason ... (read more)

4handoflixue
And then you collapse in to short vs long-term gains. Giving in to discrimination is a net long-term loss, since then you'll face it in the future... Also, for initial teaching, you want to present the SIMPLEST possible version. Rationality 102 is when you start introducing enemy agents, and then eventually combine that idea with this one after BOTH "sabotaging agents" and "consequentialism" are understood.

Kahneman suggests such an exercise for groups after pointing out that organizations generally act more rationally than individuals. The devil's advocate role and thinking at the worst possible outcome. We don't always have the luxury of having others near us for checking our thoughts. But we often have imaginary conversations with friends or parents. So it shouldn't be very difficult to assign a devil's advocate position to a imaginary voice. That should put in perspective the way we feel about the subject. It is a basic mean of delaying the strong coherence of the first good narrative.

Maybe it would be great to have an imaginary Bayesian friend...

This wouldn't work as the only exercise, but could be useful if paired with another.

Presumably all students have other things they could be doing with their time, some of it possibly fun. Near the end of the lesson, perhaps just before the last exercise, with maybe 15-20 minutes left in the session, present this option: You can leave right now and get on with your day.

Obviously this is always an option, no one is required to stay against their will, but it's usually considered bad form and it's never given as an explicit option. Tell everyone to think abou... (read more)

2handoflixue
That is an AWESOME way to grade the success of the lesson!! If half the audience leaves, then either the audience still isn't making good choices (in which case you clearly didn't teach them well), or they made the correct choice and your lesson genuinely isn't worth their time. (Or you're teaching an audience that isn't ready for it, but that's still a failing in the teaching, just on a more administrative "select your audience better" level.)

I think of this as "looking effectward", one of the basic directions in concept space (opposite causeward, making it the inverse operation of asking "why").

Another inferential path: it may be valuable to differentiate them as attitudes and events. If my motivation for getting a PhD is "I will feel terrible if I do not get a PhD", that's an attitude motivation, which in theory I have internal control over. If my motivation for getting a PhD is that I will get hired for a different sort of job, that's an event motivation, for which control is primarily external. I don't have control over whether or not a variety of job requires a PhD, but I do have control over whether or not my attitude will be negat... (read more)

I think an important pre-skill is to decide what your goals are BEFORE checking the consequences. Otherwise, you can easily retroactively change your goals to fit whatever action you want.

For example, in the sunk costs PhD scenario, it would be easy for someone to say something like "If I pursue my PhD, it will make my family proud. This is very important to me, so I will pursue my PhD." However if you asked them what their goals are BEFORE they listed the consequences of each action they probably would not list "Make family proud", bu... (read more)

4handoflixue
Conversely, you might not recognize your true rejection until you go "well, but wait, I don't want to abandon my PhD - my family wouldn't be proud of me." If you get stuck on that, it's possible that "make your family proud" really IS important to you. I do agree that it's a useful idea, but new insights aren't necessarily ex post facto rationalizations.

an audience of non-Sequence-reading experimental subjects, who were mostly either programmers or in other technical subjects, so I could go through the math fairly fast

I don't have suggestions on the main question, but I strongly recommend that you design the curricula to be consumable by corporate executives. If you can convince companies that they need to send their execs to week-long rationality seminars, to help them improve their decision-making skills, you are going to make megabucks.

We did the Really Getting Bayes game at the Mountain View meetup this week. My impression of it was that the explanation was at first a little unclear, but that once we had gotten the sense of it, it was worthwhile.

One thing that I realized during the game was the degree to which I was using the availability heuristic to provide my likelihood ratios. For instance, one object I was given was either an electrical extension cord or an audio cable. In coming up with RGB likelihoods, I thought, "Electrical extension cords are usually black or white, hence ... (read more)

3Academian
In case this wasn't done, a physical demonstration of a game like this at first is important, with a concurrent verbal description to tag it for indexing: "Step 1: we do this", "Step 2: we do this." Showing beats telling alone. Verbal or written instructions are a low bandwidth form of communication that are better used for tagging/clarification of demonstration data (i.e. naming the steps while you do them) or error-correcting after a demonstration (i.e. people can look stuff up if they get confused).

Why is teaching people to think like consequentialists a good idea again? Serious question.

If they're (relatively successful) mathematicians and programmers I don't see how it could go wrong but I'm awfully worried about some of the rest of the population. Specifically people not being able to sustain it without letting other things slip.

second edit: I should clarify. It's teaching the habit that I'm thinking about. Everyone should be able to think like a consequentialist but is instilling these reflexes gonna be a net positive?

4daenerys
Devil's Advocating Here: I do think we need to not forget that most people's minds do NOT operate like the typical LWian! I know this personally, in that I tend to make intuitive-based decisions (and by intuitive, I mean things like waking up one morning thinking "I should eat less meat", and so becoming a vegetarian for the next 8 (so far) years.) Decisions I have made intuitively like this include: atheism, vegetarianism, not drinking alcohol (that one only lasted 7 years), quitting grad school, not having children, polyamory, pretty much every career decision, liking rationality. The social situations were different enough for each of these for me to think that social concerns were not the main trigger, but I somehow feel like I've ended up making rational-style choices by following my intuition (and I recognize that I probably just lost all credibility I might have had here by writing this post :P). The upside of this, is that since I am always doing what I feel like, I rarely feel like I am having to fight myself. For example, giving up meat was amazingly easy for me, because it was like the decision had already been made. From someone who really enjoys learning about rationality, I can still see how it wouldn't mesh with many people's methods of living, without a complete lifestyle overhaul (which is an unlikely result of a single class). But I do not AT ALL think that this means that we shouldn't teach people about consequentialism or other rationality topics (I am all about spreading rationality), I just think we need to make sure that we do so in a way that can encompass a wide range of people. First figure out what percentage of the overall population you want to be accessible to, (say the top 60% intellectually, MINUS the 20% most intuitive types) and make sure that your presentations and materials are able to reach whatever your target is.
2JoachimSchipper
Your intuition appears to like LW-approved things, and you are on LW. Don't you think that learning about consequentialism might beneficially rewire one's intuition?
4daenerys
If you are implying that learning about rationality on LW made my intuitions more rational, then you should know that I made all those decisions long before joining LW about 5 months ago. However, I wouldn't be surprised with the fact the LW conforms to most of my intuitions (except the whole anti-deathism and singularity stuff) as one of the reasons I joined the site. I remember thinking "OMGWTF there are people who are OPENLY POLY on here, outside of a poly-specific group!!!" I'm sure it doesn't hurt that I find rationality and psychology to be amazingly interesting.
2JoachimSchipper
Honestly, I'm not sure what I was thinking when I wrote that comment. I'm aware you joined rather recently... Perhaps "intellectual beliefs and intuitions seem to be correlated, which suggests that one can rewire one by tinkering with the other."
3JoachimSchipper
Umm, are you under the impression that (the non-mathematical-ish part of the population/anyone) is constantly operating near their sustainable cognitive maxima? So near that adding a nearly-automatic reflex would push them over? Neither looking around nor introspection suggests that that is true.
7scav
Indeed. That would imply that our shared goal of raising the sanity waterline would cause most of the population to drown :) Mind you, I like that the OP is asking what the consequences would be. However my guess is: more people making slightly better decisions some of the time, and with no obvious mechanism for "letting other things slip", I don't see a downside.
0AspiringKnitter
What if the problem isn't that it's too cognitively taxing, but that, applied in the sloppy way most people apply their heuristics, it could lead to irrational choices or selfish behavior?
1scav
People already make irrational choices. I don't think teaching them one way to mitigate that could make things worse. What's the opposite of status quo bias? I might have some of that, whatever it is :)
0handoflixue
Upvoted because I rather like that phrasing :)
0A1987dM
Well, when teaching non-perfect people about consequentialism you should teach them about ethical injunctions as well. I don't think teaching both will be a net negative.

"You shouldn't kill because it's the wrong thing to do" can be rescued as "Because then a person will transition from 'alive' to 'dead' in the future, and this is a bad event" or "Because the interval between Outcome A and Outcome B includes the interval from Fred alive to Fred dead."

Why the fancy words? This just seems like a complicated way of saying: "Because the person would then be dead. And that is bad".

6MixedNuts
People being dead is a bad outcome. Killing people is a bad action. Consequentialism does not recognize bad actions, only actions that lead to bad outcomes.
5HonoreDB
A human corpse poofing into existence from nowhere wouldn't be in itself a bad outcome. So we need to specify that the human was once alive. An alternate phrasing might be "Because this would cause the person to die." But the word "die" is historically imprecise. Open-heart surgery stops a beating heart. Destructive uploading would cause brain death.
5wedrifid
Free food! (It doesn't count as cannibalism if the corpse has never been a member of your species!)

Does it have kuru? I'm only open to eating healthy human flesh in this scenario.

Also, if it poofs into existence from nowhere, is it creating matter out of nothing? It's creating something that still has usable energy in it, out of nothing? That could not only end world hunger and veganism, you might be able to use the newly-created corpses for fuel in some kind of power plant. Sure, you might have to go back to steam power to make it work, and sure, human bodies might not be the optimal fuel source, but if you're getting them from nowhere, that solves all our energy woes.

It also might make the planet gain mass, eventually, if you did enough of it for long enough. Hmm. Oh, well, you can use that to make spacecraft. Maybe. Or something.

That and blood pudding. And fertilizer.

I think actually, being able to poof human corpses into existence would be an improvement over the current state of affairs. It might still be sub-optimal, but it would be better.

Now I want to be able to poof human corpses into existence from nowhere. I also think maybe I should start a list of things I've said that I wouldn't have been able to predict that I would say if asked the day before.

Less Wrong: Rationality, polyamory, cannibalism.

1TheOtherDave
...though the other order would be more challenging.
1Armok_GoB
Someone need to make an SCP based of this.
0Strange7
There already is, if you're willing to combine two: http://www.scp-wiki.net/scp-871 http://www.scp-wiki.net/scp-604
2TheOtherDave
Nobody said "free." The operational costs of corpse-poofing might be prohibitive.
0Nornagest
Well, there ain't no such thing as a free lunch.
0Eliezer Yudkowsky
Less Wrong: Rationality, polyamory, cannibalism.
3TheOtherDave
"the person would then be dead" seems to pretty clearly imply that there was a person involved. In the case where a corpse poofs into existence from nowhere, there doesn't seem to have ever been a person involved. I conclude that "Because the person would then be dead" doesn't apply to the case where a corpse poofs into existence from nowhere. So I'm not sure why we would need to further specify anything here. All of that said, the whole approach of counting deaths as negative utility seems to me to be rescuing the wrong part of the original nonconsequentialist claim in the first place. It's clear that one consequence of increasing the human population from 1 billion people to 7 billion people is that many more people die per unit time, but it doesn't follow from that fact that we should reject increasing human population on consequentialist grounds. (It might be true that we should so reject it, but even if true it doesn't follow from that fact.) It seems that the part we would want to rescue from a consequentialist POV is the idea that more life-years is good, so any act that reduces expected net lifeyears is bad... and also, perhaps, the idea that more life-years/person is good, so any act that reduces expected net lifeyears/person is bad. This would also render all concerns about how we define "death" moot.
0FGonzalez
You still need to weigh emotional trauma caused by corpse-poofing.
2Eliezer Yudkowsky
Eh, people would get used to it.

To me, this comes down to what I am trying to learn as my anti-akrasia front kick: I cache the question "Why am I doing what I am doing?". While I lose some amount of focus to the question itself, I have gained key insights into many of my worst habits. For instance, my employer provides free soft drinks- I found that I would end up with multiple, open drinks at my desk. The cached question revealed I was using the action of getting a drink whenever I felt the need to stretch and leave my desk. Browsing reddit too much at work- cached question c... (read more)

Many of the pain points listed have a common trait: the decision would seem easier with less information. For example, the PhD decision is easier if you didn't know about the costs which have been sunk, the identity decisions are easier if you're not sure of your own identity, cached thought problems are easier without having that thought cached, etc...

But we know that information should never have negative value. So why not highlight that dissonance? Imagine the following exercise:

Handout: "You spent the last 3 years working toward a PhD. You passed... (read more)

1JoachimSchipper
I'm not sure this works well - last time "I" made a decision, "I" preferred five years of work for a PhD title to a $90k job now. It would seem unlikely that I'd prefer a $90k job now over two years of work for a PhD title, especially given that I'm now more sure that there are good jobs waiting for me.
1johnswentworth
Thanks, Joachim. Like I said, I don't know much about PhD programs. What would be some better numbers to make the point?
0JoachimSchipper
I'm sorry, but I have no idea - I'm in the Netherlands, which has a different academic/economic structure than the US.
0handoflixue
The problem is, that hypothetical doesn't really have any weight, unless you specify that having a PhD will still only produce a job worth $90K, at which point the audience has to wonder why this hypothetical fool started on their degree in the first place. I do like the point about paying to remove information - there's times I would happily have paid to remove information from my awareness, because I was aware it was biasing me in very annoying ways. I think learning to deal with that separately would be very useful, and probably help a lot with Consequentialism (maybe they're even the same issue? my intuition tells me they feel different internally, but I don't have a lot of good examples available right now)
6Bluehawk
The money isn't necessarily the only factor. Don't forget about location, working hours, stress levels, and job satisfaction. I'd take a $70k job that's intrinsically rewarding over a $100k job that "isn't really my type of environment" any day. Of course, I'd have to KNOW that the $70k job was intrinsically rewarding and that the $100k job wouldn't be, but if the hypothetical fool does know this about his PhD job prospects, for example he wants to be an academic and the job offers so far are in unintellectual labor, or in the family business, or in a city he/she would like to avoid settling down in, or involve 50% more hours than the target job of the same wage -- I don't know if that's useful or not, but I'll err on the side of opening my mouth.
4handoflixue
Research suggests that once you have sufficient income to meet your basic needs, that travel time is one of the biggest factors in job satisfaction. I think we tend to focus on income because it's much easier to evaluate the actual pay rate of a job - if you're promised $100K, you can expect 100K. If you're promised 40 hours and no overtime then you'll often find that tested. If you're promised low stress and high job satisfaction, well, good luck suing for breach of contract on that.
6TheOtherDave
I read the first sentence of this comment three times, with increasing incredulity, before my brain finally parsed "travel time" in the correct order. I think perhaps my expectations of LW discourse are being unduly skewed by all the HPMOR discussion.
2Bluehawk
Being promised low stress/high satisfaction and having a rough idea of what kind of work or work environment is (more or less) enjoyable to you are quite different things. A given idea of which work is enjoyable won't be 100% accurate; there are always going to be surprises from both inside the mind and out. But most people have a rough idea what kind of work they prefer to do. That's where the low stress/high satisfaction predictions come from in this scenario. Obviously one can only expect so much "enjoyment" in a work environment (and no "work" is fun and enjoyable 100% of the time), but if one type of work feels worthwhile to a given person, and the other doesn't, even if this is on the basis of inference, then for some people this is going to be a significant factor in how good/bad they feel about passing up those $90k jobs for the PhD program that might now be in question.
0handoflixue
Fair point. I'm fairly young, so most of my social group is still trying to figure out what sort of work environment they want, and how to actually identify it - a lot of entry level jobs outright lie about the work environment ("we value employee feedback, overtime only when necessary" -> "we are going to be doing another 80 hour death march this week because of an arbitrary release deadline").
0pnrjulius
In game theory, there are a number of situations where it is rational to handicap your own rationality: Reduce your number of choices, take away information, etc. Now, in game theory you're competing against someone else, whereas in this case you're only competing against (time-indexed versions of?) yourself; but it could be that the same rules apply. Maybe it really is rational to pay to not know something. Or maybe it's rational for a bounded agent to pay to be counter-biased: Knowing that I have this bias toward sunk costs, make me ignorant of all sunk costs.
5Eliezer Yudkowsky
TDT is intended to eliminate this. A TDT-agent - one that's correctly modeled by the environment, not that some other agent thinks is a CDT-agent - is supposed to never benefit from having any option taken away from it, and will never pay to avoid learning a piece of information.
4jeremysalwen
Er, this is assuming that the information revealed is not intentionally misleading, correct? Because certainly you could give a TDT agent an extra option which would be rational to take on the basis of the information available to the agent, but which would still be rigged to be worse than all other options. Or in other words, the TDT agent can never be aware of such a situation.
2Eliezer Yudkowsky
Amendment accepted.
0handoflixue
Agreed. I think one could assert "Given a perfect decision theory AND a perfect implementation, additional information is never a negative", but it's silly to live as though that were true. If you know your decision theory doesn't handle X information correctly (say, sunken costs) then it's in your best interests to either eliminate the information, or fix the decision theory. Of course, eliminating information seems to be by far the easier option...
2TheOtherDave
If I know the class of errors my decision theory tends to make given the kinds of Xes I most commonly run into, I can also adopt a third option... for want of a better term, I can patch my decision theory. E.g., "Well, I want to finish this project, but I suspect that part of that desire stems from an invalid weighting of sunk costs, so I won't take that desire at face value... I'll apply some kind of rough-and-ready discounting factor to it." This is clearly not as good as actually fixing my decision theory, but isn't as hard either, and is sometimes more practical than eliminating the information.
0handoflixue
Very true. However, "avoid X information, since it biases me" is actually an example of such a patch. Especially if the information doesn't otherwise have any useful value. How often does knowledge of sunk costs actually move you towards ideal action, rather than biasing you away from it?
0TheOtherDave
Sure, avoiding information is an example of patching a decision theory, agreed. So I guess what I'm saying is that "either eliminate the information, or fix the decision theory" is a misleading way to phrase the choice. My real choice is between fixing it and patching it, where eliminating the information is one of several ways to patch it, and not always the best. Making choices about future investments in ignorance of the existing data I have about previous investments and their ROI is probably less ideal than taking those data into consideration and applying some other patch to compensate for sunk-costing.
0handoflixue
I like the idea of phrasing it as "patching vs long-term fixes" :)

If the world were going to end right after I took an action, which action would I choose? (Alt: If everybody saw what choice I was about to make, but then circumstances changed and my decision turned out not to matter, what choice would I want to have made?)

Did answering that question feel the same as answering the actual question? If so, I'm not really thinking about consequences.

2jmmcd
I think you're onto something good here. Given any question, there are probably lots of hypothetical variations, like the world-ending or the exposure to everyone's judgement which you mention, which shouldn't make a difference but do, or should make a difference but don't. Maybe list a few more such circumstances and get the class to decide whether and why the variations make a difference.

So... how would I design an exercise to teach Checking Consequentialism?

Divide the group into pairs. One is the decider, the other is the environment. Let them play some game repeatedly, prisoners dilemma might be appropriate, but maybe it should be a little bit more complex. The algorithm of the environment is predetermined by the teacher and known to both of the players.

The decider tries to maximize utilitiy over the repeated rounds, the environment tries to minimise the winnigs of the decider, by using social interaction between the evaluated game round... (read more)

Maybe the easiest way to teach it is to teach how it applies to others. That is, train people to catch nonconsequential reasoning in arguments that others make, and then hope that they apply that to themselves. The easiest way to do that is by reflexively asking, "so what?"

0jmmcd
Nice. Many people are much better at criticising others than finding the same flaws in themselves.

Here's a long-form exercise:

  1. Break up into small groups (2-5 people)
  2. Someone in each group, picked at random, presents an upcoming problem or decision.
    The other participant(s) ask questions to clarify the problem/decision.
    The problem/decision can be either real or imaginary, but if imaginary the presenter must come up with appropriately detailed answers to questions.
  3. Everyone collaborates in generating a list of possible solutions. Five is plenty.
  4. Everyone privately notes their preferred solution.
  5. Everyone collaborates on a list of expected
... (read more)

Your check consequentialism sounds a lot like risk management. Risk is the effect of uncertainty on objectives (ISO 31 000). The risk management process involves indentifying risks, analysing how significant they are, and then treating the big ones so that they don't prevent you from attaining your objective. This is fairly straightforward to do. The difficult part is building a risk management culture where the risks are considered before making a decison, embarking on a project, etc. Just identifying the risks is often the big deal. Once you are aware that a risk exists you will probably deal with it. Sorry that I have not given you an activity, but perhaps I have given you a useful keyword to help your search.

By the way, if you don't mind my asking, once you've come up with your rationality curriculum what do you plan to do with it? Are you making inroads with whoever you would need to talk to to get this in a school curriculum, for instance?

6shokwave
I think they plan on running workshops or seminars, likely targeted at startup founders or business/consultant-type people handling large decisions (both from a capability-to-pay and a convinced-of-the-value point of view, this makes far more sense than school curriculums).
3John_Maxwell
What is the closest existing thing to this? How can we make friends with someone who is good at it? Are there any books about them doing it that we could read? Brainstorm: * Sales seminars * Continuing education programs for businesspeople, e.g. evening MBAs * Getting things done seminars, for organizations and the public * Lean thinking, kaizen seminars * Six Sigma seminars * http://www.richdadeducation.com/ (kind of scammy) * The Landmark Forum (even scammier) * http://danariely.com/speaking/ * http://personalmba.com/ (note that ways to pay Josh other than buying his book are not immediately apparent, and that he is a Less Wrong user). * http://thinkingthingsdone.com/ (also a less wrong user) * Tony Robbins, firewalking, etc. I would guess that large organizations are more willing to pay for live instruction than startup founders are. On the other hand, you wouldn't be able to suggest that they do anything that wasn't in the best interest of their employer like quit their job. If organizational seminars are going to be a goal, it might not be a bad idea to start talking to relevant organizational folks to make sure you're making a product they actually want to buy. Jane Street could be an ideal first client, since they've got prestige you can use to sell other clients, EY has a pre-existing relationship with them, and they seem genuinely interested in improving their rationality. On the other hand, their rationality may be at a level where they don't think they could benefit from this sort of workshop, or targeting the workshop at them would mean developing a different set of materials. (But these "advanced" materials might appeal to clients who had already purchased and enjoyed the "basic" materials.) The standard way to do this sort of B2B sale is to graduate to more and more important clients, since a lot of businesses will not buy a novel product unless some other businesses bought it and were happy with it. That's why getting and pleasing the
0Eliezer Yudkowsky
Initially. School curriculum would be harder to develop, so the plan is for that to happen later.