It depends on what you mean by "didn't work". The study described is published in a paper only 16 pages long. We can just read it: http://web.mit.edu/curhan/www/docs/Articles/biases/67_J_Personality_and_Social_Psychology_366,_1994.pdf
First, consider the question of, "are these predictions totally useless?" This is an important question because I stand by my claim that the answer of "never" is actually totally useless due to how trivial it is.
Despite the optimistic bias, respondents' best estimates were by no means devoid of information: The predicted completion times were highly correlated with actual completion times (r = .77, p < .001). Compared with others in the sample, respondents who predicted that they would take more time to finish actually did take more time. Predictions can be informative even in the presence of a marked prediction bias.
...
Respondents' optimistic and pessimistic predictions were both strongly correlated with their actual completion times (rs = .73 and .72, respectively; ps < .01).
Yep. Matches my experience.
We know that only 11% of students met their optimistic targets, and only 30% of students met their "best guess" targets. What about the pessimistic target? It turns out, 50% of the students did finish by that target. That's not just a quirk, because it's actually related to the distribution itself.
However, the distribution of difference scores from the best-guess predictions were markedly skewed, with a long tail on the optimistic side of zero, a cluster of scores within 5 or 10 days of zero, and virtually no scores on the pessimistic side of zero. In contrast, the differences from the worst-case predictions were noticeably more symmetric around zero, with the number of markedly pessimistic predictions balancing the number of extremely
optimistic predictions.
In other words, asking people for a best guess or an optimistic prediction results in a biased prediction that is almost always earlier than a real delivery date. On the other hand, while the pessimistic question is not more accurate (it has the same absolute error margins), it is unbiased. The reality is that the study says that people asked for a pessimistic question were equally likely to over-estimate their deadline as they were to under-estimate it. If you don't think a question that gives you a distribution centered on the right answer is useful, I'm not sure what to tell you.
The paper actually did a number of experiments. That was just the first.
In the third experiment, the study tried to understand what people are thinking about when estimating.
Proportionally more responses concerned future scenarios (M = .74) than relevant past experiences (M =.07), r(66) = 13.80, p < .001. Furthermore, a much higher proportion of subjects' thoughts involved planning for a project and imagining its likely progress (M =.71) rather than considering potential impediments (M = .03), r(66) = 18.03, p < .001.
This seems relevant considering that the idea of premortems or "worst case" questioning is to elicit impediments, and the project managers / engineering leads doing that questioning are intending to hear about impediments and will continue their questioning until they've been satisfied that the group is actually discussing that.
In the fourth experiment, the study tries to understand why it is that people don't think about their past experiences. They discovered that just prompting people to consider past experiences was insufficient, they actually needed additional prompting to make their past experience "relevant" to their current task.
Subsequent comparisons revealed that subjects in the recall-relevant condition predicted they would finish the assignment later than subjects in either the recall condition, t(79) = 1.99, p < .05, or the control condition, f(80) = 2.14, p < .04, which did not differ significantly from each other, t(& 1) < 1
...
Further analyses were performed on the difference between subjects' predicted and actual completion times. Subjects underestimated their completion times significantly in the control (M = -1.3 days), r(40) = 3.03, p < .01, and recall conditions (M = -1.0 day), t(41) = 2.10, p < .05, but not in the recall-relevant condition (M = -0.1 days), ((39) < i. Moreover, a higher percentage of subjects finished the assignments in the predicted time in the recall-relevant condition (60.0%) than in the recall and control conditions (38.1% and 29.3%, respectively), x2G, N = 123) = 7.63, p < .01. The latter two conditions did not differ significantly from each other.
...
The absence of an effect in the recall condition is rather remarkable. In this condition, subjects first described their past performance with projects similar to the computer assignment and acknowledged that they typically finish only 1 day before
deadlines. Following a suggestion to "keep in mind previous experiences with assignments," they then predicted when they would finish the computer assignment. Despite this seemingly powerful manipulation, subjects continued to make overly optimistic forecasts. Apparently, subjects were able to acknowledge their past experiences but disassociate those episodes from their present predictions.
In contrast, the impact of the recall-relevant procedure was sufficiently robust to eliminate the optimistic bias in both deadline conditions
How does this compare to the first experiment?
Interestingly, although the completion estimates were less biased in the recall-relevant condition than in the other conditions, they were not more strongly correlated with actual completion times, nor was the absolute prediction error any smaller. The optimistic bias was eliminated in the recall-relevant condition because subjects' predictions were as likely to be too long as they were to be too short. The effects of this manipulation mirror those obtained with the instruction to provide pessimistic predictions in the first study: When students predicted the completion date for their honor's thesis on the assumption that "everything went as poorly as it possibly could" they produced unbiased but no more accurate predictions than when they made their "best guesses."
It's common in engineering to perform group estimates. Does the study look at that? Yep, the fifth and last experiment asks individuals to estimate the performance of others.
As hypothesized, observers seemed more attuned to the actors' base rates than did the actors themselves. Observers spontaneously used the past as a basis for predicting actors' task completion times and produced estimates that were later than both the actors' estimates and their completion times.
So observers are more pessimistic. Actually, observers are so pessimistic that you have to average it with the optimistic estimates to get an unbiased estimate.
One of the most consistent findings throughout our investigation was that manipulations that reduced the directional (optimistic) bias in completion estimates were ineffective in in-
creasing absolute accuracy. This implies that our manipulations did not give subjects any greater insight into the particular predictions they were making, nor did they cause all subjects to become more pessimistic (see Footnote 2), but instead caused enough subjects to become overly pessimistic to counterbalance the subjects who remained overly optimistic. It remains for future research to identify those factors that lead people to make
more accurate, as well as unbiased, predictions. In the real world, absolute accuracy is sometimes not as important as (a) the proportion of times that the task is completed by the "best-guess" date and (b) the proportion of dramatically optimistic, and therefore memorable, prediction failures. By both of these criteria, factors that decrease the optimistic bias "improve" the quality of intuitive prediction.
At the end of the day, there are certain things that are known about scheduling / prediction.
gesture vaguely to all of the projects that overrun their schedule. There are many interesting reasons for why that happens and why I don't think it's a massive failure of rationality, but I'm not sure this comment is a good place to go into detail on that. The quick answer is that comical overrun of a schedule has less to do with an inability to create correct schedules from an engineering / evidence-based perspective, and much more to do with a bureaucratic or organizational refusal to accept an evidence-based schedule when a totally false but politically palatable "optimistic" schedule is preferred.
Epistemic status: Firm
The concepts underlying the Inner Simulator model and the related practical technique of Murphyjitsu are well-known and well-researched, including Kahneman’s S1/S2, mental simulation, and mental contrasting. Similarly, the problems that this unit seeks to address (such as optimism bias and the planning fallacy) have been studied in detail. There is some academic support for specific substeps of Murphyjitsu (e.g. prospective hindsight), and strong anecdotal support (but no formal research) for the overall technique, which was developed through iterated experimentation and critical feedback. See the Further Resources section for more discussion.
Claim: there is a part of your brain (not a specific physical organ, but a metaphorical one) which is keeping track of everything.
It's not actually recording all of the information in a permanent fashion. Rather, what it's doing is building up a consistent, coherent model of how things work. It watches objects fall, and builds up anticipations: heavy objects drop straight down, lighter ones flutter, sometimes wind moves things in chaotic ways. It absorbs all of the social interactions you observe, and assembles a library of tropes and clichés and standard patterns.
Whenever you observe something that doesn't match with your previous experience, you experience (some amount of) surprise and confusion, which eventually resolves in some kind of update. You were surprised to see your partner fly off the handle about X, and now you have a sense that people can be sensitive about X. You were surprised by how heavy the tungsten cube was, and now you have a sense that objects can sometimes be really, really dense.
You can think of this aggregated sense of how things work as a simulated inner world—a tiny, broad-strokes sketch of the universe that you carry around inside your head. When you move to catch a falling pen, or notice that your friend is upset just by the way they entered the room, you're using this inner simulator. It's a different sort of processing from the explicit/verbal stuff we usually call "thinking," and it results in a very different kind of output.
Inner Simulator:
Explicit/Verbal models
We don't have to think about how to catch a falling pen, or send explicit instructions to our body about how to move to do it, because our inner simulator "knows" how falling objects move, and it "knows" how to make our hand go to a particular place. Similarly, it knows what facial expressions mean, what it’s like to drive from home to work, and what sorts of things tend to go wrong given a set of circumstances. It’s a powerful tool, and learning how to access it and when to trust it is one of the first steps to becoming a whole-brain thinker.
That’s not to say that your inner simulator is superior to your explicit model maker—each has both strengths and weaknesses, and can be either the right tool or the wrong one, depending on the situation. In any given moment, you’re probably receiving feedback from both of these “advisors,” as well as other sources of information like your friends or the internet.
In a sense, it’s your job to balance the competing recommendations from all of these different advisors to arrive at the best possible decision. Your inner sim, for example, provides feedback extremely quickly and is good at any type of task where you have lots of experience to draw on, but tends to fall prey to framing effects and will sometimes sneakily substitute an easy question for a harder one. Your explicit verbal models, on the other hand, are great for abstractions and comparisons, but are slow and vulnerable to wishful thinking and ideological distortions.
You can think of your inner sim as a black box that’s capable of performing a few specific functions, given certain input. It’s very, very good at doing those functions, and not so great with most other things (for instance, inner sim is terrible at understanding large numbers, and causes us to donate the same amount of money if someone asks us to help save 8,000 hypothetical birds from oil spills as if they ask us to save 800,000). But if you need a particular kind of reality check, it helps to know which parts of reality inner sim sees most clearly.
Making good use of your inner simulator
Like most algorithms, your inner simulator will output good and useful information if you give it good and useful input, and it will output useless garbage if that’s what you feed it. It’s an especially good check on wishful thinking and motivated cognition—just imagine its response to a list of New Year’s resolutions—but you need to make sure that you aren’t rigging the game by phrasing questions the wrong way, and you need to make sure that you're asking questions within a domain where your inner sim actually has representative data.
(Because your inner sim will often answer confidently either way, even if the question is outside of your relevant experience! For instance, many people erroneously slam on the brakes the first time they start to skid on ice, even though this is counterproductive—that's because the inner sim "knows" that slamming on the brakes will stop the car, and doesn't know that this does not apply in this novel situation.)
Two useful strategies for avoiding vague, open-ended “garbage” are sticking to concrete examples and looking for next actions.
Asking for Examples
In a situation like this, your inner sim has nothing to grab onto—everything is vague, everything is open to interpretation, and clichés and stereotypes are filling in for actual understanding. It could be that your friend is in the right, and needs your commiseration; it could also be that the situation calls for some harsh truths and tough love. How can you tell, one way or another? Try some of these:
When it’s just “every little thing turns into a fight,” your inner sim literally doesn’t know what to think—there are too many possibilities.
(Another way to think of this is that both it-being-your-fault and it-being-their-fault are consistent with your inner sim's general sense of how-things-work; they're both plausible and your inner sim can't help you distinguish between them.)
But when the argument started with “Do we really have to go over to Frank’s again?” or with “Oh, hey, I see you got new shoes. Nice!” you have a much better clearer sense of what the situation really looks like. The detail helps you narrow down which swath of past experiences you want to draw your intuitions from.
Asking for examples is a handy technique for any conversation. When you keep your inner simulator engaged, and keep feeding it data, you might notice that it’s easier to:
Searching for Next Actions
A goal isn’t the same thing as a plan. I might have the goal of exercising more, but if I’m going to make that goal a reality—especially given that I’m not currently exercising as much as I “should”—then I’ll need to think about when and how I’ll get to the gym, what I’ll do when I get there, what the realistic obstacles are going to be, and how I’m going to hold the plan together moment by moment and month by month.
But even before I get to those things, I’ll need to take my next action, which might be printing out a gym coupon, or setting a reminder in my calendar, or looking at my schedule for a good time to buy workout clothes. A next action is a step that sets your plan in motion—it’s both the first thing you’d have to do to build momentum, and also the first roadblock, if left undone. Usually it’s not particularly exciting or dramatic—next actions are often as mundane as putting something on the calendar or looking something up online. If you’re not able to take your next action at the moment you think of it, it’s generally helpful to think of a trigger—some specific event or time that will remind you to follow through.
For instance, if I have a goal of applying to a particular school next fall, then my plan will likely involve things like updating my CV, looking at the application process online, checking my finances, creating a list of plausible contacts for letters-of-recommendation, making decisions about work and relationships, and a host of other things. If, after thinking it through, I decide that my next action is to spend half an hour on the school’s website, then I need a solid trigger to cause me to remember that at the end of a long day and a long commute, when I get home tired and hungry and Netflixy. That trigger might be a phone alarm, or an email reminder in my inbox, or a specific connection to something in my evening routine—when I hear the squeak of my bedroom chair, I’ll remember to go online—but whatever trigger I choose, I’ll be better off having one than not, and better off with a concrete, specific one than a vague, forgettable one.
(For more detail on the process of choosing triggers and actions, see Trigger-Action Planning.)
Some specific inner sim functions
What happens next?
Start a “mental movie” by concretely visualizing a situation, and see what your brain expects to happen. If this is the beginning of the scene, how does the scene end?
How shocked am I?
Check your “surprise-o-meter”—visualize a scenario from start to finish, and see whether you “buy” that things would actually play out that way. Common outputs are “seems right/shrug,” “surprised,” and “shocked,” and you may benefit from building a habit of rating your surprise on a scale of one to ten (so that you start to learn to feel the difference between three-out-of-ten surprised and five-out-of-ten surprised, and can detect a shift from e.g. seven down to six).
What went right/wrong?
Use your “pre-hindsight”—start by assuming that your current plan has utterly failed. What explanation leaps to mind about why this happened? What's the "obvious" answer, according to all of your past experience, and your aggregated sense of how things work?
The Murphyjitsu Technique
Murphy’s Law states “Anything that can go wrong, will go wrong.” Even worse, even people who are familiar with Murphy's Law are notoriously bad at applying it when making plans and predictions—in a classic experiment, 37 psychology students were asked to estimate how long it would take them to finish their senior theses “if everything went as poorly as it possibly could,” and they still underestimated the time it would take, as a group (the average prediction was 48.6 days, and the average actual completion time was 55.5 days).
However, where straightforward introspection fails, a deliberate use of inner sim can provide a valuable “second opinion.” Below are the steps for Murphyjitsu, a process for bulletproofing your strategies and plans.
Irresponsibly cavalier psychosocial speculation: It seems plausible that the reason this works for many people (where simply asking “what could go wrong?” fails) is that, in our evolutionary history, there was a strong selection pressure in favor of individuals with a robust excuse-generating mechanism. When you’re standing in front of the chief, and he’s looming over you with a stone axe and demanding that you explain yourself, you’re much more likely to survive if your brain is good at constructing a believable narrative in which it’s not your fault.
Inner Simulator—Further Resources
Kahneman and Tversky (1982) proposed that people often use a simulation heuristic to make judgments. Mental simulation of a scenario is used to make predictions by imagining a situation and then running the simulation to see what happens next, and it is also to give explanations for events by mentally changing prior events and seeing if the outcomes changes.
Kahneman, D. & Tversky, A. (1982). The simulation heuristic. In D. Kahneman, P. Slovic, & A. Tversky (eds.) Judgment under uncertainty: Heuristics and biases (pp. 201-208).
Research on mental simulation has found that imagining future or hypothetical events draws on much of the same neural circuitry that is used in memory. The ease with which a simulated scenario is generated often seems to be used as a cue to the likelihood of that scenario. For a review, see:
Szpunar, K.K. (2010). Episodic future thought: An emerging concept. Per- spectives on Psychological Science, 5, 142-162. http://goo.gl/g0NNI
“Mental contrasting,” sometimes referred to as gain-pain movies, is a specific algorithm for making optimism and drive more accurate and robust in the face of adversity. In her book Rethinking Positive Thinking, Dr. Gabrielle Oettingen outlines the steps of mental contrasting, along with the underlying justification and examples of results.
Oettingen, Gabrielle (2014). Rethinking Positive Thinking.
“Focusing” is a practice of introspection systematized by psychotherapist Eu- gene Gendlin which seeks to build a pathway of communication and feedback between a person’s “felt sense” of what is going on (an internal awareness which is often difficult to articulate) and their verbal explanations. It can be understood as a method of querying one’s inner simulator (and related parts of System 1). Gendlin’s (1982) book Focusing provides a guide to this technique, which can be used either individually or with others (in therapy or other debugging conversations).
Gendlin, Eugene (1982). Focusing. Second edition, Bantam Books. http://en.wikipedia.org/wiki/Focusing
Focusing, for skeptics
The idea of identifying the concrete “next action” for any plan was popularized by David Allen in his book Getting Things Done.
Allen, David (2001). Getting Things Done: The Art of Stress-Free Productivity. http://en.wikipedia.org/wiki/Getting_Things_Done
Mitchell, Russo, and Pennington (1989) developed the technique which they called “prospective hindsight.” They found that people who imagined themselves in a future world where an outcome had already occurred were able to think of more plausible paths by which it could occur, compared with people who merely considered the outcome as something that might occur. Decision making researcher Gary Klein has used this technique when consulting with organizations to run “premortems” on projects under consideration: assume that the project has already happened and failed; why did it fail? Klein’s (2007) two-page article provides a useful summary of this technique, and his (2004) book The Power of Intuition includes several case studies.
Mitchell, D., Russo, J., & Pennington, N. (1989). Back to the future: Tem- poral perspective in the explanation of events. Journal of Behavioral Decision Making, 2, 25-38. http://goo.gl/GYW6hg
Klein, G. (2007). Performing a project premortem. Harvard Business Re- view, 85, 18-19. http://hbr.org/2007/09/performing-a-project-premortem/ar/1
Klein, Gary (2004).The Power of Intuition: How to Use Your Gut Feelings to Make Better Decisions at Work.