[ Question ]

Is it true that most studies in Kahneman's 'Thinking fast and slow' are not replicable and the book is largely not supported by evidence?

by Davy Jones1 min read30th Jun 202110 comments

16

PsychologyHeuristics & BiasesProbability & StatisticsRationality
Frontpage

So I stumbled on this article.

https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow

'So, replicability [of all studies in this book] is somewhere between 12% and 46%. Even if half of the results are replicable, we do not know which results are replicable and which ones are not.'

'Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologist, rather than an objective summary of scientific evidence.'

I have no background in social sciences or statistics so I don't know if claims and math in this article are correct. Could somebody with more knowledge comment on this? This is HUGE if true.

I love this book to pieces but I don't want to go around spreading outdated science.. are there any similar books on human biases and thinking which are more recent and more robust when it comes to evidence and statistics?

New Answer
Ask Related Question
New Comment

5 Answers

https://www.researchgate.net/publication/335312555_Not_lost_in_translation_Successfully_replicating_Prospect_Theory_in_19_countries

Kahneman is rare example of a scientist critical to his own ideas,trying to falsify them rather than p-hack his way to fame. I recommend M. Lewis's book about his and Tversky's work.

My impression of The Undoing Project (Michael Lewis's book that you recommend) was that it absolutely and completely ignored the replication crisis. I was hoping for more and ended up super disappointed. It was a fun bit of biography but I don't think it has any bearing on Davy's question.

Thanks, this is helpful and in line with that article.

I read the book post-replication crisis, but had to put it down because it seemed like half its content was the type that I wouldn't expect to pass replication. I still think that the message I've gotten from this community around:

  • The difference between intuitive vs explicit/analytical reasoning
  • There are knowable, common and correctable biases in intuitive reasoning

have stood the test of time, at least in principle, albeit with less of a central role in my worldview.

I don't know how well the studies on which is based replicate, but 'Predictably irrational' from Dan Ariely is also in the Behavioural Economics field, and Dan seems to put a lot of effort into preparing valid experiments.

This reply has aged very fast...

Try books by Gerd Gigerenzer (search on Amazon). I particularly like his Empire of Chance, about different schools of probability analysis.  Bounded Rationality is pretty good, too. The difference between Gigerenzer's and Tversky's attitudes is not particularly impressive (they apparently hate each other, even though one is dead). One is decision making when in a hurry (Gigerenzer) and the other is decision-making with insufficient data (Tversky) (or maybe it's the other way around, tedious stuff). Discussions of biases is fascinating by both authors.

I agree with jp's comment in the sense that the true value of Thinking Fast & Slow may not be the specific examples but the cautionary message it gives about relying too much on our decisions and overcoming the "homo-economicus

However, I'm also interested now to see the replicability of those studies quoted by the book.."

1 comments, sorted by Highlighting new comments since Today at 4:05 AM

This is a good question!

One quibble – "not replicable" seems like a much stronger criticism than "not replicated" – I imagine you might have meant the latter, i.e. attempted replications failed to find similarly 'significant' effects and not 'the effect(s), if real, can not be replicated in principle'.

(And now I'm wondering whether "not replicable" is common and I've never noticed it before.)

Some thoughts about your question itself:

replicability [of all studies in this book] is somewhere between 12% and 46%.

That's a pretty wide range!

It would probably also, significantly, matter as to which specific studies were not be replicated (and whether the attempted replications themselves were well done) and then which parts of the book depend on those studies, or cite them as evidence. Ideally, one would want to consider something like the entire 'graph' of claims/beliefs made or argued for in the book and both the specific and overall updates one would reasonably make based on the (non-)replication of the cited/referenced studies. That would be an interesting, tho costly (if only in time), project itself!