I'm too lazy to do a better analysis now, but just to provide the barest of intuitions:

Let's say a study with trillions of participants has shown that using Strategy A works better than not using Strategy A 80% of the time. I'm about to decide whether or not to use Strategy A, and unfortunately I don't know about the study. I poll three of my friends who have all done rigorous self-experiments. (Or maybe I've done three rigorous self-experiments myself.) All it takes is a pocket calculator to show that I have a 90% chance of correctly guessing whether I should use Strategy A: .2^3 + 3 (.8.2*.2) = .104. And obviously if I poll myself, based on a single past rigorous self-experiment, I'll have an 80% chance of getting it right.

(A better analysis would probably use the normal approximation for the binomial distribution, so we could see results for all sorts of parameters, but that would be a pain to write out with my voice recognition system.)

I suspect that scientific evidence is most useful on questions that are hard to decide (e.g. if Strategy A works 51% of the time; incidentally this sort of knowledge is also the most useless), or in cases where your degree of belief matters beyond just choosing whether or not to use a strategy (seems kind of rare).

This last point about degree of belief not mattering much could explain why Bayesian statistics didn't catch on as well as frequentist statistics initially: most of the time, your exact degree of belief doesn't matter and you just need to decide whether or not to do something.

Something about your rough model disagrees with me (in addition to the stuff in gwern's comment). Tentatively I'd put my finger on strategies like your hypothetical strategy A being rarer than they look. I think it's uncommon for a prospective lifestyle change to simultaneously

  • have a much better chance than 50% of being worth implementing...
  • ...yet not obviously be a good idea a priori
  • be something you're not already doing
  • be easy for you and/or friends to test/implement
  • be non-obvious enough that published research on it doesn't already exist

(Edited to add "be" to bullet point 2.)

5gwern8yYou're making a massive assumption: that self-experimentation is not biased worse than regular clinical trials by things like selection effects. This is what I mean by methodological concerns making each self-experiment far far less than n=1. I mean, look at OP - from the sound of it, the friend did not report their results anywhere (perhaps because they were null?). Bingo, publication effect. People don't want to discuss null effects, they want to discuss positive results. I've seen this first-hand with dual n-back, among others, where I had trouble eliciting the null results even though they existed. Given this sort of bias and zero effort on self-experimenters' part to counter it, yes, you absolutely could do far worse than random by sampling 1000 self-experimenters compared to 1000 clinical trial participants! This is especially true for highly variable stuff like sleep, where you can spot any trend you like in all the noise - compare the dramatic confident anecdotes collected by Seth Roberts about vitamin D at night based on purely subjective retrospective recall of <10 nights to my actual relatively moderate findings based on 40 nights of Zeo data. (I actually have a little demonstration that someone is engaging in considerable confirmation bias, but I'm not done yet. I should be able to post the result in early May.)

Knowledge value = knowledge quality × domain importance

by John_Maxwell 1 min read16th Apr 201241 comments


Months ago, my roommate and I were discussing someone who had tried to replicate Seth Roberts' butter mind self-experiment. My roommate seemed to be making almost no inference from the person's self-reports, because they weren't part of a scientific study.

But knowledge does not come in two grades, "scientific" and "useless". Anecdotes do count as evidence, they are just weak evidence. And well designed scientific studies constitute stronger evidence then poorly designed studies. There's a continuum for knowledge quality.

Knowing that humans are biased should make us take their stories and ad hoc inferences less seriously, but not discard them altogether.

There exists some domains where most of our knowledge is fairly low-quality. But that doesn't mean they're not worth study, if the value of information in the domain is high.

For example, a friend of mine read a bunch of books on negotiation and says this is the best one. Flipping through my copy, it looks like the author is mostly just enumerating his own thoughts, stories, and theories. So one might be tempted to discard the book entirely because it isn't very scientific.

But that would be a mistake. If a smart person thinks about something for a while and comes to a conclusion, that's decent-quality evidence that the conclusion is correct. (If you disagree with me on this point, why do you think about things?)

And the value of information in the domain of negotiation can be very high: If you're a professional, being able to negotiate your salary better can net you hundreds of thousands over the course of a career. (Anchoring means your salary next year will probably just be an incremental raise from your salary last year, so starting salary is very important.)

Similarly, this self-help book is about as dopey and unscientific as they come. But doing one of the exercises from it years ago destroyed a large insecurity of mine that I was only peripherally aware of. So I probably got more out of it in instrumental terms than I would've gotten out of a chemistry textbook.

In general, self-improvement seems like a domain of really high importance that's unfortunately flooded with low-quality knowledge. If you invest two hours implementing some self-improvement scheme and find yourself operating 10% more effectively, you'll double your investment in just a week, assuming a 40 hour work week. (ALERT: this seems like a really important point! I'd write an entire post about it, but I'm not sure what else there is to say.)

Here are some free self-improvement resources where the knowledge quality seems at least middling: For people who feel like failuresFor students. For mathematiciansProductivity and general ass kicking (web implementation for that last idea). Even more ass kicking ideas that you might have seen already.