Cross posted from Overcoming Bias. Comments there.
***
I asked how it could be that experimenting in my own life could be worthwhile, given that if such things were worthwhile other people should have already figured them all out. My suggested explanations:
- I am strange: nerdy, recent, young
- Innovation: there are constantly new things to experiment with
- Findings are not spread: or so much noise is also spread that the information is lost
- Context-specificity: your findings don’t apply to me, because people are unique or situations are complicated
- I am wrong: it’s easy to underestimate nebulous costs, to overstate fleeting or illusory benefits, to want to be the kind of person who tries new things, or to be too hopeful that life can improve fast
It seems to me that 3 is the biggest: results are collected so badly as to be often worthless and are aggregated poorly. It’s not clear to what extent this is because of 4: other people’s findings are just not so useful. Personal experimentation seems worth it even without good aggregation, but probably only if you avoid the same errors of measurement yourself. It could be worth it even with purely placebo gains, if you enjoy the placebo gains enough. But in this scenario, the gains are much smaller than you imagine, so you are probably over-investing a lot. There also seems to me a real risk that everything is so context specific that what you learn will be worthless as soon as you change many other things (4).
Explanations that involve others finding experimentation a lot less worthwhile (e.g. 1) seem unlikely to help much because it looks like others often find experimentation worthwhile. The problem seems to be somewhere between others making such efforts, and me having useful information as a result. Innovation (2) seems a bad explanation because it doesn’t explain the lack of information about age-old lifestyle questions. It seems likely that I have overestimated gains relative to losses in the past (5), but gains still seem larger than losses (it’s hard to disentangle causes, but my lifestyle has obviously improved substantially over the last year or more, some of which seems directly attributable to purposeful experimentation and the rest of which seems at least not terribly damaged by it).
