Cross posted from Overcoming Bias. Comments there.

***

I’ve been wondering why experimentation seems worthwhile. I’ve given some explanations in my last few posts. The last category of explanations to consider are the ones in which my judgement is wrong. Where experimentation seems worth it because I don’t see the costs, or because I overestimate the benefits.

A lot of the cost from experimentation is plausibly nebulous and hard to account for well in a simple explicit analysis. Life can’t run smoothly on habits when they are always in flux. Mental effort is used up in keeping track. Every new thing takes a little while to do well, and to integrate into your lifestyle.

It’s easy count the costs of living in Oxford for a month in airfares and flight times, and forget the freezing afternoon you might spend negotiating to reclaim your foreign credit card from an ATM that ate it. Or the hassle of urgently buying boots, or of running up the high street looking for a working internet connection to finish your Skype call, or the sleep loss due to alien fire alarm policies at the college where you are staying. It seems to me that I have tended to underestimate such costs in the past substantially.

I mentioned in earlier posts some reasons I might overestimate the benefits. Innovation is less worth finding if it is quickly obsoleted by context specificity or further innovation. Informal data collection seems to see benefits too easily in every change. Nerds may underestimate the wisdom embodied in tradition. The first of these seems unlikely, given my experience. The others seem dangerous, but I do guard against them.

One might also overestimate the benefits if one is motivated to do so. Being willing to try new things is a telling sign about a person, or often taken to be one. Often it is a sign you should want to send. This hypothesis is supported in my case by the observation that many people I know seem to find experimentation particularly useful. However I rarely talk about this kind of stuff, and feel a bit silly when I do. Which doesn’t fit a signaling hypothesis well. Though this blog sequence undermines my claims some.

One might also be biased by other motivations. For instance if you badly hope that life can get much better, it might be hard to accept a route to that which involves sitting by and waiting when there are so many ways to aggressively search. I admit I would probably have some trouble accepting that this is as good as it gets, but I think I would at least be aware of discomfort around the topic if this was what was going on. So this seems unlikely to account for the observation.

To me underestimating the hidden costs seems by far the biggest danger.