Hypothesis: (formed before running the test below) "clearly too high" will be more of an overestimate than "clearly too low" is an underestimate; most midpoints will be high; this will bias estimates high (especially through repeated self-anchoring to high midpoints).
Anecdata: I tried using this method to estimate the number of gates in Hong Kong International Airport. (I'm currently standing in a line in HKIA, and have flown through it at least a dozen times in my life.)
So, depending on how I decide to stop, I get estimates around 195 or 351. Apparently the true answer is 90, supporting my hypothesis.
Proposal: Maybe use geometric mean instead of arithmetic mean?
I tried to clear my head and try again with geomean:
There's one suspect step around 159, but my guess here is that being anchored by one higher number (632) rather than five (5020, 2530, 1285, 662, 351) is enough to actually make me think "probably less than 159" rather than "no strong opinion about 195". Terminating at 159 also is an outperformance of arithmetic mean, but feels a bit more like luck around where the midpoints fell.
I wonder if running the algorithm would affect the performance/results of those who haven't been anchored.
Anchoring is a classic cognitive bias which has been discussed on Less Wrong before. Anchoring seems very difficult to avoid. Experiments have found that warning subjects about anchoring, or giving them cash incentives, doesn't solve the problem.
Here's an algorithm to fight anchoring that I would like to see a researcher test, based on binary search:
You could have two experimental conditions: one condition where subjects think of a number which is clearly too high first (the steps are in the order above), and another condition where subjects think of a number which is clearly too low first (steps 1 & 2 are swapped). If estimates from the two conditions are similar, the technique is successful.