I'm really confused by this. Both examples sound like exactly what you should be doing.

If the Tibetan study was to determine how many people died in childbirth, and they refused to publish because not enough people were dying to support their preferred narrative, that would be bad. But the way you wrote it sounds like it was to determine differences in deaths across two conditions, but there weren't enough deaths to tell. In that case, stopping the study instead of publishing results that are probably false is the best possible course of action.

Likewise, doing studies in isolated villages is a good way to control for confounding variables. If you wanted to measure effectiveness of a cancer drug, you do it on people who are only taking the one drug, not people taking lots of drugs that might differ across groups. If you're measuring life expectancy in AIDS, you do it on patients who only have AIDS, not patients who have AIDS and also cancer. If a physicist is measuring the radioactivity of U-235, ey makes sure to use a purified sample of U-235, not a sample mixed with U-238. I don't even see what this has to do with double-blinding (which is about placebos and making sure subjects don't know what group they're in). This is just a basic attempt to isolate the experimental variable. I agree that given infinite time and resources they should try follow-up studies in less isolated villages to see if the results carry over, but in reality this seems like the best way to do things.

Am I misunderstanding the book's points?

I suspect that the Tibet study issue could be solved with better statistical methodology (i.e. using a Bayesian framework and including semi-realistic prior information). However, setting that aside, I agree completely.

Some problems with evidence-based aid

by NancyLebovitz 1 min read23rd Dec 20108 comments


In Against Health: How Health Became the New Morality (Biopolitics, Medicine, Technoscience, and Health in the 21st Century) , there's an essay "Against Global Health?" by Vincanne Adams.

The introduction was a bit of a slog, but I think the point was that if "health" is defined as something which can and should be given to people regardless of what they think in the matter, the results can be presumptuous.

While she may be overly invested in the way things are usually done, she brings up some disquieting points about the ways insisting on double-blind tests can go wrong. For example, she mentions a project to evaluate training in safe infant delivery techniques in Tibet which was scuttled because not enough women were dying to get a good power calculation.

She describes experiments which are done in isolated communities because that's where the researchers can be reasonably sure that the subjects don't have alternative sources of care which would foul up the double-blinding. This does seem like a drunk and lamp post problem. [1] More generally, the concern is that actual care is delayed in the search for perfect information. Admittedly, it's hard to be sure about how to balance searching for information with taking action, but it strikes me as a problem that's worth some thought rather than just assuming that double-blinding is a reliable improvement.

[1] Is there a standard LW term for searching where it's easy to search rather than where the answer is likely to be?