LESSWRONG
LW

Studies and Statistics

Aug 24, 2017 by Scott Alexander


Aquinas famously said: beware the man of one book. I would add: beware the man of one study.

For example, take medical research. Suppose a certain drug is weakly effective against a certain disease. After a few years, a bunch of different research groups have gotten their hands on it and done all sorts of different studies. In the best case scenario the average study will find the true result – that it’s weakly effective.

But there are also about 5 studies that find that the drug is very good, and 5 studies missing the sign entirely and finding that the drug is actively bad. There’s even 1 study finding that the drug is very bad, maybe seriously dangerous.

71Beware The Man Of One Study
Scott Alexander
11y
2
51Debunked And Well-Refuted
Scott Alexander
11y
4
48Noisy Poll Results And Reptilian Muslim Climatologists from Mars
Scott Alexander
12y
1
42Two Dark Side Statistics Papers
Scott Alexander
12y
4
53The Control Group Is Out Of Control
Scott Alexander
11y
5
68The Cowpox of Doubt
Scott Alexander
11y
2
24How Common Are Science Failures?
Scott Alexander
11y
9
31Learning To Love Scientific Consensus
Scott Alexander
8y
1
Interlude
59My IRB Nightmare
Scott Alexander
8y
1
38The Study of Anglophysics
Scott Alexander
11y
1