"Everything is Correlated": An Anthology of the Psychology Debate

by gwern1 min read27th Apr 20192 comments

40

PsychologyAcademic PapersWorld Modeling
Frontpage
This is a linkpost for https://www.gwern.net/Everything
2 comments, sorted by Highlighting new comments since Today at 7:21 PM
New Comment

(Sort of a very long delayed followup to https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst , tracking down one specific strand of the debate.)

Does anyone know if private or corporate research has abandoned NHST and adopted better alternatives?

My suspicion is that the answer is no, because even companies that depend heavily on the correctness of their R&D output (like 3M or Dow Chemical, or marketing measurements at Amazon and Facebook) mostly have the problem that people chosen to do research are chosen by people who don't know about things like p-hacking. Even if they are diligent and careful business people, I expect a modest epistemology to conclude "do what Nature does" and stick with the current methods.

On the other hand, it seems like skipping the analysis part completely by tightly coupling research outputs to products/practices is a more common strategy, particularly for people like Amazon and Facebook where it can be very rapidly iterated. Yet it really seems like there is an opportunity to consistently win beyond what the market is doing here.

This is causing me to wonder how exactly do go about betting on an institution that placed its bets on better analysis. I feel like in order to be sure these methods are really being put to work, a new institution will have to be built for the purpose. It also seems like some careful consideration needs to be given to which market the institution will operate in, because it should be somewhere the edge provided by better analysis has the greatest impact.

One step beyond this: is inducing mimicry of good practices via effective competition something that has been analyzed in an EA context?