For those who haven't read it, take a look at Richard Feynman on cargo cult science if you want a good lecture on experimental design.

I loved it. I have a question for anyone who might know: In that 1974 speech, Richard Feyman made a very specific criticism of experimental psychology. He mentioned an "a-number-one experiment" on lab rats running through a maze by a "Mr. Young" in 1937, which corrected for a hugely non-intuitive experimental design error. But then, according to Feynman:

The next experiment, and the one after that,

... (Read more)(Click to expand thread. ⌘F to Expand All)Cmd/Ctrl F to expand all comments on this post

One interesting lead showed up on Twitter: Marvin Minsky on Usenet 10 April 1993 (sci.bio "Puling Habits out of Rats") in response to someone asking 'who was Mr Young and whatever happened with the mouse studies':

What happened around 1937 was that

  • [possibility #] 5. B. F. Skinner developed ways to control all those external variables by enclosing the experiment in a sealed, soundproof, lightproof, etc., box. The results were reliably reproducible, and a great deal was learned. The boxes were soon names "Skinner Boxes" and became t
... (Read more)(Click to expand thread. ⌘F to Expand All)Cmd/Ctrl F to expand all comments on this post
3Tesseract9y Extensive searching does not turn up a single result for such a person or study. The only result [http://answers.yahoo.com/question/index?qid=20090409202031AAEjQcV] I can find that seems in the right vein is on a different aspect of rat behavior. What turned up more than anything else were quotations of the essay itself. Apparently there is some doubt [http://www.quora.com/In-Feynmans-autobiography-Surely-Youre-Joking-Mr-Feynman!-does-the-scientist-Dr-Young-who-is-described-in-the-cargo-cult-science-section-actually-exist] as to whether he actually exists.

The Decline Effect and the Scientific Method [link]

by Dreaded_Anomaly 9y31st Dec 201024 comments

12


The Decline Effect and the Scientific Method (article @ the New Yorker)

First, as a physicist, I do have to point out that this article concerns mainly softer sciences, e.g. psychology, medicine, etc.

A summary of explanations for this effect:

  • "The most likely explanation for the decline is an obvious one: regression to the mean. As the experiment is repeated, that is, an early statistical fluke gets cancelled out."
  • "Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found."
  • "Richard Palmer... suspects that an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. ... Palmer emphasizes that selective reporting is not the same as scientific fraud. Rather, the problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results."
  • "According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. ... The current “obsession” with replicability distracts from the real problem, which is faulty design."

These problems are with the proper usage of the scientific method, not the principle of the method itself. Certainly, it's important to address them. I think the reason they appear so often in the softer sciences is that biological entities are enormously complex, and so higher-level ideas that make large generalizations are more susceptible to random error and statistical anomalies, as well as personal bias, conscious and unconscious.

For those who haven't read it, take a look at Richard Feynman on cargo cult science if you want a good lecture on experimental design.

12