Contrarian viewpoint: I think this isn't as bad as the article makes it sound.
It makes sense to pay more attention to positive results. That's where the breakthroughs are. That's the reason humans evolved positive bias in the first place.
It makes sense to focus more on new research than on replication The end of science is precise mechanisms explaining how stuff works - if someone throws out a wrong result, we'll figure out something is wrong via conceptual replication, and maybe then an exact replication might be in order. Exact replication is expensive and benefits are likely lower than the equivalent resources into new research. (Of course, this does not apply when resources are not a limiting factor)
The real-world costs of being wrong in science are low - there are costs, but these costs only increase if you add in replication and stuff. The real world benefits of being right in science are tremendous. The signal-to-noise ratio is somewhat less important than the absolute amount of signal in this scenario.
Not saying that there isn't room any for improvement, but let's carefully think about what we want to reform. I don't think replication is necessarily the answer, and if we are going to pay more attention to null results we have to be smart about how we do it and not err in the opposite direction - as in, you shouldn't ignore nulls but it really is better to pay more attention to positive results.
(Unless you are doing extremely practical research, as in medicine. Then all of this is really really bad. I'm talking about strictly about research which is still several steps away from a practical implementation)
I'm vaguely annoyed with the way they explain statistical significance issues in genomics in this paper. It makes biologists sound worse than they are. They do control the family wise error rate when the 'family' in question is just the individual genes in an experiment. It's when the family in question consists of multiple experiments, or papers, that the trouble starts.
Wow I had no idea that would be taken so negatively. Anyone want to clear up the inferential silence here?
I'd like to retract it, but I can't retract it simply on the basis that it is very unpopular if I myself do not see the problem with it. :(
I didn't downvote initially, though I'm doing so now. If other people's reactions are anything like mine, it's a combination of finding the comment incoherent, reacting in isolation to boo-lights embedded in the comment, and having lost patience with the author to the point of being uninterested in asking for explanation or granting benefit of the doubt.
Thank you, that clarifies things considerably.
Reading around, I can't seem to decipher what you mean to say with, "boo-lights embedded in the comment." Can you clarify that for me?
"boo-lights" => phrases meant to evoke an emotional rejection-response. I would give examples, but the comment itself seems to have been completely rewritten.
I saved a copy in case you wanted to do that, if you're interested.
Thank you. I believe that clarifies everything that could be clarified here.
If you're not clear about which phrases I'm referring to , I'm willing to point them out. If you are, then there seems little to add.
Incidentally, editing a comment that's already downvoted below -3 is unlikely to achieve much (except for confusing the record), since basically nobody will read the edited comment.
In the same way (and for the same reason), simply retracting or deleting the comment serves little purpose. I see fit to attempt to clarify. The worst that can happen is that it will still yet attract downvotes. I have little to lose that I haven't already lost in failing to notice I had a comment receiving such negativity. At the very least, I am now aware that it is happening.
Yup, agreed that retracting/deleting the comment is similarly unlikely to achieve much.
Certainly, none of that activity is likely to clarify anything, since few people read comments after they've been downvoted below threshold. If anything, it's more likely confuse/obscure the record, as I mentioned.
Related: The Real End of Science
From the Economist.
“I SEE a train wreck looming,” warned Daniel Kahneman, an eminent psychologist, in an open letter last year. The premonition concerned research on a phenomenon known as “priming”. Priming studies suggest that decisions can be influenced by apparently irrelevant actions or events that took place just before the cusp of choice. They have been a boom area in psychology over the past decade, and some of their insights have already made it out of the lab and into the toolkits of policy wonks keen on “nudging” the populace.
Dr Kahneman and a growing number of his colleagues fear that a lot of this priming research is poorly founded. Over the past few years various researchers have made systematic attempts to replicate some of the more widely cited priming experiments. Many of these replications have failed. In April, for instance, a paper in PLoS ONE, a journal, reported that nine separate experiments had not managed to reproduce the results of a famous study from 1998 purporting to show that thinking about a professor before taking an intelligence test leads to a higher score than imagining a football hooligan.
The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed (as the replicators claim) or the replications are (as many of the original researchers on priming contend). Either way, something is awry.
...
I recommend reading the whole thing.