Tuesday, April 9, 2013

Irreproducible Research Isn't Always Useless

In Nature, Jonathan Russell argues that something needs to be done about reproducibility of papers:
In recent years, it has become clear that biomedical science is plagued by findings that cannot be reproduced. This wastes grant money and hinders development of new treatments and cures. 
Yes, but the true answer depends on why the study irreproducible.  Is it because the authors have been dishonest with their data or is it because they were truly trying to publish something that's completely novel?
If a paper cannot be replicated, the authors should be required to amend or retract it. Funding agencies would then consider a principal investigator’s history of reproducibility in grant reviews. 
Funding agencies should already consider whether the author has had too many errata published, and on what basis each of them came to be.  There are many examples of errata published for honest mistakes like mislabeling figures, author name corrections, etc.  Fundamentally, this level of added scrutiny has to come from the funding agencies, perhaps by providing better incentives for qualified reviewers to, well, review.

I don't necessarily agree that these papers should be outright retracted (unless the paper is fraudulent), because there's always some good message to take home, even if it something like "don't ever use these experimental conditions because you will end up like this team of researchers".
If publishing amendments were explicitly rewarded, people would take the time to do it.
But the point of doing good research is to get it right the first time.  The explicit reward already comes in the form of publishing something that confirms or refutes the first study.  Plus, the team that publishes the 'amendment' (really, just another paper) get credit for doing so, in the eyes of granting agencies.

On principle, I agree with Russell that improvements are needed to identify irreproducible work, but more on a level of catching dishonest folks (which Retraction Watch sort of does, after the fact) instead of bulking up the scientific system with more administrative controls.