Science Isn't Science If It Isn't Reproducible
by Kevin A. Roth, Audra E. Cox
This Editorial introduces readers to the US National Institutes of Health's Proposed Principles and Guidelines for Reporting Preclinical Research.
The unity of all science consists alone in its method, not in its material.
Karl Pearson, The Grammar of Science1
Much has been written about the problem of reproducibility of the scientific literature.2, 3 The significance of the problem is underscored by several high-profile retractions in 2014 alone.4 Critics of peer review may remark that the current system is broken and that it often fails to discern flaws in research methods before publication. Proponents claim that the current process improves the validity of published material and that science is self-correcting, with errors eventually discovered through repetition. However, regardless of your stance, peer review quality depends on the knowledge base of reviewers involved, and it has become increasingly clear that the statistical knowledge base of the average reviewer may be insufficient. Journals are thus left with the daunting task of improving review processes to ensure that published results meet high standards for repetition and accuracy in reporting.
Recognizing the need to increase the reproducibility and rigor of preclinical scientific research, the US National Institutes of Health sponsored a joint workshop with the Nature Publishing Group and the American Association for the Advancement of Science in June 2014 and invited approximately 30 journal editors to discuss best practices, opportunities, and new ideas for supporting transparent, rigorous, and robust preclinical research publications. The American Journal of Pathology (AJP), as the most highly cited journal in experimental pathology research and a frequent publisher of NIH-funded basic/preclinical research, was included in the group of participants and given an opportunity to engage in the framing of new guidelines for preclinical research. Following extensive discussions at the workshop and post-workshop revisions, we at AJP are pleased to be one of the inaugural signatories of a set of Proposed Principles and Guidelines for Reporting Preclinical Research (https://www.nih.gov/about/reporting-preclinical-research.htm, last accessed November 24, 2014).
The Proposed Principles and Guidelines address a number of issues including statistical analysis, standards for transparent and rigorous reporting of study design, data and material sharing, and establishment of best practice guidelines.5 Many of the proposed principles and guidelines are already in place at AJP, such as no page limits on the Materials and Methods section, inclusion of complete details of reagents used, and requirements for data and material sharing. Some of the recommendations, such as inclusion of a clear policy on statistical analysis, require updates to the Instructions for Authors. We welcome the opportunity to establish a set of instructions that will assist authors and reviewers in assessing the scientific rigor of manuscript submissions, thereby facilitating the interpretation and replication of methods and results published in AJP.
Please review the revised Instructions (https://ajp.amjpathol.org/content/authorinfo) for information on some of the changes that are already underway.
It is important to state that the Proposed Principles and Guidelines will not be applied indiscriminately or without consideration of the scientific nature and unique aspects of individual manuscripts. The guidelines are not meant to be a burden to authors or a discouragement to submitting manuscripts to AJP. However, as general examples of the types of information that high quality manuscripts are expected to contain, adherence to the stated guidelines is an excellent starting point for potential authors to consider when preparing their manuscripts for submission to AJP. Authors are also encouraged to contact the Editorial Office if they have concerns or questions regarding manuscript preparation.
In summary, endorsement of the Proposed Principles and Guidelines for Reporting Preclinical Research will further ensure the quality and reproducibility of manuscripts published in AJP. It also aids the Journal in fulfilling its commitment of publishing accurate and exciting scientific advances in understanding the molecular and cellular mechanisms of disease pathogenesis.
References
- Karl Pearson. The Grammar of Science. ed 2. Adam and Charles Black, London; 1900:12.
- Collins, F.S., Tabak, L.A. Policy: NIH Plans to Enhance Reproducibility. Nature. 2014;505:612–613.
CrossRef | PubMed | Scopus (54) - Nuzzo, R. Scientific method: statistical errors. Nature. 2014;506:150–152.
CrossRef PubMed | Scopus (42) - STAP retracted. Nature. 2014;511:5–6.
- McNutt, M. Journals Unite for Reproducibility. Science. 2014;346:679.
CrossRef | PubMed | Scopus (4)
[Note: Whether one is a scientist or not, everyone can appreciate and care about the fact that scientific fraud and misconduct in pre-clinical trial studies is of great concern because unfortunately such false data is then used as the basis of therapeutic research clinical trials involving vulnerable human patients -- thus precluding them from giving ethically and legally valid informed consent, as well as exposing them to damaging even deadly side-effects. So if nothing else this article, along with others that it references, acknowledge such connections, and are a clear indication that purely experimental bench or pre-clinical science has been in disarray in many respects for some time now. And so certain players aim to clean it up to prevent further health hazards to innocent patients in clinical trials. Indeed, as I''ve stressed for a long time, science ISN'T science if it isn't reproducible (replication/reproducibility being the last required step in the traditional genuine Scientific Method). Thus one would think that in no way is a study based merely on 'Proof of Principle' to be considered 'science' or its data necessarily true or accurate. Yet nowhere in the various articles referenced below is that odd pseudo-standard called 'Proof of Principle' addressed. Nor does there seem to be any strict or uniform interpretation of these new voluntary guidelines or their definitions -- especially with respect to defining or enforcing the reproducibility of data. Guess time will tell. See also 'Journals unite for reproducibility' (November 2014), Science 7, Vol. 346, No. 6210, p. 679; DOI: 10.1126/science.aaa1724, at: https://www.sciencemag.org/content/346/6210/679.long See also 'Journals unite for reproducibility' (Nov. 6, 2014), Nature 515, 7; doi:10.1038/515007a, at: https://www.nature.com/news/journals-unite-for-reproducibility-1.16259 See also 'Principles and Guidelines in Reporting Preclinical Research' (Dec. 2, 2014), National Institutes of Health (NIH), at: https://www.nih.gov/about/reporting-preclinical-research.htm. The article first appeared here -- DNI]