Promoting Reproducibility and Integrity in Observational Research: One Approach of an Epidemiology Research Community.
Journal
Epidemiology (Cambridge, Mass.)
ISSN: 1531-5487
Titre abrégé: Epidemiology
Pays: United States
ID NLM: 9009644
Informations de publication
Date de publication:
01 05 2023
01 05 2023
Historique:
medline:
6
4
2023
pubmed:
1
2
2023
entrez:
31
1
2023
Statut:
ppublish
Résumé
To increase research reproducibility, sharing of study data, analysis code, and use of standardized reporting are increasingly advocated. However, beyond reproducibility, few initiatives have addressed the integrity of how research is conducted before manuscripts are submitted. We describe a decades-long experience with a comprehensive approach based in an academic research community around prospective cohort studies that is aimed at promoting a culture of integrity in observational research. The approach includes prespecifying hypotheses and analysis plans, which are discussed in the research community and posted; presentation and discussion of analysis results; mandatory analysis code review by a programmer; review of concordance between analysis output and manuscripts by a technical reviewer; and checks of adherence to the process, including compliance with institutional review board requirements and reporting stipulations by the National Institutes of Health. The technical core is based in shared computing and analytic environments with long-term archiving. More than simply a list of rules, our approach promotes research integrity through integrated educational elements, making it part of the "hidden curriculum," by fostering a sense of belonging, and by providing efficiency gains to the research community. Unlike reproducibility checklists, such long-term investments into research integrity require substantial and sustained funding for research personnel and computing infrastructure. Our experiences suggest avenues for how institutions, research communities, and funders involved in observational research can strengthen integrity within the research process.
Identifiants
pubmed: 36719725
doi: 10.1097/EDE.0000000000001599
pii: 00001648-202305000-00011
pmc: PMC10073307
mid: NIHMS1869215
doi:
Types de publication
Journal Article
Research Support, N.I.H., Extramural
Langues
eng
Sous-ensembles de citation
IM
Pagination
389-395Subventions
Organisme : NCI NIH HHS
ID : U01 CA167552
Pays : United States
Organisme : NCI NIH HHS
ID : U01 CA176726
Pays : United States
Organisme : NHLBI NIH HHS
ID : U01 HL145386
Pays : United States
Organisme : NCI NIH HHS
ID : UM1 CA186107
Pays : United States
Informations de copyright
Copyright © 2023 The Author(s). Published by Wolters Kluwer Health, Inc.
Déclaration de conflit d'intérêts
The authors report no conflicts of interest.
Références
Peng RD, Dominici F, Zeger SL. Reproducible epidemiologic research. Am J Epidemiol. 2006;163:783–789.
National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. The National Academies Press; 2019.
Goodman SN, Fanelli D, Ioannidis JP. What does research reproducibility mean? Sci Transl Med. 2016;8:341ps12.
Begg CB, Berlin JA. Publication bias and dissemination of clinical research. J Natl Cancer Inst. 1989;81:107–115.
Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014;505:612–613.
Franzoni C, Scellato G, Stephan P. Science policy. Changing incentives to publish. Science. 2011;333:702–703.
O’Boyle EH, Banks GC, Gonzalez-Mulé E. The Chrysalis Effect: how ugly initial results metamorphosize into beautiful articles. J Manag. 2016;43:376–399.
Munafò MR, Nosek BA, Bishop DVM, et al. A manifesto for reproducible science. Nat Hum Behav. 2017;1:0021.
Fowler J, San Lucas FA, Scheet P. System for quality-assured data analysis: flexible, reproducible scientific workflows. Genet Epidemiol. 2019;43:227–237.
Goecks J, Nekrutenko A, Taylor J, Galaxy T. Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol. 2010;11:R86.
Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9:e1003285e1003285.
Colditz GA. Constraints on data sharing: experience from the nurses’ health study. Epidemiology. 2009;20:169–171.
Poole C. A vision of accessible epidemiology. Epidemiology. 2010;21:616–618.
Assel M, Vickers AJ. Statistical code for clinical research papers in a high-impact specialist medical journal. Ann Intern Med. 2018;168:832–833.
Goldstein ND. Toward open-source epidemiology. Epidemiology. 2018;29:161–164.
Goldstein ND, Hamra GB, Harper S. Are descriptions of methods alone sufficient for study reproducibility? An example from the cardiovascular literature. Epidemiology. 2020;31:184–188.
Goldacre B, Morton CE, DeVito NJ. Why researchers should share their analytic code. BMJ. 2019;367:l6365.
Vandenbroucke JP, von Elm E, Altman DG, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med. 2007;4:e297e297.
Chambers C. The registered reports revolution lessons in cultural reform. Signif. 2019;16:23–27.
Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible research: moving toward research the public can really trust. Ann Intern Med. 2007;146:450–453.
Nosek BA, Alter G, Banks GC, et al. SCIENTIFIC STANDARDS. Promoting an open research culture. Science. 2015;348:1422–1425.
Landis SC, Amara SG, Asadullah K, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012;490:187–191.
National Institutes of Health. Research Methods Resources, 2019. Available at: https://researchmethodsresources.nih.gov/ . Accessed 14 September 2019.
Research integrity is much more than misconduct. Nature. 2019;570:5.
Vable AM, Diehl SF, Glymour MM. Code review as a simple trick to enhance reproducibility, accelerate learning, and improve the quality of your team’s research. Am J Epidemiol. 2021;190:2172–2177.
Platt RW. Code Review: an important step towards reproducible research. Am J Epidemiol. 2021;190:2178–2179.
Leek JT, Peng RD. Opinion: reproducible research can still be wrong: adopting a prevention approach. Proc Natl Acad Sci U S A. 2015;112:1645–1646.
Lash TL. The harm done to reproducibility by the culture of null hypothesis significance testing. Am J Epidemiol. 2017;186:627–635.
Lash TL, Collin LJ, Van Dyke ME. The replication crisis in epidemiology: snowball, snow job, or winter solstice? Curr Epidemiol Rep. 2018;5:175–183.
Cole P. The hypothesis generating machine. Epidemiology. 1993;4:271–273.
The Editors of Epidemiology. The registration of observational studies--when metaphors go bad. Epidemiology. 2010;21:607–609.
Lash TL, Vandenbroucke JP. Should preregistration of epidemiologic study protocols become compulsory? Reflections and a counterproposal. Epidemiology. 2012;23:184–188.
The Editors of Epidemiology. On the death of a manuscript. Epidemiology. 2002;13:495–496.
Boccia S, Rothman KJ, Panic N, et al. Registration practices for observational studies on ClinicalTrials.gov indicated low adherence. J Clin Epidemiol. 2016;70:176–182.
U.S. Food and Drug Administration. Framework for FDA’s Real-word Evidence Program, 2018. Available at: https://www.fda.gov/media/120060/download . Accessed 20 March 2022.
Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 2017;26:1033–1039.
Wang SV, Schneeweiss S, Berger ML, et al. Reporting to improve reproducibility and facilitate validity assessment for Healthcare Database Studies V1.0. Pharmacoepidemiol Drug Saf. 2017;26:1018–1032.