Low replicability can support robust and efficient science.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
17 01 2020
17 01 2020
Historique:
received:
04
03
2019
accepted:
17
12
2019
entrez:
19
1
2020
pubmed:
19
1
2020
medline:
19
1
2020
Statut:
epublish
Résumé
There is a broad agreement that psychology is facing a replication crisis. Even some seemingly well-established findings have failed to replicate. Numerous causes of the crisis have been identified, such as underpowered studies, publication bias, imprecise theories, and inadequate statistical procedures. The replication crisis is real, but it is less clear how it should be resolved. Here we examine potential solutions by modeling a scientific community under various different replication regimes. In one regime, all findings are replicated before publication to guard against subsequent replication failures. In an alternative regime, individual studies are published and are replicated after publication, but only if they attract the community's interest. We find that the publication of potentially non-replicable studies minimizes cost and maximizes efficiency of knowledge gain for the scientific community under a variety of assumptions. Provided it is properly managed, our findings suggest that low replicability can support robust and efficient science.
Identifiants
pubmed: 31953411
doi: 10.1038/s41467-019-14203-0
pii: 10.1038/s41467-019-14203-0
pmc: PMC6969070
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
358Commentaires et corrections
Type : ErratumIn
Références
Zwaan, R. A., Etz, A., Lucas, R. E. & Donnellan, M. B. Making replication mainstream. Behav. Brain Sci. 41, E120 (2017).
doi: 10.1017/S0140525X17001972
Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, 1–8 (2015).
Camerer, C. F. et al. Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018).
doi: 10.1038/s41562-018-0399-z
Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl Acad. Sci. USA 112, 15343–15347 (2015).
doi: 10.1073/pnas.1516179112
Morey, R. D. et al. The peer reviewers’ openness initiative: incentivizing open research practices through peer review. R. Soc. Open Sci. 2, 15047 (2015).
Stroebe, W. & Strack, F. The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9, 59–71 (2014).
doi: 10.1177/1745691613514450
Kunert, R. Internal conceptual replications do not increase independent replication success. Psychon. Bull. Rev. 23, 1631–1638 (2016).
doi: 10.3758/s13423-016-1030-9
Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).
doi: 10.1177/0956797611417632
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).
doi: 10.1038/nrn3475
Wagenmakers, E.-J. A practical solution to the pervasive problems of p values. Psychon. Bull. Rev. 14, 779–804 (2007).
doi: 10.3758/BF03194105
Jennison, C. & Turnbull, B. W. Statistical approaches to interim monitoring of medical trials: a review and commentary. Stat. Sci. 5, 299–317 (1990).
doi: 10.1214/ss/1177012099
Kerr, N. L. HARKing: Hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196–217 (1998).
doi: 10.1207/s15327957pspr0203_4
Ferguson, C. J. & Heene, M. A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null. Perspect. Psychol. Sci. 7, 555–561 (2012).
doi: 10.1177/1745691612459059
Ferguson, C. J. & Brannick, M. T. Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17, 120–128 (2012).
doi: 10.1037/a0024445
Wagenmakers, E.-J., Wetzels, R., Borsboom, D. & Van Der Maas, H. L. J. Why psychologists must change the way they analyze their data: the case of Psi: comment on Bem (2011). J. Pers. Soc. Psychol. 100, 426–432 (2011).
doi: 10.1037/a0022790
Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T. The preregistration revolution. Proc. Natl Acad. Sci. USA 115, 2600–2606 (2018).
doi: 10.1073/pnas.1708274114
Benjamin, D. J. et al. Redefine statistical significance. Nat. Hum. Behav. 2, 6–10 (2018).
doi: 10.1038/s41562-017-0189-z
Miller, J. & Ulrich, R. The quest for an optimal alpha. PLoS ONE 14, e0208631 (2019).
doi: 10.1371/journal.pone.0208631
Oberauer, K. & Lewandowsky, S. Addressing the theory crisis in psychology. Psychon. Bull. Rev. 26, 1596–1618 (2019).
doi: 10.3758/s13423-019-01645-2
van Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B. & Wicherts, J. M. Why publishing everything is more effective than selective publishing of statistically significant results. PLoS ONE 9, e84896 (2014).
doi: 10.1371/journal.pone.0084896
Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).
doi: 10.1371/journal.pmed.0020124
Topolinski, S. & Sparenberg, P. Turning the hands of time: clockwise movements increase preference for novelty. Soc. Psychol. Pers. Sci. 3, 308–314 (2012).
doi: 10.1177/1948550611419266
Wagenmakers, E.-J. et al. Turning the hands of time again: a purely confirmatory replication study and a bayesian analysis. Front. Psychol. 6, 494 (2015).
doi: 10.3389/fpsyg.2015.00494
Wilson, B. M. & Wixted, J. T. The prior odds of testing a true effect in cognitive and social psychology. Adv. Methods Pract. Psychol. Sci. 1, 186–197 (2018).
doi: 10.1177/2515245918767122
Miller, J. What is the probability of replicating a statistically significant effect? Psychon. Bull. Rev. 16, 617–640 (2009).
doi: 10.3758/PBR.16.4.617
Lewandowsky, S., Brown, G. D. A., Wright, T. & Nimmo, L. M. Timeless memory: evidence against temporal distinctiveness models of short-term memory for serial order. J. Mem. Lang. 54, 20–38 (2006).
doi: 10.1016/j.jml.2005.08.004
Gl�nzel, W., Schlemmer, B. & Thijs, B. Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon. Scientometrics 58, 571–586 (2003).
doi: 10.1023/B:SCIE.0000006881.30700.ea
Rouder, J. N., Speckman, P. I., Sun, D. & Morey, R. D. Bayesian t tests for accepting and rejecting the null hypothesis. Psychon. Bull. Rev. 16, 225–237 (2009).
doi: 10.3758/PBR.16.2.225
Eyre-Walker, A. & Stoletzki, N. The assessment of science: the relative merits of post-publication review, the impact factor, and the number of citations. PLoS Biol. 11, e1001675 (2013).
doi: 10.1371/journal.pbio.1001675
Ioannidis, J. P. A. et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383, 166–175 (2014).
doi: 10.1016/S0140-6736(13)62227-8
Coles, N. A., Tiokhin, L., Scheel, A. M., Isager, P. M. & Lakens, D. The costs and benefits of replication studies. Behav. Brain Sci. 41, e124 (2018).
doi: 10.1017/S0140525X18000596
Field, S. M., Hoekstra, R., Bringmann, L. F. and van Ravenzwaaij, D. When and why to replicate: as easy as 1, 2, 3? Collabra: Psychology 5, (2019).
Miller, J. & Ulrich, R. Optimizing research payoff. Perspect. Psychol. Sci. 11, 664–691 (2016).
doi: 10.1177/1745691616649170
Chalmers, I. & Glasziou, P. Avoidable waste in the production and reporting of research evidence. Lancet 374, 86–89 (2009).
doi: 10.1016/S0140-6736(09)60329-9
Ioannidis, J. P. A. How to make more published research true. PLoS Med. 11, e1001747 (2014).
doi: 10.1371/journal.pmed.1001747
Baribault, B. et al. Metastudies for robust tests of theory. Proc. Natl Acad. Sci. USA 115, 2607–2612 (2018).
doi: 10.1073/pnas.1708285114
Francis, G. The psychology of replication and replication in psychology. Perspect. Psychol. Sci. 7, 585–594 (2012).
doi: 10.1177/1745691612459520
Francis, G. Too good to be true: publication bias in two prominent studies from experimental psychology. Psychon. Bull. Rev. 19, 151–156 (2012).
doi: 10.3758/s13423-012-0227-9
Publons global state of peer review 2018. Tech. Rep. (Clarivate Analytics, 2018). https://doi.org/10.14322/publons.gspr2018 .
Greitemeyer, T. Article retracted, but the message lives on. Psychon. Bull. Rev. 21, 557–561 (2014).
doi: 10.3758/s13423-013-0500-6
Arslan, R. Revised: Are studies that replicate cited more? https://rubenarslan.github.io/posts/2019-01-02-are-studies-that-replicate-cited-more/ (2019).
Mayr, S., Erdfelder, E., Buchner, A. & Faul, F. A short tutorial of GPower. Tutor. Quant. Methods Psychol. 3, 51–59 (2007).
doi: 10.20982/tqmp.03.2.p051