Quantifying convergence and consistency.

consistency convergence cumulative evidence index evidence meta‐analysis

Journal

The European journal of neuroscience
ISSN: 1460-9568
Titre abrégé: Eur J Neurosci
Pays: France
ID NLM: 8918110

Informations de publication

Date de publication:
15 Oct 2024
Historique:
revised: 21 08 2024
received: 01 01 2024
accepted: 19 09 2024
medline: 15 10 2024
pubmed: 15 10 2024
entrez: 15 10 2024
Statut: aheadofprint

Résumé

The reproducibility crisis highlights several unresolved issues in science, including the need to develop measures that gauge both the consistency and convergence of data sets. While existing meta-analytic methods quantify the consistency of evidence, they do not quantify its convergence: the extent to which different types of empirical methods have provided evidence to support a hypothesis. To address this gap in meta-analysis, we and colleagues developed a summary metric-the cumulative evidence index (CEI)-which uses Bayesian statistics to quantify the degree of both consistency and convergence of evidence regarding causal hypotheses between two phenomena. Here, we outline the CEI's underlying model, which quantifies the extent to which studies of four types-positive intervention, negative intervention, positive non-intervention and negative non-intervention-lend credence to any of three types of causal relations: excitatory, inhibitory or no-connection. Along with p-values and other measures, the CEI can provide a more holistic perspective on a set of evidence by quantitatively expressing epistemic principles that scientists regularly employ qualitatively. The CEI can thus address the reproducibility crisis by formally demonstrating how convergent evidence across multiple study types can yield progress toward scientific consensus, even when an individual type of study fails to yield reproducible results.

Identifiants

DOI: 10.1111/ejn.16561 PMID: 39403790
pubmed: 39403790
doi: 10.1111/ejn.16561
doi:

Types de publication

Journal Article

Langues

eng

Sous-ensembles de citation

IM

Subventions

Organisme : Leslie Chair in Pioneering Brain Research
Organisme : NIH HHS
ID : T32EB016640-02
Pays : United States
Organisme : NCATS NIH HHS
ID : UL1TR000124
Pays : United States
Organisme : NCI NIH HHS
ID : T32CA201160
Pays : United States

Informations de copyright

© 2024 The Author(s). European Journal of Neuroscience published by Federation of European Neuroscience Societies and John Wiley & Sons Ltd.

Références

Bovens, L., & Hartmann, S. (2003). Bayesian epistemology. Oxford University Press. https://doi.org/10.1093/0199269750.001.0001
Claveau, F. (2013). The independence condition in the variety‐of‐evidence thesis. Philosophy of Science, 80, 94–118. https://doi.org/10.1086/668877
Claveau, F., & Grenier, O. (2019). The variety‐of‐evidence thesis: A Bayesian exploration of its surprising failures. Synthese, 196(8), 3001–3028. https://doi.org/10.1007/s11229-017-1607-5
Danks, D., & Plis, S. (2019). Amalgamating evidence of dynamics. Synthese, 196(8), 3213–3230. https://doi.org/10.1007/s11229-017-1568-8
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018). Meta‐analysis and the science of research synthesis. Nature, 555, 175–182. https://doi.org/10.1038/nature25753
Leek, J., McShane, B. B., Gelman, A., Colquhoun, D., Nuijten, M. B., & Goodman, S. N. (2017). Five ways to fix statistics. Nature, 551, 557–559. https://doi.org/10.1038/d41586-017-07522-z
Magliacane, S., Claassen, T., & Mooij, J. M. (2016). Ancestral causal inference. Advances in Neural Information Processing Systems, 29, 399–401.
Matiasz, N. J., Wood, J., Doshi, P., Speier, W., Beckemeyer, B., Wang, W., Hsu, W., & Silva, A. J. (2018). ResearchMaps.org for integrating and planning research. PLoS ONE, 13(5), e0195271. https://doi.org/10.1371/journal.pone.0195271
Munafò, M. R., & Smith, G. D. (2018). Robust research needs many lines of evidence. Nature, 553, 399–401. https://doi.org/10.1038/d41586-018-01023-3
Osimani, B., & Landes, J. (2023). Varieties of error and varieties of evidence in scientific inference. The British Journal for the Philosophy of Science, 74(1), 117–170. https://doi.org/10.1086/714803
Shiffrin, R. M., Börner, K., & Stigler, S. M. (2018). Scientific progress despite irreproducibility: A seeming paradox. Proceedings of the National Academy of Sciences, 115(11), 2632–2639. https://doi.org/10.1073/pnas.1711786114
Silva, A. J., Landreth, A., & Bickle, J. (2013). Engineering the next revolution in neuroscience: The new science of experiment planning. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199731756.001.0001
Yang, Y., Macleod, M., Pan, J., Lagisz, M., & Nakagawa, S. (2023). Advanced methods and implementations for the meta‐analyses of animal models: Current practices and future recommendations. Neuroscience & Biobehavioral Reviews, 146, 105016. https://doi.org/10.1016/j.neubiorev.2022.105016

Auteurs

Nicholas J Matiasz (NJ)

Department of Neurobiology, David Geffen School of Medicine at UCLA, University of California Los Angeles, Los Angeles, California, USA.
Medical Imaging Informatics, Department of Radiological Sciences, University of California Los Angeles, Los Angeles, California, USA.
Department of Bioengineering, University of California Los Angeles, Los Angeles, California, USA.
ORCID: 0000-0001-8245-3545

Justin Wood (J)

Department of Neurobiology, David Geffen School of Medicine at UCLA, University of California Los Angeles, Los Angeles, California, USA.
Medical Imaging Informatics, Department of Radiological Sciences, University of California Los Angeles, Los Angeles, California, USA.
Department of Computer Science, University of California Los Angeles, Los Angeles, California, USA.

Alcino J Silva (AJ)

Department of Neurobiology, David Geffen School of Medicine at UCLA, University of California Los Angeles, Los Angeles, California, USA.
Department of Psychiatry & Biobehavioral Sciences, University of California Los Angeles, Los Angeles, California, USA.
Department of Psychology, University of California Los Angeles, Los Angeles, California, USA.
Integrative Center for Learning and Memory, University of California Los Angeles, Los Angeles, California, USA.
Brain Research Institute, University of California Los Angeles, Los Angeles, California, USA.

Classifications MeSH