Why experimental variation in neuroimaging should be embraced.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
31 Oct 2024
31 Oct 2024
Historique:
received:
22
09
2023
accepted:
21
10
2024
medline:
1
11
2024
pubmed:
1
11
2024
entrez:
1
11
2024
Statut:
epublish
Résumé
In a perfect world, scientists would develop analyses that are guaranteed to reveal the ground truth of a research question. In reality, there are countless viable workflows that produce distinct, often conflicting, results. Although reproducibility places a necessary bound on the validity of results, it is not sufficient for claiming underlying validity, eventual utility, or generalizability. In this work we focus on how embracing variability in data analysis can improve the generalizability of results. We contextualize how design decisions in brain imaging can be made to capture variation, highlight examples, and discuss how variability capture may improve the quality of results.
Identifiants
pubmed: 39482294
doi: 10.1038/s41467-024-53743-y
pii: 10.1038/s41467-024-53743-y
doi:
Types de publication
Journal Article
Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
9411Subventions
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (NIMH)
ID : RF1MH130859
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (NIMH)
ID : R01MH120482
Informations de copyright
© 2024. The Author(s).
Références
Ritchie, S. J. et al. Sex differences in the adult human brain: evidence from 5216 UK biobank participants. Cereb. Cortex 28, 2959–2975 (2018).
pubmed: 29771288
pmcid: 6041980
doi: 10.1093/cercor/bhy109
Aycheh, H. M. et al. Biological brain age prediction using cortical thickness data: a large scale cohort study. Front. Aging Neurosci. 10, 252 (2018).
pubmed: 30186151
pmcid: 6113379
doi: 10.3389/fnagi.2018.00252
Hidese, S. et al. Association of obesity with cognitive function and brain structure in patients with major depressive disorder. J. Affect. Disord. 225, 188–194 (2018).
pubmed: 28837952
doi: 10.1016/j.jad.2017.08.028
Ledig, C., Schuh, A., Guerrero, R., Heckemann, R. A. & Rueckert, D. Structural brain imaging in Alzheimer’s disease and mild cognitive impairment: biomarker analysis and shared morphometry database. Sci. Rep. 8, 11258 (2018).
pubmed: 30050078
pmcid: 6062561
doi: 10.1038/s41598-018-29295-9
Tolan, E. & Isik, Z. Graph theory based classification of brain connectivity network for autism spectrum disorder. in Bioinformatics and Biomedical Engineering 520–530 (Springer International Publishing, 2018).
Zhu, X., Du, X., Kerich, M., Lohoff, F. W. & Momenan, R. Random forest based classification of alcohol dependence patients and healthy controls using resting state MRI. Neurosci. Lett. 676, 27–33 (2018).
pubmed: 29626649
pmcid: 5960433
doi: 10.1016/j.neulet.2018.04.007
Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
doi: 10.1126/science.aac4716
Esteban, O. et al. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat. Methods 16, 111–116 (2019).
pubmed: 30532080
doi: 10.1038/s41592-018-0235-4
Glasser, M. F. et al. The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage 80, 105–124 (2013).
pubmed: 23668970
doi: 10.1016/j.neuroimage.2013.04.127
Volkow, N. D. et al. The conception of the ABCD study: from substance use to a broad NIH collaboration. Dev. Cogn. Neurosci. 32, 4–7 (2018).
pubmed: 29051027
doi: 10.1016/j.dcn.2017.10.002
Markiewicz, C. J. et al. OpenNeuro: an open resource for sharing of neuroimaging data. bioRxiv 2021.06.28.450168 https://doi.org/10.1101/2021.06.28.450168 (2021).
Rougier, N. P. et al. Sustainable computational science: the ReScience initiative. PeerJ Comput Sci. 3, e142 (2017).
pubmed: 34722870
pmcid: 8530091
doi: 10.7717/peerj-cs.142
Plesser, H. E. Reproducibility vs. replicability: a brief history of a confused terminology. Front. Neuroinform. 11, 76 (2017).
pubmed: 29403370
doi: 10.3389/fninf.2017.00076
Charter, R. A. Methodological commentary: effect of measurement error on tests of statistical significance. J. Clin. Exp. Neuropsychol. 19, 458–462 (1997).
pubmed: 9268818
doi: 10.1080/01688639708403872
Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638 (1979).
doi: 10.1037/0033-2909.86.3.638
Thompson, W. H., Wright, J., Bissett, P. G. & Poldrack, R. A. Meta-research: dataset decay and the problem of sequential analyses on open datasets. Elife 9, e53498 (2020).
pubmed: 32425159
pmcid: 7237204
doi: 10.7554/eLife.53498
Paus, T. Population neuroscience: why and how. Hum. Brain Mapp. 31, 891–903 (2010).
pubmed: 20496380
pmcid: 6871127
doi: 10.1002/hbm.21069
Nielsen, M., Haun, D., Kärtner, J. & Legare, C. H. The persistent sampling bias in developmental psychology: a call to action. J. Exp. Child Psychol. 162, 31–38 (2017).
pubmed: 28575664
pmcid: 10675994
doi: 10.1016/j.jecp.2017.04.017
Charpentier, C. J. et al. How representative are neuroimaging samples? Large-scale evidence for trait anxiety differences between fMRI and behaviour-only research participants. Soc. Cogn. Affect. Neurosci. 16, 1057–1070 (2021).
pubmed: 33950220
pmcid: 8483285
doi: 10.1093/scan/nsab057
Li, J. et al. Cross-Ethnicity/race Generalization Failure of RSFC-Based Behavioral Prediction and Potential Consequences. https://juser.fz-juelich.de/record/910372 (2022).
Guyatt, G. H. et al. Users’ guides to the medical literature: IX. A method for grading health care recommendations. JAMA 274, 1800–1804 (1995).
pubmed: 7500513
doi: 10.1001/jama.1995.03530220066035
Jennings, R. G. & Van Horn, J. D. Publication bias in neuroimaging research: implications for meta-analyses. Neuroinformatics 10, 67–80 (2012).
pubmed: 21643733
doi: 10.1007/s12021-011-9125-y
Traut, N. et al. Cerebellar volume in autism: literature meta-analysis and analysis of the autism brain imaging data exchange cohort. Biol. Psychiatry 83, 579–588 (2018).
pubmed: 29146048
doi: 10.1016/j.biopsych.2017.09.029
Steegen, S., Tuerlinckx, F., Gelman, A. & Vanpaemel, W. Increasing transparency through a multiverse analysis. Perspect. Psychol. Sci. 11, 702–712 (2016).
pubmed: 27694465
doi: 10.1177/1745691616658637
Guest, O. & Martin, A. E. How computational modeling can force theory building in psychological science. Perspect. Psychol. Sci. 16, 789–802 (2021).
Glatard, T. et al. Reproducibility of neuroimaging analyses across operating systems. Front. Neuroinform. 9, 12 (2015).
pubmed: 25964757
doi: 10.3389/fninf.2015.00012
Zuo, X.-N. et al. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1, 140049 (2014).
pubmed: 25977800
pmcid: 4421932
doi: 10.1038/sdata.2014.49
Van Essen, D. C. et al. The WU-Minn human connectome project: an overview. Neuroimage 80, 62–79 (2013).
pubmed: 23684880
doi: 10.1016/j.neuroimage.2013.05.041
Jack, C. R. Jr et al. The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI methods. J. Magn. Reson. Imaging 27, 685–691 (2008).
pubmed: 18302232
pmcid: 2544629
doi: 10.1002/jmri.21049
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).
pubmed: 23571845
doi: 10.1038/nrn3475
Regier, D. A. et al. DSM-5 field trials in the United States and Canada, Part II: test-retest reliability of selected categorical diagnoses. Am. J. Psychiatry 170, 59–70 (2013).
pubmed: 23111466
doi: 10.1176/appi.ajp.2012.12070999
Nikolaidis, A. et al. Suboptimal phenotypic reliability impedes reproducible human neuroscience. bioRxiv 2022.07.22.501193 https://doi.org/10.1101/2022.07.22.501193 (2022).
Gell, M. et al. The burden of reliability: how measurement noise limits brain-behaviour predictions. bioRxiv 2023.02.09.527898 https://doi.org/10.1101/2023.02.09.527898 (2023).
Falk, E. B. et al. What is a representative brain? Neuroscience meets population science. Proc. Natl. Acad. Sci. USA 110, 17615–17622 (2013).
pubmed: 24151336
pmcid: 3816464
doi: 10.1073/pnas.1310134110
Varoquaux, G. Cross-validation failure: small sample sizes lead to large error bars. Neuroimage 180, 68–77 (2018).
pubmed: 28655633
doi: 10.1016/j.neuroimage.2017.06.061
Silberzahn, R. et al. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv. Methods Pract. Psychol. Sci. 1, 337–356 (2018).
doi: 10.1177/2515245917747646
Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).
pubmed: 32483374
pmcid: 7771346
doi: 10.1038/s41586-020-2314-9
Schilling, K. G. et al. Tractography dissection variability: what happens when 42 groups dissect 14 white matter bundles on the same dataset? Neuroimage 243, 118502 (2021).
pubmed: 34433094
doi: 10.1016/j.neuroimage.2021.118502
Fornito, A., Zalesky, A. & Bullmore, E. T. Network scaling effects in graph analytic studies of human resting-state FMRI data. Front. Syst. Neurosci. 4, 22 (2010).
pubmed: 20592949
pmcid: 2893703
Li, X. et al. Moving beyond processing and analysis-related variation in neuroscience. bioRxiv 2021.12.01.470790 https://doi.org/10.1101/2021.12.01.470790 (2021).
Bridgeford, E. W. et al. Eliminating accidental deviations to minimize generalization error and maximize replicability: Applications in connectomics and genomics. PLoS Comput. Biol. 17, e1009279 (2021).
pubmed: 34529652
pmcid: 8500408
doi: 10.1371/journal.pcbi.1009279
Bowring, A., Maumet, C. & Nichols, T. E. Exploring the impact of analysis software on task fMRI results. Hum. Brain Mapp. 40, 3362–3384 (2019).
pubmed: 31050106
pmcid: 6618324
doi: 10.1002/hbm.24603
Bowring, A., Nichols, T. E. & Maumet, C. Isolating the sources of pipeline-variability in group-level task-fMRI results. Hum. Brain Mapp. https://doi.org/10.1002/hbm.25713 (2021).
Bhagwat, N. et al. Understanding the impact of preprocessing pipelines on neuroimaging cortical surface analyses. Gigascience 10, giaa155 (2021).
Fonov, V. S., Evans, A. C., McKinstry, R. C., Almli, C. R. & Collins, D. L. Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neuroimage Suppl. 1, S102 (2009).
doi: 10.1016/S1053-8119(09)70884-5
Lancaster, J. L. et al. Bias between MNI and Talairach coordinates analyzed using the ICBM-152 brain template. Hum. Brain Mapp. 28, 1194–1205 (2007).
pubmed: 17266101
pmcid: 6871323
doi: 10.1002/hbm.20345
Bouthillier, X. et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems 3, (2021).
Parker, D. S. Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic (University of California (Los Angeles). Computer Science Department, 1997).
Skare, S., Hedehus, M., Moseley, M. E. & Li, T. Q. Condition number as a measure of noise performance of diffusion tensor data acquisition schemes with MRI. J. Magn. Reson. 147, 340–352 (2000).
pubmed: 11097823
doi: 10.1006/jmre.2000.2209
Kiar, G. et al. Comparing perturbation models for evaluating stability of neuroimaging pipelines. Int. J. High Perform. Comput. Appl. 34, 491–501 (2020).
Kiar, G. et al. Numerical uncertainty in analytical pipelines lead to impactful variability in brain networks. bioRxiv 2020.10.15.341495 https://doi.org/10.1101/2020.10.15.341495 (2021).
Di Martino, A. et al. The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Mol. Psychiatry 19, 659–667 (2014).
pubmed: 23774715
doi: 10.1038/mp.2013.78
Gorgolewski, K. J. et al. NeuroVault.org: a web-based repository for collecting and sharing unthresholded statistical maps of the human brain. Front. Neuroinform. 9, 8 (2015).
pubmed: 25914639
pmcid: 4392315
doi: 10.3389/fninf.2015.00008
Dockès, J. et al. NeuroQuery, comprehensive meta-analysis of human brain mapping. Elife 9, e53385 (2020).
Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., & Wager, T. D. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8, 665–670 (2011).
Krstajic, D., Buturovic, L. J., Leahy, D. E. & Thomas, S. Cross-validation pitfalls when selecting and assessing regression and classification models. J. Cheminform. 6, 10 (2014).
pubmed: 24678909
pmcid: 3994246
doi: 10.1186/1758-2946-6-10
Traut, N. et al. Insights from an autism imaging biomarker challenge: promises and threats to biomarker discovery. Neuroimage 255, 119171 (2022).
pubmed: 35413445
doi: 10.1016/j.neuroimage.2022.119171
Olivetti, E., Greiner, S. & Avesani, P. ADHD diagnosis from multiple data sources with batch effects. Front. Syst. Neurosci. 6, 70 (2012).
pubmed: 23060755
pmcid: 3465911
doi: 10.3389/fnsys.2012.00070
Gau, R. et al. Brainhack: developing a culture of open, inclusive, community-driven neuroscience. Neuron 109, 1769–1775 (2021).
pubmed: 33932337
pmcid: 9153215
doi: 10.1016/j.neuron.2021.04.001
Craddock, C. et al. Towards automated analysis of connectomes: the configurable pipeline for the analysis of connectomes (c-pac). Front. Neuroinform. 42, 10–3389 (2013).
Glatard, T. et al. Boutiques: a flexible framework to integrate command-line applications in computing platforms. Gigascience 7, giy016 (2018).
Amstutz, P. et al. Common Workflow Language, draft 3. https://doi.org/10.6084/m9.figshare.3115156.v1 (2016).
Halchenko, Y. et al. DataLad: distributed system for joint management of code, data, and their relationship. J. Open Source Softw. 6, 3262 (2021).
pubmed: 39469147
doi: 10.21105/joss.03262
Kiar, G. et al. Verificarlo/fuzzy: Fuzzy v0.5.0. https://doi.org/10.5281/zenodo.5027708 (2021).
Chatelain, Y. et al. A numerical variability approach to results stability tests and its application to neuroimaging. arXiv [physics.med-ph] (2023).
Gau, R. et al. COBIDAS checklist. Preprint at https://doi.org/10.17605/OSF.IO/ANVQY (2019).
Salkind, N. J. (Ed.) Encyclopedia of research design. SAGE Publications, Inc., https://doi.org/10.4135/9781412961288 (2010).
Gueorguieva, R. & Krystal, J. H. Move over ANOVA: progress in analyzing repeated-measures data and its reflection in papers published in the archives of general psychiatry. Arch. Gen. Psychiatry 61, 310–317 (2004).
pubmed: 14993119
doi: 10.1001/archpsyc.61.3.310
Simmons, J., Nelson, L. & Simonsohn, U. Pre‐registration: why and how. J. Consum. Psychol. 31, 151–162 (2021).
doi: 10.1002/jcpy.1208
Baribault, B. et al. Metastudies for robust tests of theory. Proc. Natl. Acad. Sci. USA 115, 2607–2612 (2018).
pubmed: 29531092
pmcid: 5856505
doi: 10.1073/pnas.1708285114
Brandmaier, A. M. et al. Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED). Elife 7, e35718 (2018).
Kiar, G., Chatelain, Y., Salari, A., Evans, A. C. & Glatard, T. Data augmentation through Monte Carlo arithmetic leads to more generalizable classification in connectomics. Neurons Behav. Data Theory https://doi.org/10.51628/001c.28328 (2021).
Bellec, P., Rosa-Neto, P., Benali, H. & Evans, A. C. Multi-level bootstrap analysis of stable clusters (BASC) in resting-state fMRI. NeuroImage 47, S123 (2009).
Nikolaidis, A. et al. Bagging improves reproducibility of functional parcellation of the human brain. Neuroimage 214, 116678 (2020).
pubmed: 32119986
doi: 10.1016/j.neuroimage.2020.116678
Doshi, J. et al. MUSE: MUlti-atlas region Segmentation utilizing Ensembles of registration algorithms and parameters, and locally optimal atlas selection. Neuroimage 127, 186–195 (2016).
pubmed: 26679328
doi: 10.1016/j.neuroimage.2015.11.073