Psychiatric neuroimaging designs for individualised, cohort, and population studies.
Journal
Neuropsychopharmacology : official publication of the American College of Neuropsychopharmacology
ISSN: 1740-634X
Titre abrégé: Neuropsychopharmacology
Pays: England
ID NLM: 8904907
Informations de publication
Date de publication:
14 Aug 2024
14 Aug 2024
Historique:
received:
01
04
2024
accepted:
11
06
2024
revised:
30
05
2024
medline:
15
8
2024
pubmed:
15
8
2024
entrez:
14
8
2024
Statut:
aheadofprint
Résumé
Psychiatric neuroimaging faces challenges to rigour and reproducibility that prompt reconsideration of the relative strengths and limitations of study designs. Owing to high resource demands and varying inferential goals, current designs differentially emphasise sample size, measurement breadth, and longitudinal assessments. In this overview and perspective, we provide a guide to the current landscape of psychiatric neuroimaging study designs with respect to this balance of scientific goals and resource constraints. Through a heuristic data cube contrasting key design features, we discuss a resulting trade-off among small sample, precision longitudinal studies (e.g., individualised studies and cohorts) and large sample, minimally longitudinal, population studies. Precision studies support tests of within-person mechanisms, via intervention and tracking of longitudinal course. Population studies support tests of generalisation across multifaceted individual differences. A proposed reciprocal validation model (RVM) aims to recursively leverage these complementary designs in sequence to accumulate evidence, optimise relative strengths, and build towards improved long-term clinical utility.
Identifiants
pubmed: 39143320
doi: 10.1038/s41386-024-01918-y
pii: 10.1038/s41386-024-01918-y
doi:
Types de publication
Journal Article
Review
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2024. The Author(s).
Références
Noble S, Spann MN, Tokoglu F, Shen X, Constable RT, Scheinost D. Influences on the test–retest reliability of functional connectivity mri and its relationship with behavioral utility. Cereb Cortex. 2017;27:5415–29.
pubmed: 28968754
pmcid: 6248395
doi: 10.1093/cercor/bhx230
Milham MP, Vogelstein J, Xu T. Removing the reliability bottleneck in functional magnetic resonance imaging research to achieve clinical utility. JAMA Psychiatry. 2021;78:587–8.
pubmed: 33439234
doi: 10.1001/jamapsychiatry.2020.4272
Marek S, Tervo-Clemmens B, Calabro FJ, Montez DF, Kay BP, Hatoum AS, et al. Reproducible brain-wide association studies require thousands of individuals. Nature 2022;603:654–60.
pubmed: 35296861
pmcid: 8991999
doi: 10.1038/s41586-022-04492-9
Nour MM, Liu Y, Dolan RJ. Functional neuroimaging in psychiatry and the case for failing better. Neuron 2022;110:2524–44.
pubmed: 35981525
doi: 10.1016/j.neuron.2022.07.005
Karvelis P, Paulus MP, Diaconescu AO. Individual differences in computational psychiatry: a review of current challenges. Neurosci Biobehav Rev. 2023;148:105137.
pubmed: 36940888
doi: 10.1016/j.neubiorev.2023.105137
Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 2020;582:84–88.
pubmed: 32483374
pmcid: 7771346
doi: 10.1038/s41586-020-2314-9
Woo C-W, Chang LJ, Lindquist MA, Wager TD. Building better biomarkers: brain models in translational neuroimaging. Nat Neurosci. 2017;20:365–77.
pubmed: 28230847
pmcid: 5988350
doi: 10.1038/nn.4478
Kraus B, Zinbarg R, Braga RM, Nusslock R, Mittal VA, Gratton C. Insights from personalized models of brain and behavior for identifying biomarkers in psychiatry. Neurosci Biobehav Rev. 2023;152:105259.
Gratton C, Nelson SM, Gordon EM. Brain-behavior correlations: two paths toward reliability. Neuron 2022;110:1446–9.
pubmed: 35512638
doi: 10.1016/j.neuron.2022.04.018
Tervo-Clemmens B, Marek S, Barch DM. Tailoring psychiatric neuroimaging to translational goals. JAMA Psychiatry. 2023;80:765–6.
pubmed: 37314757
pmcid: 11195020
doi: 10.1001/jamapsychiatry.2023.1416
Laumann TO, Zorumski CF, Dosenbach NU. Precision neuroimaging for localization-related psychiatry. JAMA Psychiatry. 2023;80:763–4.
pubmed: 37285163
pmcid: 11150161
doi: 10.1001/jamapsychiatry.2023.1576
Ooi LQR, Orban C, Nichols TE, Zhang S, Tan TWK, Kong R, et al. MRI economics: balancing sample size and scan duration in brain wide association studies. 2024:2024.02.16.580448.
March JS, Silva SG, Compton S, Shapiro M, Califf R, Krishnan R. The case for practical clinical trials in psychiatry. AJP 2005;162:836–46.
doi: 10.1176/appi.ajp.162.5.836
Revelle W. Personality structure and measurement: the contributions of Raymond Cattell. Br J Psychol. 2009;100:253–7.
pubmed: 19351450
doi: 10.1348/000712609X413809
De Ribaupierre A, Lecerf T. On the importance of intraindividual variability in cognitive development. J Intell. 2018;6:17.
pubmed: 31162444
pmcid: 6480780
doi: 10.3390/jintelligence6020017
Tiemeier H, Muetzel R. Population Neuroscience. In: Taylor E, Verhulst FC, Wong J, Yoshida K, Nikapota A, editors. Mental Health and Illness of Children and Adolescents, Singapore: Springer; 2020. p. 1–22.
Paus T. Population neuroscience: why and how. Hum Brain Mapp. 2010;31:891–903.
pubmed: 20496380
pmcid: 6871127
doi: 10.1002/hbm.21069
Tervo-Clemmens B, Marek S, Chauvin RJ, Van AN, Kay BP, Laumann TO, et al. Reply to: Multivariate BWAS can be replicable with moderate sample sizes. Nature 2023;615:E8–E12.
pubmed: 36890374
pmcid: 9995264
doi: 10.1038/s41586-023-05746-w
Rosenberg MD, Finn ES. How to establish robust brain–behavior relationships without thousands of individuals. Nat Neurosci. 2022;25:835–7.
pubmed: 35710985
doi: 10.1038/s41593-022-01110-9
Ricard JA, Parker TC, Dhamala E, Kwasa J, Allsop A, Holmes AJ. Confronting racially exclusionary practices in the acquisition and analyses of neuroimaging data. Nat Neurosci. 2023;26:4–11.
pubmed: 36564545
doi: 10.1038/s41593-022-01218-y
Smith JD. Single-case experimental designs: a systematic review of published research and current standards. Psychol Methods. 2012;17:510.
pubmed: 22845874
doi: 10.1037/a0029312
Gordon EM, Laumann TO, Gilmore AW, Newbold DJ, Greene DJ, Berg JJ, et al. Precision functional mapping of individual human brains. Neuron 2017;95:791–807.
pubmed: 28757305
pmcid: 5576360
doi: 10.1016/j.neuron.2017.07.011
Gratton C, Laumann TO, Nielsen AN, Greene DJ, Gordon EM, Gilmore AW, et al. Functional brain networks are dominated by stable group and individual factors, not cognitive or daily variation. Neuron 2018;98:439–52.
pubmed: 29673485
pmcid: 5912345
doi: 10.1016/j.neuron.2018.03.035
Schmiedek F, Lövdén M, Lindenberger U. Hundred days of cognitive training enhance broad cognitive abilities in adulthood: Findings from the COGITO study. Frontiers in Aging. Neuroscience 2010;2:27.
Poldrack RA, Laumann TO, Koyejo O, Gregory B, Hover A, Chen M-Y, et al. Long-term neural and physiological phenotyping of a single human. Nat Commun. 2015;6:8885.
pubmed: 26648521
doi: 10.1038/ncomms9885
Demeter DV, Greene DJ. The promise of precision functional mapping for neuroimaging in psychiatry. Neuropsychopharmacol. 2024. https://doi.org/10.1038/s41386-024-01941-z .
Laumann TO, Gordon EM, Adeyemo B, Snyder AZ, Joo SJ, Chen M-Y, et al. Functional system and areal organization of a highly sampled individual human brain. Neuron 2015;87:657–70.
pubmed: 26212711
pmcid: 4642864
doi: 10.1016/j.neuron.2015.06.037
Pritschet L, Santander T, Taylor CM, Layher E, Yu S, Miller MB, et al. Functional reorganization of brain networks across the human menstrual cycle. NeuroImage 2020;220:117091.
pubmed: 32621974
doi: 10.1016/j.neuroimage.2020.117091
Laumann TO, Ortega M, Hoyt CR, Seider NA, Siegel JS, Nguyen AL, et al. Brain network reorganisation in an adolescent after bilateral perinatal strokes. Lancet Neurol. 2021;20:255–6.
pubmed: 33743230
doi: 10.1016/S1474-4422(21)00062-4
Lynch CJ, Power JD, Scult MA, Dubin M, Gunning FM, Liston C. Rapid precision functional mapping of individuals using multi-echo fMRI. Cell Reports. 2020;33.
Newbold DJ, Laumann TO, Hoyt CR, Hampton JM, Montez DF, Raut RV, et al. Plasticity and spontaneous activity pulses in disused human brain circuits. Neuron 2020;107:580–9.
pubmed: 32778224
pmcid: 7419711
doi: 10.1016/j.neuron.2020.05.007
Lynch CJ, Elbau IG, Ng TH, Wolk D, Zhu S, Ayaz A, et al. Automated optimization of TMS coil placement for personalized functional network engagement. Neuron 2022;110:3263–77.
pubmed: 36113473
doi: 10.1016/j.neuron.2022.08.012
Krause M, Lutz W, Boehnke JR. The role of sampling in clinical trial design. Psychother Res. 2011;21:243–51.
pubmed: 21347979
doi: 10.1080/10503307.2010.549520
Tyrer S, Heyman B. Sampling in epidemiological research: issues, hazards and pitfalls. BJPsych Bull. 2016;40:57–60.
pubmed: 27087985
pmcid: 4817645
doi: 10.1192/pb.bp.114.050203
Samet JM, Muñoz A. Evolution of the cohort study. Epidemiol Rev. 1998;20:1–14.
pubmed: 9762505
doi: 10.1093/oxfordjournals.epirev.a017964
Noble S, Scheinost D, Constable RT. A decade of test-retest reliability of functional connectivity: a systematic review and meta-analysis. Neuroimage 2019;203:116157.
pubmed: 31494250
doi: 10.1016/j.neuroimage.2019.116157
Marinescu IE, Lawlor PN, Kording KP. Quasi-experimental causality in neuroscience and behavioural research. Nat Hum Behav 2018;2:891–8.
pubmed: 30988445
doi: 10.1038/s41562-018-0466-5
Vaidya AR, Pujara MS, Petrides M, Murray EA, Fellows LK. Lesion studies in contemporary neuroscience. Trends Cogn Sci. 2019;23:653–71.
pubmed: 31279672
pmcid: 6712987
doi: 10.1016/j.tics.2019.05.009
Siddiqi SH, Kording KP, Parvizi J, Fox MD. Causal mapping of human brain function. Nat Rev Neurosci. 2022;23:361–75.
pubmed: 35444305
pmcid: 9387758
doi: 10.1038/s41583-022-00583-8
Ross LN, Bassett DS. Causation in neuroscience: keeping mechanism meaningful. Nat Rev Neurosci. 2024;25:81–90.
Philip NS, Barredo J, Aiken E, Carpenter LL. Neuroimaging mechanisms of therapeutic transcranial magnetic stimulation for major depressive disorder. Biol Psychiatry Cogn Neurosci Neuroimaging. 2018;3:211–22.
pubmed: 29486862
Ashkan K, Rogers P, Bergman H, Ughratdar I. Insights into the mechanisms of deep brain stimulation. Nat Rev Neurol. 2017;13:548–54.
pubmed: 28752857
doi: 10.1038/nrneurol.2017.105
Hollunder B, Ostrem JL, Sahin IA, Rajamani N, Oxenford S, Butenko K, et al. Mapping dysfunctional circuits in the frontal cortex using deep brain stimulation. Nat. Neuroscience. 2024:27:573–86.
Wall MB, Harding R, Zafar R, Rabiner EA, Nutt DJ, Erritzoe D. Neuroimaging in psychedelic drug development: past, present, and future. Mol Psychiatry. 2023;28:3573–80.
pubmed: 37759038
pmcid: 10730398
doi: 10.1038/s41380-023-02271-0
Shulman EP, Smith AR, Silva K, Icenogle G, Duell N, Chein J, et al. The dual systems model: review, reappraisal, and reaffirmation. Developmental Cogn Neurosci. 2016;17:103–17.
doi: 10.1016/j.dcn.2015.12.010
Luna B, Wright C. Adolescent brain development: Implications for the juvenile criminal justice system. 2016. 2016.
Casey BJ, Getz S, Galvan A. The adolescent brain. Dev Rev. 2008;28:62–77.
pubmed: 18688292
pmcid: 2500212
doi: 10.1016/j.dr.2007.08.003
Steinberg L. A dual systems model of adolescent risk-taking. Dev Psychobiol. 2010;52:216–24.
pubmed: 20213754
doi: 10.1002/dev.20445
Tervo-Clemmens B, Quach A, Calabro FJ, Foran W, Luna B. Meta-analysis and review of functional neuroimaging differences underlying adolescent vulnerability to substance use. NeuroImage 2020;209:116476.
pubmed: 31875520
doi: 10.1016/j.neuroimage.2019.116476
Hedges EP, Dimitrov M, Zahid U, Vega BB, Si S, Dickson H, et al. Reliability of structural MRI measurements: the effects of scan session, head tilt, inter-scan interval, acquisition sequence, FreeSurfer version and processing stream. Neuroimage 2022;246:118751.
pubmed: 34848299
doi: 10.1016/j.neuroimage.2021.118751
Bethlehem RA, Seidlitz J, White SR, Vogel JW, Anderson KM, Adamson C, et al. Brain charts for the human lifespan. Nature 2022;604:525–33.
pubmed: 35388223
pmcid: 9021021
doi: 10.1038/s41586-022-04554-y
Rutherford S, Kia SM, Wolfers T, Fraza C, Zabihi M, Dinga R, et al. The normative modeling framework for computational psychiatry. Nat Protoc. 2022;17:1711–34.
pubmed: 35650452
pmcid: 7613648
doi: 10.1038/s41596-022-00696-5
Bučková BR, Fraza C, Rehák R, Kolenič M, Beckmann C, Španiel F, et al. Using normative models pre-trained on cross-sectional data to evaluate longitudinal changes in neuroimaging data. 2023:2023.06.09.544217.
Tervo-Clemmens B, Calabro FJ, Parr AC, Fedor J, Foran W, Luna B. A canonical trajectory of executive function maturation from adolescence to adulthood. Nat Commun. 2023;14:1–17.
doi: 10.1038/s41467-023-42540-8
Steel Z, Marnane C, Iranpour C, Chey T, Jackson JW, Patel V, et al. The global prevalence of common mental disorders: a systematic review and meta-analysis 1980-2013. Int J Epidemiol. 2014;43:476–93.
pubmed: 24648481
pmcid: 3997379
doi: 10.1093/ije/dyu038
Seitzman BA, Gratton C, Laumann TO, Gordon EM, Adeyemo B, Dworetsky A, et al. Trait-like variants in human functional brain networks. Proc Natl Acad Sci USA. 2019;116:22851–61.
pubmed: 31611415
pmcid: 6842602
doi: 10.1073/pnas.1902932116
Lynch CJ Jr, Elbau I, Ng T, Ayaz A, Zhu S, Manfredi N, et al. Expansion of a frontostriatal salience network in individuals with depression. bioRxiv. 2023:2023–08.
Owens MM, Potter A, Hyatt CS, Albaugh M, Thompson WK, Jernigan T, et al. Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study. PloS One. 2021;16:e0257535.
pubmed: 34555056
pmcid: 8460025
doi: 10.1371/journal.pone.0257535
Liu S, Abdellaoui A, Verweij KJ, van Wingen GA. Replicable brain–phenotype associations require large-scale neuroimaging data. Nature Human. Behaviour 2023;7:1344–56.
Varoquaux G, Poldrack RA. Predictive models avoid excessive reductionism in cognitive neuroimaging. Curr Opin Neurobiol. 2019;55:1–6.
pubmed: 30513462
doi: 10.1016/j.conb.2018.11.002
Heeringa SG, Berglund PA. A guide for population-based analysis of the Adolescent Brain Cognitive Development (ABCD) Study baseline data. BioRxiv. 2020. 2020.
Marek S, Laumann TO. Replicability and generalizability in population psychiatric neuroimaging. Neuropsychopharmacol. 2024. https://doi.org/10.1038/s41386-024-01960-w .
Laird AR. Large, open datasets for human connectomics research: considerations for reproducible and responsible data use. Neuroimage 2021;244:118579.
pubmed: 34536537
doi: 10.1016/j.neuroimage.2021.118579
Jahanshad N, Lenzini P, Bijsterbosch J. Current best practices and future opportunities for reproducible findings using large-scale neuroimaging in psychiatry. Neuropsychopharmacol. 2024. https://doi.org/10.1038/s41386-024-01938-8 .
Traut N, Heuer K, Lemaître G, Beggiato A, Germanaud D, Elmaleh M, et al. Insights from an autism imaging biomarker challenge: Promises and threats to biomarker discovery. NeuroImage 2022;255:119171.
pubmed: 35413445
doi: 10.1016/j.neuroimage.2022.119171
Spisak T, Bingel U, Wager TD. Multivariate BWAS can be replicable with moderate sample sizes. Nature 2023;615:E4–E7.
pubmed: 36890392
pmcid: 9995263
doi: 10.1038/s41586-023-05745-x
Eickhoff SB, Langner R. Neuroimaging-based prediction of mental traits: road to utopia or Orwell? PLoS Biol. 2019;17:e3000497.
pubmed: 31725713
pmcid: 6879158
doi: 10.1371/journal.pbio.3000497
Thompson PM, Stein JL, Medland SE, Hibar DP, Vasquez AA, Renteria ME, et al. The ENIGMA Consortium: large-scale collaborative analyses of neuroimaging and genetic data. Brain Imaging Behav. 2014;8:153–82.
pubmed: 24399358
pmcid: 4008818
doi: 10.1007/s11682-013-9269-5
Norman LJ, Sudre G, Price J, Shaw P. Subcortico-cortical dysconnectivity in ADHD: a voxel-wise mega-analysis across multiple cohorts. AJP. 2024:appi.ajp.20230026.
Cuthbert BN, Insel TR. Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC Med. 2013;11:126.
pubmed: 23672542
pmcid: 3653747
doi: 10.1186/1741-7015-11-126
Kotov R, Krueger RF, Watson D, Achenbach TM, Althoff RR, Bagby RM, et al. The hierarchical taxonomy of psychopathology (HiTOP): a dimensional alternative to traditional nosologies. J Abnorm Psychol. 2017;126:454–77.
pubmed: 28333488
doi: 10.1037/abn0000258
Greene AS, Constable RT. Clinical promise of brain-phenotype modeling: a review. JAMA Psychiatry. 2023;80:848–54.
Dhamala E, Yeo BTT, Holmes AJ. One size does not fit all: methodological considerations for brain-based predictive modeling in psychiatry. Biol Psychiatry. 2022. https://doi.org/10.1016/j.biopsych.2022.09.024 .
doi: 10.1016/j.biopsych.2022.09.024
pubmed: 36577634
Easley T, Chen R, Hannon K, Dutt R, Bijsterbosch J. Population modeling with machine learning can enhance measures of mental health - Open-data replication. Neuroimage: Rep. 2023;3:100163.
doi: 10.1016/j.ynirp.2023.100163
Hermosillo RJ, Moore LA, Feczko E, Miranda-Domínguez Ó, Pines A, Dworetsky A, et al. A precision functional atlas of personalized network topography and probabilities. Nat Neurosci. 2024;27:1000–13.
Byington N, Grimsrud G, Mooney MA, Cordova M, Doyle O, Hermosillo RJ, et al. Polyneuro risk scores capture widely distributed connectivity patterns of cognition. Dev Cogn Neurosci. 2023;60:101231.
pubmed: 36934605
pmcid: 10031023
doi: 10.1016/j.dcn.2023.101231
He T, An L, Chen P, Chen J, Feng J, Bzdok D, et al. Meta-matching as a simple framework to translate phenotypic predictive models from big to small data. Nat Neurosci. 2022;25:795–804.
pubmed: 35578132
pmcid: 9202200
doi: 10.1038/s41593-022-01059-9
Greene AS, Shen X, Noble S, Horien C, Hahn CA, Arora J, et al. Brain–phenotype models fail for individuals who defy sample stereotypes. Nature 2022;609:109–18.
pubmed: 36002572
pmcid: 9433326
doi: 10.1038/s41586-022-05118-w
Winter NR, Leenings R, Ernsting J, Sarink K, Fisch L, Emden D, et al. Quantifying deviations of brain structure and function in major depressive disorder across neuroimaging modalities. JAMA Psychiatry. 2022;79:879–88.
pubmed: 35895072
pmcid: 9330277
doi: 10.1001/jamapsychiatry.2022.1780
Kang K, Seidlitz J, Bethlehem RA, Xiong J, Jones MT, Mehta K, et al. Study design features that improve effect sizes in cross-sectional and longitudinal brain-wide association studies. bioRxiv. 2023. 2023.
Amanat S, Requena T, Lopez-Escamez JA. A systematic review of extreme phenotype strategies to search for rare variants in genetic studies of complex disorders. Genes 2020;11:987.
pubmed: 32854191
pmcid: 7564972
doi: 10.3390/genes11090987
Preacher KJ, Rucker DD, MacCallum RC, Nicewander WA. Use of the extreme groups approach: a critical reexamination and new recommendations. Psychol Methods. 2005;10:178.
pubmed: 15998176
doi: 10.1037/1082-989X.10.2.178
Fisher JE, Guha A, Heller W, Miller GA. Extreme-groups designs in studies of dimensional phenomena: Advantages, caveats, and recommendations. J Abnorm Psychol. 2020;129:14.
pubmed: 31657600
doi: 10.1037/abn0000480
Komeyer V, Eickhoff SB, Grefkes C, Patil KR, Raimondo F. A framework for confounder considerations in AI-driven precision medicine. 2024:2024.02.02.24302198.
Feczko E, Fair DA. Methods and challenges for assessing heterogeneity. Biol Psychiatry. 2020;88:9–17.
pubmed: 32386742
pmcid: 8404882
doi: 10.1016/j.biopsych.2020.02.015
Flake JK, Fried EI. Measurement schmeasurement: questionable measurement practices and how to avoid them. Adv Methods Pr Psychological Sci. 2020;3:456–65.
Fried EI, Flake JK, Robinaugh DJ. Revisiting the theoretical and methodological foundations of depression measurement. Nat Rev Psychol 2022;1:358–68.
pubmed: 38107751
pmcid: 10723193
doi: 10.1038/s44159-022-00050-2
Hedge C, Powell G, Sumner P. The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behav Res. 2018;50:1166–86.
doi: 10.3758/s13428-017-0935-1
Gell M, Eickhoff SB, Omidvarnia A, Küppers V, Patil KR, Satterthwaite TD, et al. the burden of reliability: how measurement noise limits brain-behaviour predictions. 2024:2023.02.09.527898.
Nikolaidis A, Chen AA, He X, Shinohara R, Vogelstein J, Milham M, et al. Suboptimal phenotypic reliability impedes reproducible human neuroscience. 2022:2022.07.22.501193.
Piantadosi S, Byar DP, Green SB. The ecological fallacy. Am J Epidemiol. 1988;127:893–904.
pubmed: 3282433
doi: 10.1093/oxfordjournals.aje.a114892
Pepe MS, Etzioni R, Feng Z, Potter JD, Thompson ML, Thornquist M, et al. Phases of biomarker development for early detection of cancer. J Natl Cancer Inst. 2001;93:1054–61.
pubmed: 11459866
doi: 10.1093/jnci/93.14.1054
Gordon EM, Chauvin RJ, Van AN, Rajesh A, Nielsen A, Newbold DJ, et al. A somato-cognitive action network alternates with effector regions in motor cortex. Nature 2023;617:351–9.
pubmed: 37076628
pmcid: 10172144
doi: 10.1038/s41586-023-05964-2
Tukey JW. We need both exploratory and confirmatory. Am Statistician. 1980;34:23–25.
doi: 10.1080/00031305.1980.10482706
Fife DA, Rodgers JL. Understanding the exploratory/confirmatory data analysis continuum: moving beyond the “replication crisis”. Am Psychol. 2022;77:453.
pubmed: 34780242
doi: 10.1037/amp0000886
Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med. 2016;8:341ps12.