Multi-site benchmark classification of major depressive disorder using machine learning on cortical and subcortical measures.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
11 Jan 2024
11 Jan 2024
Historique:
received:
23
01
2023
accepted:
19
11
2023
medline:
12
1
2024
pubmed:
12
1
2024
entrez:
11
1
2024
Statut:
epublish
Résumé
Machine learning (ML) techniques have gained popularity in the neuroimaging field due to their potential for classifying neuropsychiatric disorders. However, the diagnostic predictive power of the existing algorithms has been limited by small sample sizes, lack of representativeness, data leakage, and/or overfitting. Here, we overcome these limitations with the largest multi-site sample size to date (N = 5365) to provide a generalizable ML classification benchmark of major depressive disorder (MDD) using shallow linear and non-linear models. Leveraging brain measures from standardized ENIGMA analysis pipelines in FreeSurfer, we were able to classify MDD versus healthy controls (HC) with a balanced accuracy of around 62%. But after harmonizing the data, e.g., using ComBat, the balanced accuracy dropped to approximately 52%. Accuracy results close to random chance levels were also observed in stratified groups according to age of onset, antidepressant use, number of episodes and sex. Future studies incorporating higher dimensional brain imaging/phenotype features, and/or using more advanced machine and deep learning methods may yield more encouraging prospects.
Identifiants
pubmed: 38212349
doi: 10.1038/s41598-023-47934-8
pii: 10.1038/s41598-023-47934-8
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
1084Informations de copyright
© 2024. The Author(s).
Références
Kessler, R. C. & Bromet, E. J. The epidemiology of depression across cultures. Annu. Rev. Public Health 34, 119–138 (2013).
pubmed: 23514317
pmcid: 4100461
doi: 10.1146/annurev-publhealth-031912-114409
Cho, Y. et al. Factors associated with quality of life in patients with depression: A nationwide population-based study. PLOS ONE 14, e0219455 (2019).
pubmed: 31295291
pmcid: 6623963
doi: 10.1371/journal.pone.0219455
Cai, H. et al. Prevalence of suicidality in major depressive disorder: A systematic review and meta-analysis of comparative studies. Front. Psychiatry 12, (2021).
Cleare, A. F. S. W. C. D. K. M. B. M. L. P. A. J. A multidimensional tool to quantify treatment resistance in depression: The Maudsley staging method. J. Clin. Psychiatry 70, 12363 (2009).
Han, L. K. M. et al. Brain aging in major depressive disorder: Results from the ENIGMA major depressive disorder working group. Mol. Psychiatry https://doi.org/10.1038/s41380-020-0754-0 (2020).
doi: 10.1038/s41380-020-0754-0
pubmed: 32989244
pmcid: 8589681
Kraus, C., Kadriu, B., Lanzenberger, R., Zarate, C. A. Jr. & Kasper, S. Prognosis and improved outcomes in major depression: A review. Transl. Psychiatry 9, 1–17 (2019).
doi: 10.1038/s41398-019-0460-3
Gorman, J. M. Comorbid depression and anxiety spectrum disorders. Depress. Anxiety 4, 160–168 (1996).
pubmed: 9166648
doi: 10.1002/(SICI)1520-6394(1996)4:4<160::AID-DA2>3.0.CO;2-J
Steffen, A., Nübel, J., Jacobi, F., Bätzing, J. & Holstiege, J. Mental and somatic comorbidity of depression: A comprehensive cross-sectional analysis of 202 diagnosis groups using German nationwide ambulatory claims data. BMC Psychiatry 20, 142 (2020).
pubmed: 32228541
pmcid: 7106695
doi: 10.1186/s12888-020-02546-8
Arnone, D., McIntosh, A. M., Ebmeier, K. P., Munafò, M. R. & Anderson, I. M. Magnetic resonance imaging studies in unipolar depression: Systematic review and meta-regression analyses. Eur. Neuropsychopharmacol. 22, 1–16 (2012).
pubmed: 21723712
doi: 10.1016/j.euroneuro.2011.05.003
Schmaal, L. et al. Cortical abnormalities in adults and adolescents with major depression based on brain scans from 20 cohorts worldwide in the ENIGMA Major Depressive Disorder Working Group. Mol. Psychiatry 22, 900–909 (2017).
pubmed: 27137745
doi: 10.1038/mp.2016.60
Schmaal, L. et al. Subcortical brain alterations in major depressive disorder: Findings from the ENIGMA Major Depressive Disorder working group. Mol. Psychiatry 21, 806–812 (2016).
pubmed: 26122586
doi: 10.1038/mp.2015.69
Thompson, P. M. et al. The ENIGMA Consortium: Large-scale collaborative analyses of neuroimaging and genetic data. Brain Imaging Behav. 8, 153–182 (2014).
pubmed: 24399358
pmcid: 4008818
doi: 10.1007/s11682-013-9269-5
Zhao, Y.-J. et al. Brain grey matter abnormalities in medication-free patients with major depressive disorder: A meta-analysis. Psychol. Med. 44, 2927–2937 (2014).
pubmed: 25065859
doi: 10.1017/S0033291714000518
Gao, S., Calhoun, V. D. & Sui, J. Machine learning in major depression: From classification to treatment outcome prediction. CNS Neurosci. Ther. 24, 1037–1052 (2018).
pubmed: 30136381
pmcid: 6324186
doi: 10.1111/cns.13048
Kambeitz, J. et al. Detecting neuroimaging biomarkers for depression: A meta-analysis of multivariate pattern recognition studies. Biol. Psychiatry 82, 330–338 (2017).
pubmed: 28110823
doi: 10.1016/j.biopsych.2016.10.028
Flint, C. et al. Systematic misestimation of machine learning performance in neuroimaging studies of depression. Neuropsychopharmacol. https://doi.org/10.1038/s41386-021-01020-7 (2021).
doi: 10.1038/s41386-021-01020-7
Stolicyn, A. et al. Automated classification of depression from structural brain measures across two independent community-based cohorts. Hum. Brain Mapp. 41, 3922–3937 (2020).
pubmed: 32558996
pmcid: 7469862
doi: 10.1002/hbm.25095
Algermissen, J. & Mehler, D. May the power be with you: Are there highly powered studies in neuroscience, and how can we get more of them? J. Neurophysiol. 119, (2018).
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F. & Baker, C. I. Circular analysis in systems neuroscience: The dangers of double dipping. Nat. Neurosci. 12, 535–540 (2009).
pubmed: 19396166
pmcid: 2841687
doi: 10.1038/nn.2303
Zhang-James, Y., Hoogman, M., Franke, B. & Faraone, S. V. Machine Learning And MRI-Based Diagnostic Models For ADHD: Are We There Yet? 2020.10.20.20216390 https://www.medrxiv.org/content/ https://doi.org/10.1101/2020.10.20.20216390v1 (2020). https://doi.org/10.1101/2020.10.20.20216390 .
Duerden, E., Chakravarty, M., Lerch, J. & Taylor, M. Sex-based differences in cortical and subcortical development in 436 individuals aged 4–54 years. Cereb. Cortex (New York, N.Y. : 1991) 30, (2019).
Gennatas, E. D. et al. Age-related effects and sex differences in gray matter density, volume, mass, and cortical thickness from childhood to young adulthood. J. Neurosci. 37, 5065–5073 (2017).
pubmed: 28432144
pmcid: 5444192
doi: 10.1523/JNEUROSCI.3550-16.2017
Schmaal, L. et al. ENIGMA MDD: Seven years of global neuroimaging studies of major depression through worldwide data sharing. Transl. Psychiatry 10, 1–19 (2020).
doi: 10.1038/s41398-020-0842-6
Shrout, P. E. & Rodgers, J. L. Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69, 487–510 (2018).
pubmed: 29300688
doi: 10.1146/annurev-psych-122216-011845
Takao, H., Hayashi, N. & Ohtomo, K. Effect of scanner in longitudinal studies of brain volume changes. J. Magn. Reson. Imaging 34, 438–444 (2011).
pubmed: 21692137
doi: 10.1002/jmri.22636
Brown, E. C., Clark, D. L., Hassel, S., MacQueen, G. & Ramasubbu, R. Intrinsic thalamocortical connectivity varies in the age of onset subtypes in major depressive disorder. Neuropsychiatr. Dis. Treat. 15, 75–82 (2018).
pubmed: 30613149
pmcid: 6306066
doi: 10.2147/NDT.S184425
LeWinn, K. Z., Sheridan, M. A., Keyes, K. M., Hamilton, A. & McLaughlin, K. A. Sample composition alters associations between age and brain structure. Nat. Commun. 8, 874 (2017).
pubmed: 29026076
pmcid: 5638928
doi: 10.1038/s41467-017-00908-7
Solanes, A. et al. Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site. Psychiatry Res. Neuroimaging 314, 111313 (2021).
pubmed: 34098248
doi: 10.1016/j.pscychresns.2021.111313
Johnson, W. E., Li, C. & Rabinovic, A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 8, 118–127 (2007).
pubmed: 16632515
doi: 10.1093/biostatistics/kxj037
Fortin, J.-P. et al. Harmonization of cortical thickness measurements across scanners and sites. Neuroimage 167, 104–120 (2018).
doi: 10.1016/j.neuroimage.2017.11.024
pubmed: 29155184
Fortin, J.-P. et al. Harmonization of multi-site diffusion tensor imaging data. Neuroimage 161, 149–170 (2017).
pubmed: 28826946
doi: 10.1016/j.neuroimage.2017.08.047
Radua, J. et al. Increased power by harmonizing structural MRI site differences with the ComBat batch adjustment method in ENIGMA. NeuroImage 218, (2020).
Abraham, A. et al. Deriving reproducible biomarkers from multi-site resting-state data: An Autism-based example. NeuroImage 147, 736–745 (2017).
pubmed: 27865923
doi: 10.1016/j.neuroimage.2016.10.045
Pomponio, R. et al. Harmonization of large MRI datasets for the analysis of brain imaging patterns throughout the lifespan. Neuroimage 208, 116450 (2020).
pubmed: 31821869
doi: 10.1016/j.neuroimage.2019.116450
Chen, A. A. et al. Removal of Scanner Effects in Covariance Improves Multivariate Pattern Analysis in Neuroimaging Data. bioRxiv 858415 (2020). https://doi.org/10.1101/858415 .
Mårtensson, G. et al. The reliability of a deep learning model in clinical out-of-distribution MRI data: A multicohort study. Med. Image Anal. 66, 101714 (2020).
pubmed: 33007638
doi: 10.1016/j.media.2020.101714
Rozycki, M. et al. Multisite machine learning analysis provides a robust structural imaging signature of schizophrenia detectable across diverse patient populations and within individuals. Schizophr. Bull. 44, 1035–1044 (2018).
pubmed: 29186619
doi: 10.1093/schbul/sbx137
Zindler, T., Frieling, H., Neyazi, A., Bleich, S. & Friedel, E. Simulating ComBat: How batch correction can lead to the systematic introduction of false positive results in DNA methylation microarray studies. BMC Bioinform. 21, (2020).
Dinga, R., Schmaal, L., Penninx, B. W. J. H., Veltman, D. J. & Marquand, A. F. Controlling for effects of confounding variables on machine learning predictions. http://biorxiv.org/lookup/doi/ https://doi.org/10.1101/2020.08.17.255034 (2020). https://doi.org/10.1101/2020.08.17.255034 .
Tran, H. T. N. et al. A benchmark of batch-effect correction methods for single-cell RNA sequencing data. Genome Biol. 21, 12 (2020).
pubmed: 31948481
pmcid: 6964114
doi: 10.1186/s13059-019-1850-9
Dadi, K. et al. Benchmarking functional connectome-based predictive models for resting-state fMRI. NeuroImage 192, 115–134 (2019).
pubmed: 30836146
doi: 10.1016/j.neuroimage.2019.02.062
Jung, J.-H. et al. Penalized logistic regression using functional connectivity as covariates with an application to mild cognitive impairment. Commun. Stat. Appl. Methods 27, 603–624 (2020).
Caprihan, A., Pearlson, G. D. & Calhoun, V. D. Application of principal component analysis to distinguish patients with schizophrenia from healthy controls based on fractional anisotropy measurements. Neuroimage 42, 675–682 (2008).
pubmed: 18571937
doi: 10.1016/j.neuroimage.2008.04.255
Kim, D. et al. Machine learning classification of first-onset drug-naive MDD using structural MRI. IEEE Access 7, 153977–153985 (2019).
doi: 10.1109/ACCESS.2019.2949128
Ma, Q. et al. Classification of multi-site MR images in the presence of heterogeneity using multi-task learning. NeuroImage Clin. 19, 476–486 (2018).
pubmed: 29984156
pmcid: 6029565
doi: 10.1016/j.nicl.2018.04.037
Wen, J. et al. Convolutional neural networks for classification of Alzheimer’s disease: Overview and reproducible evaluation. Med. Image Anal. 63, 101694 (2020).
pubmed: 32417716
doi: 10.1016/j.media.2020.101694
Dockès, J., Varoquaux, G. & Poline, J.-B. Preventing dataset shift from breaking machine-learning biomarkers. http://arxiv.org/abs/2107.09947 (2021).
Hopkins, W., Li, X., Crow, T. & Roberts, N. Vertex- and atlas-based comparisons in measures of cortical thickness, gyrification and white matter volume between humans and chimpanzees. Brain Struct. Funct. 222, (2017).
Petrusic, I., Marko, D., Kacar, K. & Zidverc-Trajkovic, J. Migraine with aura: Surface-based analysis of the cerebral cortex with magnetic resonance imaging. Korean J. Radiol. 19, 767 (2018).
pubmed: 29962883
pmcid: 6005951
doi: 10.3348/kjr.2018.19.4.767
Hellewell, S. C. et al. Profound and reproducible patterns of reduced regional gray matter characterize major depressive disorder. Transl. Psychiatry 9, (2019).
Ho, T. C. et al. Subcortical shape alterations in major depressive disorder: Findings from the ENIGMA major depressive disorder working group. Hum. Brain Mapp. 43, 341–351 (2022).
pubmed: 32198905
doi: 10.1002/hbm.24988
Xu, D. et al. Diffusion tensor imaging brain structural clustering patterns in major depressive disorder. Hum. Brain Mapp. 42, 5023–5036 (2021).
pubmed: 34312935
pmcid: 8449115
doi: 10.1002/hbm.25597
Qin, K. et al. Using graph convolutional network to characterize individuals with major depressive disorder across multiple imaging sites. eBioMedicine 78, 103977 (2022).
pubmed: 35367775
pmcid: 8983334
doi: 10.1016/j.ebiom.2022.103977
Ramezani, M. et al. Temporal-lobe morphology differs between healthy adolescents and those with early-onset of depression. NeuroImage Clin. 6, 145–155 (2014).
pubmed: 25379426
pmcid: 4215529
doi: 10.1016/j.nicl.2014.08.007
Tu, P.-C. et al. Regional cortical thinning in patients with major depressive disorder: A surface-based morphometry study. Psychiatry Res. Neuroimaging 202, 206–213 (2012).
doi: 10.1016/j.pscychresns.2011.07.011
Lener, M. et al. Cortical abnormalities and association with symptom dimensions across the depressive spectrum. J. Affect. Disord. 190, 529–536 (2015).
pubmed: 26571102
pmcid: 4764252
doi: 10.1016/j.jad.2015.10.027
Fung, G. et al. Distinguishing bipolar and major depressive disorders by brain structural morphometry: A pilot study. BMC Psychiatry 15, (2015).
Iscan, Z. et al. Test–retest reliability of freesurfer measurements within and between sites: Effects of visual approval process. Hum. Brain Mapp. 36, 3472–3485 (2015).
pubmed: 26033168
pmcid: 4545736
doi: 10.1002/hbm.22856
Li, J. et al. Cortical structural differences in major depressive disorder correlate with cell type-specific transcriptional signatures. Nat. Commun. 12, 1647 (2021).
pubmed: 33712584
pmcid: 7955076
doi: 10.1038/s41467-021-21943-5
Qiu, L. et al. Characterization of major depressive disorder using a multiparametric classification approach based on high resolution structural images. J. Psychiatry Neurosci. 39, 78–86 (2014).
pubmed: 24083459
pmcid: 3937284
Li, J. et al. White-matter functional topology: A neuromarker for classification and prediction in unmedicated depression. Transl. Psychiatry 10, 1–10 (2020).
doi: 10.1038/s41398-020-01053-4
Gao, K. et al. Deep transfer learning for cerebral cortex using area-preserving geometry mapping. Cereb. Cortex https://doi.org/10.1093/cercor/bhab394 (2021).
doi: 10.1093/cercor/bhab394
pubmed: 34148074
pmcid: 9618162
Yang, X. et al. Sex differences in the clinical characteristics and brain gray matter volume alterations in unmedicated patients with major depressive disorder. Sci. Rep. 7, 2515 (2017).
pubmed: 28559571
pmcid: 5449404
doi: 10.1038/s41598-017-02828-4
Liang, S. et al. White matter abnormalities in major depression biotypes identified by diffusion tensor imaging. Neurosci. Bull. 35, 867–876 (2019).
pubmed: 31062333
pmcid: 6754492
doi: 10.1007/s12264-019-00381-w
Winter, N. R. et al. Quantifying deviations of brain structure and function in major depressive disorder across neuroimaging modalities. JAMA Psychiatry 79, 879–888 (2022).
pubmed: 35895072
pmcid: 9330277
doi: 10.1001/jamapsychiatry.2022.1780
Goya-Maldonado, R. et al. Differentiating unipolar and bipolar depression by alterations in large-scale brain networks. Hum. Brain Mapp. 37, 808–818 (2016).
pubmed: 26611711
doi: 10.1002/hbm.23070
Whalley, H. C. et al. Prediction of depression in individuals at high familial risk of mood disorders using functional magnetic resonance imaging. PLOS ONE 8, e57357 (2013).
pubmed: 23483904
pmcid: 3590244
doi: 10.1371/journal.pone.0057357
Missing Data: Five Practical Guidelines—Daniel A. Newman (2014). https://journals.sagepub.com/doi/full/ https://doi.org/10.1177/1094428114548590 .
Han, X. et al. Reliability of MRI-derived measurements of human cerebral cortical thickness: The effects of field strength, scanner upgrade and manufacturer. Neuroimage 32, 180–194 (2006).
pubmed: 16651008
doi: 10.1016/j.neuroimage.2006.02.051
Reuter, M., Schmansky, N. J., Rosas, H. D. & Fischl, B. Within-subject template estimation for unbiased longitudinal image analysis. Neuroimage 61, 1402–1418 (2012).
pubmed: 22430496
doi: 10.1016/j.neuroimage.2012.02.084
Desikan, R. S. et al. An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neuroimage 31, 968–980 (2006).
pubmed: 16530430
doi: 10.1016/j.neuroimage.2006.01.021
Cortes, C. & Vapnik, V. Support-vector networks. Mach. Learn. 20, 273–297 (1995).
doi: 10.1007/BF00994018
Cramer, J. The Origins of Logistic Regression. https://econpapers.repec.org/paper/tinwpaper/20020119.htm (2002).
Wang, J., Chen, Q. & Chen, Y. RBF Kernel Based Support Vector Machine with Universal Approximation and Its Application. In Advances in Neural Networks—ISNN 2004 (eds Yin, F.-L., Wang, J. & Guo, C.) 512–517 (Springer, 2004). https://doi.org/10.1007/978-3-540-28647-9_85 .
Fawagreh, K., Gaber, M. M. & Elyan, E. Random forests: From early developments to recent advancements. Syst. Sci. Control Eng. 2, 602–609 (2014).
doi: 10.1080/21642583.2014.956265
Lebedev, A. V. et al. Random Forest ensembles for detection and prediction of Alzheimer’s disease with a good between-cohort robustness. Neuroimage Clin. 6, 115–125 (2014).
pubmed: 25379423
pmcid: 4215532
doi: 10.1016/j.nicl.2014.08.023
Tharwat, A., Gaber, T., Ibrahim, A. & Hassanien, A. E. Linear discriminant analysis: A detailed tutorial. AI Commun. 30, 169–190 (2017).
doi: 10.3233/AIC-170729
Schulz, M.-A. et al. Different scaling of linear models and deep learning in UKBiobank brain images versus machine-learning datasets. Nat. Commun. 11, 1–15 (2020).
doi: 10.1038/s41467-020-18037-z
Kohoutová, L. et al. Toward a unified framework for interpreting machine-learning models in neuroimaging. Nat. Protoc. 15, 1399–1435 (2020).
pubmed: 32203486
pmcid: 9533325
doi: 10.1038/s41596-019-0289-5
Wilcox, R. R. Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy. Vol. xiii, 258 (Springer-Verlag Publishing, 2001). https://doi.org/10.1007/978-1-4757-3522-2
Pinaya, W. H. L. et al. Using normative modelling to detect disease progression in mild cognitive impairment and Alzheimer’s disease in a cross-sectional multi-cohort study. Sci. Rep. 11, 15746 (2021).
pubmed: 34344910
pmcid: 8333350
doi: 10.1038/s41598-021-95098-0