Deep learning based joint fusion approach to exploit anatomical and functional brain information in autism spectrum disorders.
ABIDE
Autism spectrum disorders
Deep Learning
Functional connectivity
Multi-modal machine learning
Structural MRI
Journal
Brain informatics
ISSN: 2198-4018
Titre abrégé: Brain Inform
Pays: Germany
ID NLM: 101673751
Informations de publication
Date de publication:
09 Jan 2024
09 Jan 2024
Historique:
received:
20
09
2023
accepted:
20
12
2023
medline:
9
1
2024
pubmed:
9
1
2024
entrez:
9
1
2024
Statut:
epublish
Résumé
The integration of the information encoded in multiparametric MRI images can enhance the performance of machine-learning classifiers. In this study, we investigate whether the combination of structural and functional MRI might improve the performances of a deep learning (DL) model trained to discriminate subjects with Autism Spectrum Disorders (ASD) with respect to typically developing controls (TD). We analyzed both structural and functional MRI brain scans publicly available within the ABIDE I and II data collections. We considered 1383 male subjects with age between 5 and 40 years, including 680 subjects with ASD and 703 TD from 35 different acquisition sites. We extracted morphometric and functional brain features from MRI scans with the Freesurfer and the CPAC analysis packages, respectively. Then, due to the multisite nature of the dataset, we implemented a data harmonization protocol. The ASD vs. TD classification was carried out with a multiple-input DL model, consisting in a neural network which generates a fixed-length feature representation of the data of each modality (FR-NN), and a Dense Neural Network for classification (C-NN). Specifically, we implemented a joint fusion approach to multiple source data integration. The main advantage of the latter is that the loss is propagated back to the FR-NN during the training, thus creating informative feature representations for each data modality. Then, a C-NN, with a number of layers and neurons per layer to be optimized during the model training, performs the ASD-TD discrimination. The performance was evaluated by computing the Area under the Receiver Operating Characteristic curve within a nested 10-fold cross-validation. The brain features that drive the DL classification were identified by the SHAP explainability framework. The AUC values of 0.66±0.05 and of 0.76±0.04 were obtained in the ASD vs. TD discrimination when only structural or functional features are considered, respectively. The joint fusion approach led to an AUC of 0.78±0.04. The set of structural and functional connectivity features identified as the most important for the two-class discrimination supports the idea that brain changes tend to occur in individuals with ASD in regions belonging to the Default Mode Network and to the Social Brain. Our results demonstrate that the multimodal joint fusion approach outperforms the classification results obtained with data acquired by a single MRI modality as it efficiently exploits the complementarity of structural and functional brain information.
Sections du résumé
BACKGROUND
BACKGROUND
The integration of the information encoded in multiparametric MRI images can enhance the performance of machine-learning classifiers. In this study, we investigate whether the combination of structural and functional MRI might improve the performances of a deep learning (DL) model trained to discriminate subjects with Autism Spectrum Disorders (ASD) with respect to typically developing controls (TD).
MATERIAL AND METHODS
METHODS
We analyzed both structural and functional MRI brain scans publicly available within the ABIDE I and II data collections. We considered 1383 male subjects with age between 5 and 40 years, including 680 subjects with ASD and 703 TD from 35 different acquisition sites. We extracted morphometric and functional brain features from MRI scans with the Freesurfer and the CPAC analysis packages, respectively. Then, due to the multisite nature of the dataset, we implemented a data harmonization protocol. The ASD vs. TD classification was carried out with a multiple-input DL model, consisting in a neural network which generates a fixed-length feature representation of the data of each modality (FR-NN), and a Dense Neural Network for classification (C-NN). Specifically, we implemented a joint fusion approach to multiple source data integration. The main advantage of the latter is that the loss is propagated back to the FR-NN during the training, thus creating informative feature representations for each data modality. Then, a C-NN, with a number of layers and neurons per layer to be optimized during the model training, performs the ASD-TD discrimination. The performance was evaluated by computing the Area under the Receiver Operating Characteristic curve within a nested 10-fold cross-validation. The brain features that drive the DL classification were identified by the SHAP explainability framework.
RESULTS
RESULTS
The AUC values of 0.66±0.05 and of 0.76±0.04 were obtained in the ASD vs. TD discrimination when only structural or functional features are considered, respectively. The joint fusion approach led to an AUC of 0.78±0.04. The set of structural and functional connectivity features identified as the most important for the two-class discrimination supports the idea that brain changes tend to occur in individuals with ASD in regions belonging to the Default Mode Network and to the Social Brain.
CONCLUSIONS
CONCLUSIONS
Our results demonstrate that the multimodal joint fusion approach outperforms the classification results obtained with data acquired by a single MRI modality as it efficiently exploits the complementarity of structural and functional brain information.
Identifiants
pubmed: 38194126
doi: 10.1186/s40708-023-00217-4
pii: 10.1186/s40708-023-00217-4
doi:
Types de publication
Journal Article
Langues
eng
Pagination
2Informations de copyright
© 2023. The Author(s).
Références
Abbas SQ, Chi L, Chen YPP (2023) Deepmnf: Deep multimodal neuroimaging framework for diagnosing autism spectrum disorder. Artif Intell Med 136:102475
doi: 10.1016/j.artmed.2022.102475
Acosta JN, Falcone GJ, Rajpurkar P et al (2022) Multimodal biomedical ai. Nat Med 28:1773–1784. https://doi.org/10.1038/s41591-022-01981-2
doi: 10.1038/s41591-022-01981-2
Adolphs R (2009) The social brain: neural basis of social knowledge. Ann Rev Psychol 60:693–716
doi: 10.1146/annurev.psych.60.110707.163514
Aglinskas A, Hartshorne JK, Anzellotti S (2022) Contrastive machine learning reveals the structure of neuroanatomical variation within autism. Science 376(6597):1070–1074
doi: 10.1126/science.abm2461
Akhavan Aghdam M, Sharifi A, Pedram MM (2018) Combination of rs-fmri and smri data to discriminate autism spectrum disorders in young children using deep belief network. J Digital Imag 31:895–903
doi: 10.1007/s10278-018-0093-8
Alaerts K, Swinnen SP, Wenderoth N (2016) Sex differences in autism: a resting-state fmri investigation of functional brain connectivity in males and females. Soc Cognit Affect Neurosci 11(6):1002–1016
doi: 10.1093/scan/nsw027
American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders: DSM-5, 5th edn. Autor, Washington, DC
doi: 10.1176/appi.books.9780890425596
Anderson JS, Druzgal TJ, Froehlich A et al (2011) Decreased interhemispheric functional connectivity in autism. Cereb Cortex 21(5):1134–1146
doi: 10.1093/cercor/bhq190
Anderson JS, Nielsen JA, Froehlich AL et al (2011) Functional connectivity magnetic resonance imaging classification of autism. Brain 134(12):3742–3754
doi: 10.1093/brain/awr263
Assaf M, Jagannathan K, Calhoun VD et al (2010) Abnormal functional connectivity of default mode sub-networks in autism spectrum disorder patients. Neuroimage 53(1):247–256
doi: 10.1016/j.neuroimage.2010.05.067
Belmonte MK, Allen G, Beckel-Mitchener A et al (2004) Autism and abnormal development of brain connectivity. J Neurosci 24(42):9228–9231
doi: 10.1523/JNEUROSCI.3340-04.2004
Xa Bi, Zhao J, Xu Q et al (2018) Abnormal functional connectivity of resting state network detection based on linear ica analysis in autism spectrum disorder. Front Physiol 9:475
doi: 10.3389/fphys.2018.00475
Bigler ED, Mortensen S, Neeley ES et al (2007) Superior temporal gyrus, language function, and autism. Devel Neuropsychol 31(2):217–238
doi: 10.1080/87565640701190841
Brothers L (1990) The social brain: a project for integrating primate behavior and neuropsychology in a new domain. Conc Neurosci 1:27–51
Broyd SJ, Demanuele C, Debener S et al (2009) Default-mode brain dysfunction in mental disorders: a systematic review. Neurosci Biobehav Rev 33(3):279–296
doi: 10.1016/j.neubiorev.2008.09.002
Chen H, Nomi JS, Uddin LQ et al (2017) Intrinsic functional connectivity variance and state-specific under-connectivity in autism. Human Brain Mapping 38(11):5740–5755
doi: 10.1002/hbm.23764
Chen L, Chen Y, Zheng H et al (2021) Changes in the topological organization of the default mode network in autism spectrum disorder. Brain Imag Behav 15:1058–1067
doi: 10.1007/s11682-020-00312-8
Chollet F et al (2015) Keras. https://github.com/fchollet/keras
Cohen J (1988) Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Routledge; Taylor and Francis.
Craddock C, Sikka S, Cheung B et al (2013) Towards automated analysis of connectomes: The configurable pipeline for the analysis of connectomes (c-pac). Front Neuroinform 42:10–3389
Dekhil O, Ali M, Haweel R et al (2020) A comprehensive framework for differentiating autism spectrum disorder from neurotypicals by fusing structural mri and resting state functional mri. Semin Pediat Neurol 34:100805
doi: 10.1016/j.spen.2020.100805
Di Martino A, Yan CG, Li Q et al (2014) The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Mol Psych 19:659–667. https://doi.org/10.1038/mp.2013.78
doi: 10.1038/mp.2013.78
Di Martino A, O’Connor D, Chen B et al (2017) Enhancing studies of the connectome in autism using the autism brain imaging data exchange ii. Scient Data 4:170010. https://doi.org/10.1038/sdata.2017.10
doi: 10.1038/sdata.2017.10
Feng Y, Kang X, Wang H et al (2023) The relationships between dynamic resting-state networks and social behavior in autism spectrum disorder revealed by fuzzy entropy-based temporal variability analysis of large-scale network. Cerebral Cortex 33(3):764–776
doi: 10.1093/cercor/bhac100
Ferrari E, Bosco P, Calderoni S et al (2020) Dealing with confounders and outliers in classification medical studies: The autism spectrum disorders case study. Artif Intell Med 108:101926. https://doi.org/10.1016/j.artmed.2020.101926
doi: 10.1016/j.artmed.2020.101926
Ferrari E, Retico A, Bacciu D (2020) Measuring the effects of confounders in medical supervised classification problems: the confounding index (ci). Artif Intell Med 103:101804. https://doi.org/10.1016/j.artmed.2020.101804
doi: 10.1016/j.artmed.2020.101804
Fischl B (2012) Freesurfer. NeuroImage 62:774–781. https://doi.org/10.1016/j.neuroimage.2012.01.021
doi: 10.1016/j.neuroimage.2012.01.021
Floris DL, Filho JOA, Lai MC et al (2021) Towards robust and replicable sex differences in the intrinsic brain function of autism. Mol Autism. https://doi.org/10.1186/s13229-021-00415-z
doi: 10.1186/s13229-021-00415-z
Fortin JP, Parker D, Tunç B et al (2017) Harmonization of multi-site diffusion tensor imaging data. NeuroImage 161:149–170. https://doi.org/10.1016/j.neuroimage.2017.08.047
doi: 10.1016/j.neuroimage.2017.08.047
Frith U, Frith C (2010) The social brain: allowing humans to boldly go where no other species has been. Philos Trans Royal Soci B Biol Sci 365(1537):165–176
doi: 10.1098/rstb.2009.0160
Gotts SJ, Simmons WK, Milbury LA et al (2012) Fractionation of social brain circuits in autism spectrum disorders. Brain 135(9):2711–2725
doi: 10.1093/brain/aws160
Gusnard DA, Akbudak E, Shulman GL et al (2001) Medial prefrontal cortex and self-referential mental activity: relation to a default mode of brain function. Proc Natl Acad Sci 98(7):4259–4264
doi: 10.1073/pnas.071043098
Haar S, Berman S, Behrmann M et al (2016) Anatomical abnormalities in autism? Cereb Cortex. https://doi.org/10.1093/cercor/bhu242
doi: 10.1093/cercor/bhu242
von dem Hagen EA, Stoyanova RS, Baron-Cohen S et al (2013) Reduced functional connectivity within and between ‘social’resting state networks in autism spectrum conditions. Soc Cognit Affect Neurosci 8(6):694–701
doi: 10.1093/scan/nss053
Heinsfeld AS, Franco AR, Craddock RC et al (2018) Identification of autism spectrum disorder using deep learning and the abide dataset. NeuroImage Clin 17:16–23
doi: 10.1016/j.nicl.2017.08.017
Hu F, Chen AA, Horng H et al (2023) Image harmonization: a review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization. NeuroImage. https://doi.org/10.1016/j.neuroimage.2023.120125
doi: 10.1016/j.neuroimage.2023.120125
Huang SC, Pareek A, Seyyedi S et al (2020) Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. npj Digit Med. https://doi.org/10.1038/s41746-020-00341-z
doi: 10.1038/s41746-020-00341-z
Ingalhalikar M, Shinde S, Karmarkar A et al (2021) Functional connectivity-based prediction of autism on site harmonized abide dataset. IEEE Trans Biomed Eng 68(12):3628–3637. https://doi.org/10.1109/TBME.2021.3080259
doi: 10.1109/TBME.2021.3080259
Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Bach F, Blei D (eds) Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 37. PMLR, Lille, France, pp 448–456, https://proceedings.mlr.press/v37/ioffe15.html
Johnson WE, Li C, Rabinovic A (2006) Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 8(1):118–127. https://doi.org/10.1093/biostatistics/kxj037
doi: 10.1093/biostatistics/kxj037
Just MA, Keller TA, Malave VL et al (2012) Autism as a neural systems disorder: a theory of frontal-posterior underconnectivity. Neurosci Biobehav Rev 36(4):1292–1313
doi: 10.1016/j.neubiorev.2012.02.007
Katuwal GJ, Baum SA, Cahill ND et al (2016) Divide and conquer: Sub-grouping of asd improves asd detection based on brain morphometry. PLoS ONE 11(4):1–24. https://doi.org/10.1371/journal.pone.0153331
doi: 10.1371/journal.pone.0153331
Keown CL, Shih P, Nair A et al (2013) Local functional overconnectivity in posterior brain regions is associated with symptom severity in autism spectrum disorders. Cell Rep 5(3):567–572
doi: 10.1016/j.celrep.2013.10.003
Khodatars M, Shoeibi A, Sadeghi D et al (2021) Deep learning for neuroimaging-based diagnosis and rehabilitation of autism spectrum disorder: A review. Comput Biol Med. https://doi.org/10.1016/j.compbiomed.2021.104949
doi: 10.1016/j.compbiomed.2021.104949
Klein A, Tourville J (2012) 101 labeled brain images and a consistent human cortical labeling protocol. Front Neurosci. https://doi.org/10.3389/fnins.2012.00171
doi: 10.3389/fnins.2012.00171
Le Couteur A, Haden G, Hammal D et al (2008) Diagnosing autism spectrum disorders in pre-school children using two standardised assessment instruments: the adi-r and the ados. J Autism Devel Disord 38:362–372
doi: 10.1007/s10803-007-0403-3
Lee Y, By Park, James O et al (2017) Autism spectrum disorder related functional connectivity changes in the language network in children, adolescents and adults. Front Human Neurosci 11:418
doi: 10.3389/fnhum.2017.00418
Lin CS, Ku HL, Chao HT et al (2014) Neural network of body representation differs between transsexuals and cissexuals. PLoS ONE 9(1):1–10. https://doi.org/10.1371/journal.pone.0085914
doi: 10.1371/journal.pone.0085914
Liu G, Shi L, Qiu J et al (2022) Two neuroanatomical subtypes of males with autism spectrum disorder revealed using semi-supervised machine learning. Mol Autism 13(1):1–14
doi: 10.1186/s13229-022-00489-3
Lombardo MV, Lai MC, Baron-Cohen S (2019) Big data approaches to decomposing heterogeneity across the autism spectrum. Mol Psych 24(10):1435–1450
doi: 10.1038/s41380-018-0321-0
Lord C, Rutter M, DiLavore P et al (2012) Edition (ados-2) manual (part i). Modules 1–4 (Autism Diagnostic Observation Schedule, Second)
Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. In: Guyon I, Luxburg UV, Bengio S et al (eds) Advances in Neural Information Processing Systems 30. Inc; New york, Curran Associates, pp 4765–4774
Maenner MJ, Warren Z, Williams AR et al (2023) Prevalence and characteristics of autism spectrum disorder among children aged 8 years-autism and developmental disabilities monitoring network, 11 sites, united states, 2020. MMWR Surveill Summ 72(2):1
doi: 10.15585/mmwr.ss7202a1
Marquand AF, Rezek I, Buitelaar J et al (2016) Understanding heterogeneity in clinical cohorts using normative models: beyond case-control studies. Biol Psych 80(7):552–561
doi: 10.1016/j.biopsych.2015.12.023
Mars RB, Neubert FX, Noonan MP et al (2012) On the relationship between the “default mode network’’ and the “social brain’’. Front Human Neurosci 6:189
doi: 10.3389/fnhum.2012.00189
Maximo JO, Keown CL, Nair A et al (2013) Approaches to local connectivity in autism using resting state functional connectivity mri. Front Human Neurosci 7:605
doi: 10.3389/fnhum.2013.00605
Moridian P, Ghassemi N, Jafari M et al (2022) Automatic autism spectrum disorder detection using artificial intelligence methods with mri neuroimaging: A review. Front Mol Neurosci. https://doi.org/10.3389/fnmol.2022.999605
doi: 10.3389/fnmol.2022.999605
Mueller S, Keeser D, Samson AC et al (2013) Convergent findings of altered functional and structural brain connectivity in individuals with high functioning autism: a multimodal mri study. PloS ONE 8(6):e67329
doi: 10.1371/journal.pone.0067329
Nielsen JA, Zielinski BA, Fletcher PT et al (2014) Abnormal lateralization of functional connectivity between language and default mode regions in autism. Mol Autism 5(1):1–11
doi: 10.1186/2040-2392-5-8
Niu K, Guo J, Pan Y et al (2020) Multichannel deep attention neural networks for the classification of autism spectrum disorder using neuroimaging and personal characteristic data. Complexity 2020:1–9
Okamoto N, Akama H (2021) Extended invariant information clustering is effective for leave-one-site-out cross-validation in resting state functional connectivity modeling. Front Neuroinform. https://doi.org/10.3389/fninf.2021.709179
doi: 10.3389/fninf.2021.709179
Patriquin MA, DeRamus T, Libero LE et al (2016) Neuroanatomical and neurofunctional markers of social cognition in autism spectrum disorder. Human Brain Mapping 37(11):3957–3978
doi: 10.1002/hbm.23288
Pereira AM, Campos BM, Coan AC et al (2018) Differences in cortical structure and functional mri connectivity in high functioning autism. Front Neurol 9:539
doi: 10.3389/fneur.2018.00539
Pomponio R, Erus G, Habes M et al (2020) Harmonization of large mri datasets for the analysis of brain imaging patterns throughout the lifespan. NeuroImage 208:116450. https://doi.org/10.1016/j.neuroimage.2019.116450
doi: 10.1016/j.neuroimage.2019.116450
Qiu S, Chang GH, Panagia M et al (2018) Fusion of deep learning models of mri scans, mini-mental state examination, and logical memory test enhances diagnosis of mild cognitive impairment. Alzheimer’s Demen Diagn Assess Dis Monit 10:737–749
doi: 10.1016/j.dadm.2018.08.013
Raichle ME, MacLeod AM, Snyder AZ et al (2001) A default mode of brain function. Proc Natl Acad Sci 98(2):676–682
doi: 10.1073/pnas.98.2.676
Rakić M, Cabezas M, Kushibar K et al (2020) Improving the detection of autism spectrum disorder by combining structural and functional mri information. NeuroImage Clin 25:102181
doi: 10.1016/j.nicl.2020.102181
Reardon AM, Li K, Hu XP (2021) Improving between-group effect size for multi-site functional connectivity data via site-wise de-meaning. Front Comput Neurosci. https://doi.org/10.3389/fncom.2021.762781
doi: 10.3389/fncom.2021.762781
Rudie JD, Brown J, Beck-Pancer D et al (2013) Altered functional and structural brain network organization in autism. NeuroImage Clin 2:79–94
doi: 10.1016/j.nicl.2012.11.006
Rutherford S, Kia SM, Wolfers T et al (2022) The normative modeling framework for computational psychiatry. Nat Prot 17(7):1711–1734
doi: 10.1038/s41596-022-00696-5
Rutter M, Le Couteur A, Lord C et al (2003) Autism diagnostic interview-revised. Los Angeles CA Western Psychol Serv 29(2003):30
Saponaro S, Giuliano A, Bellotti R et al (2022) Multi-site harmonization of mri data uncovers machine-learning discrimination capability in barely separable populations: An example from the abide dataset. NeuroImage Clin 35:103082. https://doi.org/10.1016/j.nicl.2022.103082
doi: 10.1016/j.nicl.2022.103082
Segal A, Parkes L, Aquino K et al (2023) Regional, circuit and network heterogeneity of brain abnormalities in psychiatric disorders. Nat Neurosci 26(9):1613–1629
doi: 10.1038/s41593-023-01404-6
Shan X, Uddin LQ, Xiao J et al (2022) Mapping the heterogeneous brain structural phenotype of autism spectrum disorder using the normative model. Biol Psych 91(11):967–976
doi: 10.1016/j.biopsych.2022.01.011
Spasov SE, Passamonti L, Duggento A et al (2018) A multi-modal convolutional neural network framework for the prediction of alzheimer’s disease. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, pp 1271–1274
Srivastava N, Hinton G, Krizhevsky A et al (2014) Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(56):1929–1958
Vargason T, Grivas G, Hollowood-Jones KL et al (2020) Towards a multivariate biomarker-based diagnosis of autism spectrum disorder: Review and discussion of recent advancements. Semin Pediat Neurol 34:100803. https://doi.org/10.1016/j.spen.2020.100803
doi: 10.1016/j.spen.2020.100803
Venugopalan J, Tong L, Hassanzadeh HR et al (2021) Multimodal deep learning models for early detection of alzheimer’s disease stage. Scient Rep 11(1):3254
doi: 10.1038/s41598-020-74399-w
Weng SJ, Wiggins JL, Peltier SJ et al (2010) Alterations of resting state functional connectivity in the default network in adolescents with autism spectrum disorders. Brain Res 1313:202–214
doi: 10.1016/j.brainres.2009.11.057
Xie Y, Xu Z, Xia M et al (2022) Alterations in connectome dynamics in autism spectrum disorder: A harmonized mega- and meta-analysis study using the autism brain imaging data exchange dataset. Biol Psych 91(11):945–955. https://doi.org/10.1016/j.biopsych.2021.12.004
doi: 10.1016/j.biopsych.2021.12.004
Yala A, Lehman C, Schuster T et al (2019) A deep learning mammography-based model for improved breast cancer risk prediction. Radiology 292(1):60–66
doi: 10.1148/radiol.2019182716
Yerys BE, Gordon EM, Abrams DN et al (2015) Default mode network segregation and social deficits in autism spectrum disorder: Evidence from non-medicated children. NeuroImage Clin 9:223–232
doi: 10.1016/j.nicl.2015.07.018
Yoo Y, Tang LY, Li DK et al (2019) Deep learning of brain lesion patterns and user-defined clinical and mri features for predicting conversion to multiple sclerosis from clinically isolated syndrome. Comp Meth Biomech Biomed Eng Imag Visualiz 7(3):250–259
doi: 10.1080/21681163.2017.1356750
Zabihi M, Floris DL, Kia SM et al (2020) Fractionating autism based on neuroanatomical normative modeling. Transl Psych 10(1):384
doi: 10.1038/s41398-020-01057-0
Zhang J, Fang S, Yao Y et al (2023) Parsing the heterogeneity of brain-symptom associations in autism spectrum disorder via random forest with homogeneous canonical correlation. J Affect Disord 335:36–43
doi: 10.1016/j.jad.2023.04.102