Impact of metadata in multimodal classification of bone tumours.
Bone neoplasm
Classification
Deep learning
Metadata
Radiography
Journal
BMC musculoskeletal disorders
ISSN: 1471-2474
Titre abrégé: BMC Musculoskelet Disord
Pays: England
ID NLM: 100968565
Informations de publication
Date de publication:
19 Oct 2024
19 Oct 2024
Historique:
received:
08
07
2024
accepted:
08
10
2024
medline:
20
10
2024
pubmed:
20
10
2024
entrez:
19
10
2024
Statut:
epublish
Résumé
The accurate classification of bone tumours is crucial for guiding clinical decisions regarding treatment and follow-up. However, differentiating between various tumour types is challenging due to the rarity of certain entities, high intra-class variability, and limited training data in clinical practice. This study proposes a multimodal deep learning model that integrates clinical metadata and X-ray imaging to improve the classification of primary bone tumours. The dataset comprises 1,785 radiographs from 804 patients collected between 2000 and 2020, including metadata such as age, affected bone site, tumour position, and gender. Ten tumour types were selected, with histopathology or tumour board decisions serving as the reference standard. Our model is based on the NesT image classification model and a multilayer perceptron with a joint fusion architecture. Descriptive statistics included incidence and percentage ratios for discrete parameters, and mean, standard deviation, median, and interquartile range for continuous parameters. The mean age of the patients was 33.62 ± 18.60 years, with 54.73% being male. Our multimodal deep learning model achieved 69.7% accuracy in classifying primary bone tumours, outperforming the Vision Transformer model by five percentage points. SHAP values indicated that age had the most substantial influence among the considered metadata. The joint fusion approach developed in this study, integrating clinical metadata and imaging data, outperformed state-of-the-art models in classifying primary bone tumours. The use of SHAP values provided insights into the impact of different metadata on the model's performance, highlighting the significant role of age. This approach has potential implications for improving diagnostic accuracy and understanding the influence of clinical factors in tumour classification.
Identifiants
pubmed: 39427131
doi: 10.1186/s12891-024-07934-9
pii: 10.1186/s12891-024-07934-9
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
822Informations de copyright
© 2024. The Author(s).
Références
Hinterwimmer F, Consalvo S, Neumann J, Rueckert D, von Eisenhart-Rothe R, Burgkart R. Applications of machine learning for imaging-driven diagnosis of musculoskeletal malignancies—a scoping review. Eur Radiol. 2022;32(10):7173–84.
doi: 10.1007/s00330-022-08981-3
pubmed: 35852574
pmcid: 9474640
Picci P, Manfrini M, Donati DM, Gambarotti M, Righi A, Vanel D, et al. Diagnosis of Musculoskeletal tumors and Tumor-like conditions: clinical, radiological and histological correlations. -the Rizzoli Case Archive: Springer; 2020.
doi: 10.1007/978-3-030-29676-6
Grimer RJ, Briggs TW. Earlier diagnosis of bone and soft-tissue tumours. J bone Joint Surg Br Volume. 2010;92(11):1489–92.
doi: 10.1302/0301-620X.92B11.24326
Grimer RJ, Carter SR, Pynsent PB. The cost-effectiveness of limb salvage for bone tumours. J bone Joint Surg Br Volume. 1997;79(4):558–61.
doi: 10.1302/0301-620X.79B4.0790558
Rechl H, Kirchhoff C, Wortler K, Lenze U, Topfer A, von Eisenhart-Rothe R. [Diagnosis of malignant bone and soft tissue tumors]. Der Orthopade. 2011;40(10):931–41. quiz 42 – 3.
doi: 10.1007/s00132-011-1821-7
pubmed: 21874363
Clark MA, Thomas JM. Delay in referral to a specialist soft-tissue sarcoma unit. Eur J Surg Oncol. 2005;31(4):443–8.
doi: 10.1016/j.ejso.2004.11.016
pubmed: 15837054
Gaume M, Chevret S, Campagna R, Larousserie F, Biau D. The appropriate and sequential value of standard radiograph, computed tomography and magnetic resonance imaging to characterize a bone tumor. Sci Rep. 2022;12(1):1–9.
doi: 10.1038/s41598-022-10218-8
Salom M, Chiari C, Alessandri JMG, Willegger M, Windhager R, Sanpera I. Diagnosis and staging of malignant bone tumours in children: what is due and what is new? J Child Orthop. 2021;15(4):312–21.
doi: 10.1302/1863-2548.15.210107
pubmed: 34476020
pmcid: 8381400
Kindblom LG. Bone tumors: epidemiology, classification, pathology. Imaging of bone tumors and tumor-like lesions: techniques and applications. 2009:1–15.
Kharat AT, Singhal S. A peek into the future of radiology using big data applications. Indian J Radiol Imaging. 2017;27(2):241–8.
pubmed: 28744087
pmcid: 5510324
Naguib SM, Kassem MA, Hamza HM, Fouda MM, Saleh MK, Hosny KM. Automated system for classifying uni-bicompartmental knee osteoarthritis by using redefined residual learning with convolutional neural network. Heliyon. 2024;10(10).
Hosny KM, Said W, Elmezain M, Kassem MA. Explainable deep inherent learning for multi-classes skin lesion classification. Appl Soft Comput. 2024;159:111624.
doi: 10.1016/j.asoc.2024.111624
Kassem MA, Naguib SM, Hamza HM, Fouda MM, Saleh MK, Hosny KM. Explainable transfer learning-based Deep Learning Model for Pelvis fracture detection. Int J Intell Syst. 2023;2023(1):3281998.
Breden S, Hinterwimmer F, Consalvo S, Neumann J, Knebel C, von Eisenhart-Rothe R, et al. Deep learning-based detection of bone tumors around the knee in X-rays of children. J Clin Med. 2023;12(18):5960.
doi: 10.3390/jcm12185960
pubmed: 37762901
pmcid: 10531620
Savage N. How AI is improving cancer diagnostics. Nature. 2020;579:S14.
doi: 10.1038/d41586-020-00847-2
pubmed: 32214265
Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med. 2022;28(1):31–8.
doi: 10.1038/s41591-021-01614-0
pubmed: 35058619
Moch H. Soft Tissue and Bone Tumours WHO Classification of Tumours/Volume 3. WHO Classification of Tumours. 2020;3.
Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the reporting of Observational studies in Epidemiology (STROBE): explanation and elaboration. Epidemiology. 2007;18(6):805–35.
doi: 10.1097/EDE.0b013e3181577511
pubmed: 18049195
Mongan J, Moy L, Kahn CE Jr. Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiological Society of North America; 2020. p. e200029.
He K, Zhang X, Ren S, Sun J, editors. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition; 2016.
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. 2014 Sep 4.
Bloier M, Hinterwimmer F, Breden S, Consalvo S, Neumann J, Wilhelm N, et al. editors. Detection and segmentation of heterogeneous bone tumours in limited radiographs. Current directions in Biomedical Engineering. De Gruyter; 2022.
Huang S-C, Pareek A, Seyyedi S, Banerjee I, Lungren MP. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit Med. 2020;3(1):1–9.
doi: 10.1038/s41746-020-00341-z
Chen T, Guestrin C, XGBoost:. A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; San Francisco, California, USA: Association for Computing Machinery; 2016. pp. 785–94.
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770-778).
Zhang Z, Zhang H, Zhao L, Chen T, Arik S, Pfister T, editors. Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. Proceedings of the AAAI Conference on Artificial Intelligence; 2022.
Raghu M, Unterthiner T, Kornblith S, Zhang C, Dosovitskiy A. Do vision transformers see like convolutional neural networks? Advances. Neural Inform Process Syst. 2021;34:12116–28.
Li Z, Kamnitsas K, Glocker B. Analyzing overfitting under class imbalance in neural networks for image segmentation. IEEE Trans Med Imaging. 2020;40(3):1065–77.
doi: 10.1109/TMI.2020.3046692
Samek W. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296. 2017.
Lundberg S. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874. 2017.
Yu AC, Mohajer B, Eng J. External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiology: Artif Intell. 2022;4(3):e210064.
Park SH, Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology. 2018;286(3):800–9.
doi: 10.1148/radiol.2017171920
pubmed: 29309734
Consalvo S, Hinterwimmer F, Neumann J, Steinborn M, Salzmann M, Seidl F, et al. Two-phase deep learning algorithm for detection and differentiation of Ewing Sarcoma and Acute Osteomyelitis in paediatric radiographs. Anticancer Res. 2022;42(9):4371–80.
doi: 10.21873/anticanres.15937
pubmed: 36039445
von Schacky CE, Wilhelm NJ, Schäfer VS, Leonhardt Y, Gassert FG, Foreman SC, et al. Multitask deep learning for segmentation and classification of primary bone tumors on radiographs. Radiology. 2021;301(2):398–406.
doi: 10.1148/radiol.2021204531
Cai G, Zhu Y, Wu Y, Jiang X, Ye J, Yang D. A multimodal transformer to fuse images and metadata for skin disease classification. The Visual Computer. 2023 Jul;39(7):2781-93.
Liu R, Pan D, Xu Y, Zeng H, He Z, Lin J, Zeng W, Wu Z, Luo Z, Qin G, Chen W. A deep learning–machine learning fusion approach for the classification of benign, malignant, and intermediate bone tumors. European Radiology. 2022 Feb;32(2):1371-83.
Xu J, Gao Y, Liu W, Huang K, Zhao S, Lu L, et al. editors. RemixFormer: A Transformer Model for Precision Skin Tumor Differential Diagnosis via Multi-modal Imaging and Non-imaging Data. Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part III; 2022: Springer.