Lung pneumonia severity scoring in chest X-ray images using transformers.
Automatic prediction
Chest X-ray
Severity quantification
Vision transformer
Journal
Medical & biological engineering & computing
ISSN: 1741-0444
Titre abrégé: Med Biol Eng Comput
Pays: United States
ID NLM: 7704869
Informations de publication
Date de publication:
09 Apr 2024
09 Apr 2024
Historique:
received:
30
10
2023
accepted:
24
02
2024
medline:
9
4
2024
pubmed:
9
4
2024
entrez:
8
4
2024
Statut:
aheadofprint
Résumé
To create robust and adaptable methods for lung pneumonia diagnosis and the assessment of its severity using chest X-rays (CXR), access to well-curated, extensive datasets is crucial. Many current severity quantification approaches require resource-intensive training for optimal results. Healthcare practitioners require efficient computational tools to swiftly identify COVID-19 cases and predict the severity of the condition. In this research, we introduce a novel image augmentation scheme as well as a neural network model founded on Vision Transformers (ViT) with a small number of trainable parameters for quantifying COVID-19 severity and other lung diseases. Our method, named Vision Transformer Regressor Infection Prediction (ViTReg-IP), leverages a ViT architecture and a regression head. To assess the model's adaptability, we evaluate its performance on diverse chest radiograph datasets from various open sources. We conduct a comparative analysis against several competing deep learning methods. Our results achieved a minimum Mean Absolute Error (MAE) of 0.569 and 0.512 and a maximum Pearson Correlation Coefficient (PC) of 0.923 and 0.855 for the geographic extent score and the lung opacity score, respectively, when the CXRs from the RALO dataset were used in training. The experimental results reveal that our model delivers exceptional performance in severity quantification while maintaining robust generalizability, all with relatively modest computational requirements. The source codes used in our work are publicly available at https://github.com/bouthainas/ViTReg-IP .
Identifiants
pubmed: 38589723
doi: 10.1007/s11517-024-03066-3
pii: 10.1007/s11517-024-03066-3
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : Agencia Estatal de Investigación
ID : PID2021-126701OB-I00
Informations de copyright
© 2024. The Author(s).
Références
Organization WH (2020) The true death toll of COVID-19: estimating global excess mortality. https://www.who.int/data/stories/the-true-death-toll-of-covid-19-estimating-global-excess-mortality
Ng M-Y, Lee EY, Yang J, Yang F, Li X, Wang H, Lui MM-s, Lo CS-Y, Leung B, Khong P-L et al (2020) Imaging profile of the COVID-19 infection: radiologic findings and literature review. Radiology: Cardiothoracic Imaging 2:(1)
Yoon SH, Lee KH, Kim JY, Lee YK, Ko H, Kim KH, Park CM, Kim Y-H (2020) Chest radiographic and CT findings of the 2019 novel coronavirus disease (COVID-19): analysis of nine patients treated in Korea. Korean J Radiol 21(4):494–500
doi: 10.3348/kjr.2020.0132
pubmed: 32100485
pmcid: 7082662
Guan W-j, Ni Z-y, Hu Y, Liang W-h, Ou C-q, He J-x, Liu L, Shan H, Lei C-l, Hui DS (2020) Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med 382(18):1708–1720
doi: 10.1056/NEJMoa2002032
pubmed: 32109013
Toussie D, Voutsinas N, Finkelstein M, Cedillo MA, Manna S, Maron SZ, Jacobi A, Chung M, Bernheim A, Eber C (2020) Clinical and chest radiography features determine patient outcomes in young and middle-aged adults with COVID-19. Radiol 297(1):197. pmid:32407255
Jacobi A, Chung M, Bernheim A, Eber C (2020) Portable chest X-ray in coronavirus disease-19 (COVID-19): a pictorial review. Clinical imaging 64:5–42
doi: 10.1016/j.clinimag.2020.04.001
Rubin GD, Ryerson CJ, Haramati LB, Sverzellati N, Kanne JP, Raoof S, Schluger NW, Volpi A, Yim J-J, Martin IB et al (2020) The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner society. Radiol 296(1):172–180
doi: 10.1148/radiol.2020201365
Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, Zhang L, Fan G, Xu J, Gu X et al (2020) Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The lancet 395(10223):497–506
doi: 10.1016/S0140-6736(20)30183-5
Wong HYF, Lam HYS, Fong AH-T, Leung ST, Chin TW-Y, Lo CSY, Lui MM-S, Lee JCY, Chiu KW-H, Chung TW-H et al (2020) Frequency and distribution of chest radiographic findings in patients positive for COVID-19. Radiol 296(2):72–78
doi: 10.1148/radiol.2020201160
Gietema HA, Zelis N, Nobel JM, Lambriks LJ, Van Alphen LB, Oude Lashof AM, Wildberger JE, Nelissen IC, Stassen PM (2020) CT in relation to RT-PCR in diagnosing COVID-19 in the Netherlands: a prospective study. PloS one 15(7):0235844
doi: 10.1371/journal.pone.0235844
Dennie C, Hague C, Lim RS, Manos D, Memauri BF, Nguyen ET, Taylor J (2020) Canadian Society of Thoracic Radiology/Canadian Association of Radiologists consensus statement regarding chest imaging in suspected and confirmed COVID-19. Can Assoc Radiol J 71(4):470–481
doi: 10.1177/0846537120924606
pubmed: 32380844
Nair A, Rodrigues J, Hare S, Edey A, Devaraj A, Jacob J, Johnstone A, McStay R, Denton E, Robinson G (2020) A British society of thoracic imaging statement: considerations in designing local imaging diagnostic algorithms for the COVID-19 pandemic. Clin Radiol 75(5):329–334
doi: 10.1016/j.crad.2020.03.008
pubmed: 32265036
pmcid: 7128118
Kong W, Agarwal PP (2020) Chest imaging appearance of COVID-19 infection. Radiology: Cardiothoracic Imaging 2:(1)
Zhang X, Han L, Sobeih T, Han L, Dempsey N, Lechareas S, Tridente A, Chen H, White S, Zhang D (2023) CXR-Net: a multitask deep learning network for explainable and accurate diagnosis of COVID-19 pneumonia from chest X-ray images. IEEE J Biomed Health Inform 27(2):980–991. https://doi.org/10.1109/JBHI.2022.3220813
doi: 10.1109/JBHI.2022.3220813
pubmed: 36350854
Dornaika F, Bekhouche S, Arganda-Carreras I (2020) Robust regression with deep CNNs for facial age estimation: an empirical study. Expert Syst Appl 141
Bougourzi F, Dornaika F, Taleb-Ahmed A (2022) Deep learning based face beauty prediction via dynamic robust losses and ensemble regression. Knowl-Based Syst 242:108246
doi: 10.1016/j.knosys.2022.108246
Liu P, Fu B, Yang SX, Deng L, Zhong X, Zheng H (2021) Optimizing survival analysis of XGBoost for ties to predict disease progression of breast cancer. IEEE Trans Biomed Eng 68(1):148–160. https://doi.org/10.1109/TBME.2020.2993278
doi: 10.1109/TBME.2020.2993278
pubmed: 32406821
Li J, Chen J, Tang Y, Wang C, Landman BA, Zhou SK (2023) Transforming medical imaging with transformers? A comparative review of key properties, current progresses, and future perspectives. Med Image Anal 85
Ting DSW, Carin L, Dzau V, Wong TY (2020) Digital technology and COVID-19. Nature Med 26(4):459–461
doi: 10.1038/s41591-020-0824-5
pubmed: 32284618
Latif S, Usman M, Manzoor S, Iqbal W, Qadir J, Tyson G, Castro I, Razi A, Boulos MNK, Weller A et al (2020) Leveraging data science to combat COVID-19: a comprehensive review. IEEE Trans Artif Intell 1(1):85–103
doi: 10.1109/TAI.2020.3020521
pubmed: 37982070
Shi F, Wang J, Shi J, Wu Z, Wang Q, Tang Z, He K, Shi Y, Shen D (2020) Review of artificial intelligence techniques in imaging data acquisition, segmentation, and diagnosis for COVID-19. IEEE Rev Biomed Eng 14:4–15
doi: 10.1109/RBME.2020.2987975
Islam M, Poly TN, Alsinglawi B, Lin MC, Hsu M-H, Li Y-CJ et al (2021) A state-of-the-art survey on artificial intelligence to fight COVID-19. J Clin Med 10(9):1961
doi: 10.3390/jcm10091961
pubmed: 34063302
pmcid: 8124542
Oh Y, Park S, Ye JC (2020) Deep learning COVID-19 features on CXR using limited training data sets. IEEE Trans Med Imaging 39(8):2688–2700
doi: 10.1109/TMI.2020.2993291
pubmed: 32396075
Hammoudi K, Benhabiles H, Melkemi M, Dornaika F, Arganda-Carreras I, Collard D, Scherpereel A (2021) Deep learning on chest X-ray images to detect and evaluate pneumonia cases at the era of COVID-19. J Med Syst 45(7):1–10
doi: 10.1007/s10916-021-01745-4
Sunnetci KM, Alkan A (2023) Biphasic majority voting-based comparative COVID-19 diagnosis using chest X-ray images. Expert Syst Appl 216
Zhu Y, Wang S, Wang S, Wu Q, Wang L, Li H, Wang M, Niu M, Zha Y, Tian J (2021) Mix contrast for COVID-19 mild-to-critical prediction. IEEE Trans Biomed Eng 68(12):3725–3736. https://doi.org/10.1109/TBME.2021.3085576
doi: 10.1109/TBME.2021.3085576
pubmed: 34061732
Dhere A, Sivaswamy J (2022) COVID detection from chest X-ray images using multi-scale attention. IEEE J Biomed Health Inform 26(4):1496–1505. https://doi.org/10.1109/JBHI.2022.3151171
doi: 10.1109/JBHI.2022.3151171
pubmed: 35157603
Joshi AM, Nayak DR (2022) MFL-Net: an efficient lightweight multi-scale feature learning CNN for COVID-19 diagnosis from CT images. IEEE J Biomed Health Inform 26(11):5355–5363. https://doi.org/10.1109/JBHI.2022.3196489
doi: 10.1109/JBHI.2022.3196489
pubmed: 35981061
Irmak E (2021) COVID-19 disease severity assessment using CNN model. IET Image Processing 15(8):1814–1824
doi: 10.1049/ipr2.12153
pubmed: 34230837
pmcid: 8251482
Orsi MA, Oliva G, Toluian T, Pittino CV, Panzeri M, Cellina M (2020) Feasibility, reproducibility, and clinical validity of a quantitative chest X-ray assessment for COVID-19. Am J Trop Med Hyg 103(2):822
doi: 10.4269/ajtmh.20-0535
pubmed: 32618262
pmcid: 7410416
Liang W, Liang H, Ou L, Chen B, Chen A, Li C, Li Y, Guan W, Sang L, Lu J et al (2020) Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Internal Med 180(8):1081–1089
doi: 10.1001/jamainternmed.2020.2033
Liang W, Yao J, Chen A, Lv Q, Zanin M, Liu J, Wong S, Li Y, Lu J, Liang H et al (2020) Early triage of critically ill COVID-19 patients using deep learning. Nature Comm 11(1):1–7
doi: 10.1038/s41467-020-17280-8
Colombi D, Bodini FC, Petrini M, Maffi G, Morelli N, Milanese G, Silva M, Sverzellati N, Michieletti E (2020) Well-aerated lung on admitting chest CT to predict adverse outcome in COVID-19 pneumonia. Radiol
Wang L, Lin ZQ, Wong A (2020) COVID-Net: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images. Sci Rep 10(1):1–12
Cohen JP, Dao L, Roth K, Morrison P, Bengio Y, Abbasi AF, Shen B, Mahsa HK, Ghassemi M, Li H et al (2020) Predicting COVID-19 pneumonia severity on chest X-ray with deep learning. Cureus 12:(7)
Ridley E (2020) AI can assess COVID-19 severity on chest X-rays. https://www.auntminnie.com/index.aspx?sec=ser &sub=def &pag=dis &ItemID=129674
Li MD, Arun NT, Gidwani M, Chang K, Deng F, Little BP, Mendoza DP, Lang M, Lee SI, O’Shea A et al (2020) Automated assessment and tracking of COVID-19 pulmonary disease severity on chest radiographs using convolutional siamese neural networks. Radiol Artif Intell 2(4)
Li MD, Arun NT, Aggarwal M, Gupta S, Singh P, Little BP, Mendoza DP, Corradi GC, Takahashi MS, Ferraciolli SF et al (2022) Multi-population generalizability of a deep learning-based chest radiograph severity score for COVID-19. Medicine 101:(29)
Frid-Adar M, Amer R, Gozes O, Nassar J, Greenspan H (2021) COVID-19 in CXR: from detection and severity scoring to patient disease monitoring. IEEE J Biomed Health Inform 25(6):1892–1903
doi: 10.1109/JBHI.2021.3069169
pubmed: 33769939
Wong A, Lin Z, Wang L, Chung A, Shen B, Abbasi A, Hoshmand-Kochi M, Duong T (2021) Towards computer-aided severity assessment via deep neural networks for geographic and opacity extent scoring of SARS-CoV-2 chest X-rays. Sci Rep 11(1):1–8
doi: 10.1038/s41598-021-88538-4
Kwon YJ, Toussie D, Finkelstein M, Cedillo MA, Maron SZ, Manna S, Voutsinas N, Eber C, Jacobi A, Bernheim A et al (2020) Combining initial radiographs and clinical variables improves deep learning prognostication in patients with COVID-19 from the emergency department. Radiol Artif Intell 3(2)
Signoroni A, Savardi M, Benini S, Adami N, Leonardi R, Gibellini P, Vaccher F, Ravanelli M, Borghesi A, Maroldi R et al (2021) BS-NET: learning COVID-19 pneumonia severity on a large chest X-ray dataset. Med Image Anal 71
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review. Comput Intell Neurosci 2018
Willemink MJ, Koszek WA, Hardell C, Wu J, Fleischmann D, Harvey H, Folio LR, Summers RM, Rubin DL, Lungren MP (2020) Preparing medical imaging data for machine learning. Radiol 295(1):4–15
doi: 10.1148/radiol.2020192224
Ridzuan M, Bawazir AA, Navarette IG, Almakky I, Yaqub M (2022) Challenges in COVID-19 chest X-ray classification: problematic data or ineffective approaches? arXiv preprint arXiv:2201.06052
Yun S, Han D, Oh SJ, Chun S, Choe J, Yoo Y (2019) CutMix: regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6023–6032
Slika B, Dornaika F, Hammoudi K, Hoang V (2023) Automatic quantification of lung infection severity in chest X-ray images. In: 2023 IEEE Statistical Signal Processing Workshop (SSP), pp 418–422. IEEE
Savardi M, Signoroni A, Benini S, Adami N, Farina D (2021) Brixia score COVID-19 dataset. OSF. osf.io/2j8d5
Danilov V, Proutski A, Kirpich A, Litmanovich D, Gankin Y (2022) Dataset for COVID-19 segmentation and severity scoring. Mendeley Data. https://data.mendeley.com/datasets/36fjrg9s69/2
Cohen JP, Morrison P, Dao L (2020) COVID-19 image data collection. arXiv preprint arXiv:2003.11597
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778
Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 10012–10022
Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1251–1258
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1–9
Howard A, Sandler M, Chu G, Chen L-C, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, Le QV, Adam H (2019) Searching for MobileNetV3. https://doi.org/10.48550/ARXIV.1905.02244
Degerli A, Ahishali M, Kiranyaz S, Chowdhury MEH, Gabbouj M (2021) Reliable COVID-19 detection using chest X-ray images. In: 2021 IEEE International Conference on Image Processing (ICIP), pp 185–189. https://doi.org/10.1109/ICIP42928.2021.9506442
DeVries T, Taylor GW (2017) Improved regularization of convolutional neural networks with cutout. https://doi.org/10.48550/ARXIV.1708.04552
Walawalkar D, Shen Z, Liu Z, Savvides M (2020) Attentive CutMix: an enhanced data augmentation approach for deep learning based image classification. arXiv (2020). https://doi.org/10.48550/ARXIV.2003.13048
Zhang H, Cisse M, Dauphin YN, Lopez-Paz D (2017) Mixup: beyond empirical risk minimization. https://doi.org/10.48550/ARXIV.1710.09412
Chen P, Liu S, Zhao H, Jia J (2020) GridMask data augmentation. https://doi.org/10.48550/ARXIV.2001.04086 abs/2001.04086
Hammoudi K, Cabani A, Slika B, Benhabiles H, Dornaika F, Melkemi M (2023) SuperpixelGridMasks data augmentation: application to precision health and other real-world data. Journal of Healthcare Informatics Research, pp 1–19
Kim J-H, Choo W, Song HO (2020) Puzzle mix: exploiting saliency and local statistics for optimal mixup. https://doi.org/10.48550/ARXIV.2009.06962
Chen J-N, Sun S, He J, Torr P, Yuille A, Bai S (2022) Transmix: attend to mix for vision transformers. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Buslaev A, Iglovikov VI, Khvedchenya E, Parinov A, Druzhinin M, Kalinin AA (2020) Albumentations: fast and flexible image augmentations. Information 11(2):125
doi: 10.3390/info11020125
Mansoor A, Cerrolaza JJ, Perez G, Biggs E, Okada K, Nino G, Linguraru MG (2020) A generic approach to lung field segmentation from chest radiographs using deep space and shape learning. IEEE Transactions on Biomedical Engineering 67(4):1206–1220. https://doi.org/10.1109/TBME.2019.2933508
doi: 10.1109/TBME.2019.2933508
pubmed: 31425015
Fan T, Wang G, Li Y, Wang H (2020) Ma-Net: a multi-scale attention network for liver and tumor segmentation. IEEE Access 8:179656–179665. https://doi.org/10.1109/ACCESS.2020.3025372
doi: 10.1109/ACCESS.2020.3025372
Li H, Xiong P, An J, Wang L (2018) Pyramid attention network for semantic segmentation. https://doi.org/10.48550/ARXIV.1805.10180
Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 234–241. Springer