A Deep-Learning Diagnostic Support System for the Detection of COVID-19 Using Chest Radiographs: A Multireader Validation Study.
Journal
Investigative radiology
ISSN: 1536-0210
Titre abrégé: Invest Radiol
Pays: United States
ID NLM: 0045377
Informations de publication
Date de publication:
01 06 2021
01 06 2021
Historique:
pubmed:
2
12
2020
medline:
13
5
2021
entrez:
1
12
2020
Statut:
ppublish
Résumé
Five publicly available databases comprising normal CXR, confirmed COVID-19 pneumonia cases, and other pneumonias were used. After the harmonization of the data, the training set included 7966 normal cases, 5451 with other pneumonia, and 258 CXRs with COVID-19 pneumonia, whereas in the testing data set, each category was represented by 100 cases. Eleven blinded radiologists with various levels of expertise independently read the testing data set. The data were analyzed separately with the newly proposed artificial intelligence-based system and by consultant radiologists and residents, with respect to positive predictive value (PPV), sensitivity, and F-score (harmonic mean for PPV and sensitivity). The χ2 test was used to compare the sensitivity, specificity, accuracy, PPV, and F-scores of the readers and the system. The proposed system achieved higher overall diagnostic accuracy (94.3%) than the radiologists (61.4% ± 5.3%). The radiologists reached average sensitivities for normal CXR, other type of pneumonia, and COVID-19 pneumonia of 85.0% ± 12.8%, 60.1% ± 12.2%, and 53.2% ± 11.2%, respectively, which were significantly lower than the results achieved by the algorithm (98.0%, 88.0%, and 97.0%; P < 0.00032). The mean PPVs for all 11 radiologists for the 3 categories were 82.4%, 59.0%, and 59.0% for the healthy, other pneumonia, and COVID-19 pneumonia, respectively, resulting in an F-score of 65.5% ± 12.4%, which was significantly lower than the F-score of the algorithm (94.3% ± 2.0%, P < 0.00001). When other pneumonia and COVID-19 pneumonia cases were pooled, the proposed system reached an accuracy of 95.7% for any pathology and the radiologists, 88.8%. The overall accuracy of consultants did not vary significantly compared with residents (65.0% ± 5.8% vs 67.4% ± 4.2%); however, consultants detected significantly more COVID-19 pneumonia cases (P = 0.008) and less healthy cases (P < 0.00001). The system showed robust accuracy for COVID-19 pneumonia detection on CXR and surpassed radiologists at various training levels.
Identifiants
pubmed: 33259441
pii: 00004424-202106000-00002
doi: 10.1097/RLI.0000000000000748
doi:
Types de publication
Journal Article
Multicenter Study
Validation Study
Langues
eng
Sous-ensembles de citation
IM
Pagination
348-356Informations de copyright
Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.
Déclaration de conflit d'intérêts
Conflicts of interest and sources of funding: none declared.
Références
Wen Z, Chi Y, Zhang L, et al. Coronavirus disease 2019: initial detection on chest CT in a retrospective multicenter study of 103 Chinese subjects. Radiol Cardiothorac Imaging . 2020;2:e200092.
Wong HYF, Lam HYS, Fong AH-T, et al. Frequency and distribution of chest radiographic findings in patients positive for COVID-19. Radiology . 2020;296:E72–E78.
Li K, Wu J, Wu F, et al. The clinical and chest CT features associated with severe and critical COVID-19 pneumonia. Invest Radiol . 2020;55:327–331.
Salehi S, Abedi A, Balakrishnan S, et al. Coronavirus disease 2019 (COVID-19): a systematic review of imaging findings in 919 patients. AJR Am J Roentgenol . 2020;215:87–93.
Lyu P, Liu X, Zhang R, et al. The performance of chest CT in evaluating the clinical severity of COVID-19 pneumonia: identifying critical cases based on CT characteristics. Invest Radiol . 2020;55:412–421.
Ai T, Yang Z, Hou H, et al. Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology . 2020;296:E32–E40.
Xie X, Zhong Z, Zhao W, et al. Chest CT for typical coronavirus disease 2019 (COVID-19) pneumonia: relationship to negative RT-PCR testing. Radiology . 2020;296:E41–E45.
Caruso D, Zerunian M, Polici M, et al. Chest CT features of COVID-19 in Rome, Italy. Radiology . 2020;296:E79–E85.
Choi H, Qi X, Yoon SH, et al. Extension of coronavirus disease 2019 (COVID-19) on chest CT and implications for chest radiograph interpretation. Radiol Cardiothorac Imaging . 2020;2:e200107.
Weinstock MB, Echenique A, Joshua W, et al. Chest x-ray findings in 636 ambulatory patients with COVID-19 presenting to an urgent care center: a normal chest x-ray is no guarantee. J Urgent Care Med . 2020;14:13–18.
McKinney SM, Sieniek M, Godbole V, et al. International evaluation of an AI system for breast cancer screening. Nature . 2020;577:89–94.
Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature . 2017;542:115–118.
Bullock J, Luccioni A, Pham KH, et al. Mapping the landscape of artificial intelligence applications against COVID-19. ArXiv e-prints 2003.11336. April 23, 2020. Available at: https://arxiv.org/abs/2003.11336v1 . Accessed August 5, 2020.
Shi F, Wang J, Shi J, et al. Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19. IEEE Rev Biomed Eng . 2020;PP.
Bai HX, Wang R, Xiong Z, et al. Artificial intelligence augmentation of radiologist performance in distinguishing COVID-19 from pneumonia of other origin at chest CT. Radiology . 2020;296:E156–E165.
Narin A, Kaya C, Pamuk Z. Automatic detection of coronavirus disease (COVID-19) using x-ray images and deep convolutional neural networks. ArXiv . 2020;2003:10849. Available at: https://arxiv.org/abs/2003.10849 . Accessed June, 2020.
Asif S, Wenhui Y, Jin H, et al. Classification of COVID-19 from chest x-ray images using deep convolutional neural networks. medRxiv preprint: 2020.05.01.20088211. Available at: https://doi.org/10.1101/2020.05.01.20088211 . Accessed August 2020.
Khan AI, Shah JL, Bhat MM. CoroNet: a deep neural network for detection and diagnosis of COVID-19 from chest x-ray images. Comput Methods Programs Biomed . 2020;196:105581. Accessed June 2020.
Ozturk T, Talo M, Yildirim EA, et al. Automated detection of COVID-19 cases using deep neural networks with x-ray images. Comput Biol Med . 2020;121:103792.
Zhu J, Shen B, Abbasi A, et al. Deep transfer learning artificial intelligence accurately stages COVID-19 lung disease severity on portable chest radiographs. PLoS One . 2020;15:e0236621.
Liang W, Yao J, Chen A, et al. Early triage of critically ill COVID-19 patients using deep learning. Nat Commun . 2020;11:3543.
Rubin GD, Ryerson CJ, Haramati LB, et al. The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner society. Radiology . 2020;296:172–180.
Revel MP, Parker AP, Prosch H, et al. COVID-19 patients and the radiology department—advice from the European Society of Radiology (ESR) and the European Society of Thoracic Imaging (ESTI). Eur Radiol . 2020;30:4903–4909.
Rai A, MacGregor K, Hunt B, et al. Proof of concept: phantom study to ensure quality and safety of portable chest radiography through glass during the COVID-19 pandemic. Invest Radiol . 2020. doi:10.1097/RLI.0000000000000716.
doi: 10.1097/RLI.0000000000000716
Padley SPG, Hansell DM, Flower CDR, et al. Comparative accuracy of high resolution computed tomography and chest radiography in the diagnosis of chronic diffuse infiltrative lung disease. Clin Radiol . 1991;44:222–226.
Ebner L, Bütikofer Y, Ott D, et al. Lung nodule detection by microdose CT versus chest radiography (standard and dual-energy subtracted). Am J Roentgenol . 2015;204:727–735.
Cohen JF, Korevaar DA, Altmann DG, et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open . 2016;6:e012799.
Cohen JP, Morrison P, Dao L, et al. COVID-19 Image Data Collection. Available at: https://github.com/ieee8023/covid-chestxray-dataset . 2020. Updated July 31, 2020. Accessed May 2020.
Wang L, Lin ZQ, Wong A, et al. DarwinAI Corp, Canada and Vision and Image Processing Research Group, University of Waterloo, Canada. Figure 1 COVID-19 Chest X-ray data initiative. Available at: https://github.com/agchung/Figure1-COVID-chestxray-dataset . Updated May 4, 2020. Accessed May 2020.
Wang L, Lin ZQ, Wong A, et al. DarwinAI Corp, Canada and Vision and Image Processing Research Group, University of Waterloo, Canada. Actualmed COVID-19 Chest X-ray data initiative. Available at: https://github.com/agchung/Actualmed-COVID-chestxray-dataset . Updated May 1, 2020. Accessed May 2020.
Chowdhury M, Rahman T, Khandakar A. Qatar University, Doha, Qatar and University of Dhaka, Bangladesh. Can AI help in screening Viral and COVID-19 pneumonia? IEEE Access . 2020;8:132665–132676. COVID-19 Radiography Database. Available at: https://www.kaggle.com/tawsifurrahman/covid19-radiography-database . Updated March 28, 2020. Accessed May 2020.
Radiological Society of North America. RSNA Pneumonia Detection Challenge. Available at: https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/data . Updated November 1, 2018. Accessed May 2020.
Wang L, Lin ZQ, Wong A, et al. COVID-net: a tailored deep convolutional neural network Design for Detection of COVID-19 cases from chest X-ray images. ArXiv e-prints . 2003.09871. May 11, 2020. Available at: https://arxiv.org/abs/2003.09871 . Accessed June 13, 2020.
Sandler M, Howard A, Zhu M, et al. Mobilenetv2: inverted residuals and linear bottlenecks. Proc IEEE Conf Comp Vis Pattern Recog . 2018;4510–4520.
Chen H, Lundberg S, Lee S-I. Checkpoint ensembles: ensemble Methods from a single training process. ArXiv:1710.03282. October 9, 2017. Available at: https://arxiv.org/abs/1710.03282 . Accessed October 5, 2020.
Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database. Proc IEEE Conf Comp Vis Pattern Recog . 2009;248–255.
Keras. Version 2.2.4. Francois Chollet. GitHub. 2015. Available at: https://keras.io . Accessed June 2020.
Collins J, Stern EJ. Chest Radiology: The Essentials . 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2014.
Vilar J, Domingo ML, Soto C, et al. Radiology of bacterial pneumonia. Eur J Radiol . 2004;51:102–113.
Christe A, Walti L, Charimo J, et al. Imaging patterns of pneumocystis jirovecii pneumonia in HIV-positive and renal transplant patients—a multicentre study. Swiss Med Wkly . 2019;149:w20130.
Franquet T, Giménez A, Hidalgo A. Imaging of opportunistic fungal infections in immunocompromised patient. Eur J Radiol . 2004;51:130–138.
Davies L, Petitti DB, Martin L, et al. Defining, estimating, and communicating overdiagnosis in cancer screening. Ann Intern Med . 2018;169:36–43.
Welch HG, Prorok PC, O'Malley JA, et al. Breast-cancer tumor size, overdiagnosis, and mammography screening effectiveness. N Engl J Med . 2016;375:1438–1447.
Jajodia A, Ebner L, Heidinger B, et al. Imaging in corona virus disease 2019 (COVID-19)—a scoping review. Eur J Radiol Open . 2020;100237:7.
Bernheim A, Mei X, Huang M, et al. Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection. Radiology . 2020;295:200463.
Xiong Y, Sun D, Liu Y, et al. Clinical and high-resolution CT features of the COVID-19 infection: comparison of the initial and follow-up changes. Invest Radiol . 2020;55:332–339.
Xu X, Jiang X, Ma C, et al. A deep learning system to screen novel coronavirus disease 2019 pneumonia. Engineering (Beijing) . 2020. February 21, 2020. Available at: https://doi.org/10.1016/j.eng.2020.04.010 . Accessed August 2020.
doi: 10.1016/j.eng.2020.04.010
Gozes O, Frid-Adar M, Greenspan H, et al. Rapid AI development cycle for the coronavirus (COVID-19) pandemic: initial results for automated detection & patient monitoring using deep learning CT image analysis. ArXiv . e-prints: 2003.05037. March 24, 2020. Available at: https://arxiv.org/abs/2003.05037 . Accessed August 2020.
Chen J, Wu L, Zhang J, et al. Deep learning–based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography: a prospective study. medRxiv . preprint: 2020.02.25.20021568. Available at: https://doi.org/10.1101/2020.02.25.20021568 . Accessed August 2020.
doi: 10.1101/2020.02.25.20021568
Leonardi A, Scipione R, Alfieri G, et al. Role of computed tomography in predicting critical disease in patients with covid-19 pneumonia: a retrospective study using a semiautomatic quantitative method. Eur J Radiol . 2020;130:109202.
Grégory J, Raynaud L, Galy A, et al. Extension of COVID-19 pulmonary parenchyma lesions based on real-life visual assessment on initial chest CT is an independent predictor of poor patient outcome. Infect Dis (Lond) . 2020;52:838–840.
Murphy K, Smits H, Knoops AJG, et al. COVID-19 on chest radiographs: a multireader evaluation of an artificial intelligence system. Radiology . 2020;296:E166–E172.
Li MD, Arun NT, Gidwani M, et al. Automated assessment and tracking of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese neural networks. Radiol Artif Intell . 2020;2:4.
Toussie D, Voutsinas N, Finkelstein M, et al. Clinical and chest radiography features determine patient outcomes in young and middle-aged adults with COVID-19. Radiology . 2020;297:E197–E206.