Weakly supervised pneumonia localization in chest X-rays using generative adversarial networks.
Generative adversarial networks GAN
pneumonia localization
weakly supervised
Journal
Medical physics
ISSN: 2473-4209
Titre abrégé: Med Phys
Pays: United States
ID NLM: 0425746
Informations de publication
Date de publication:
Nov 2021
Nov 2021
Historique:
revised:
12
07
2021
received:
19
10
2020
accepted:
27
07
2021
pubmed:
31
8
2021
medline:
18
11
2021
entrez:
30
8
2021
Statut:
ppublish
Résumé
Automatic localization of pneumonia on chest X-rays (CXRs) is highly desirable both as an interpretive aid to the radiologist and for timely diagnosis of the disease. However, pneumonia's amorphous appearance on CXRs and complexity of normal anatomy in the chest present key challenges that hinder accurate localization. Existing studies in this area are either not optimized to preserve spatial information of abnormality or depend on expensive expert-annotated bounding boxes. We present a novel generative adversarial network (GAN)-based machine learning approach for this problem, which is weakly supervised (does not require any location annotations), was trained to retain spatial information, and can produce pixel-wise abnormality maps highlighting regions of abnormality (as opposed to bounding boxes around abnormality). Our method is based on the Wasserstein GAN framework and, to the best of our knowledge, the first application of GANs to this problem. Specifically, from an abnormal CXR as input, we generated the corresponding pseudo normal CXR image as output. The pseudo normal CXR is the "hypothetical" normal, if the same abnormal CXR were not to have any abnormalities. We surmise that the difference between the pseudo normal and the abnormal CXR highlights the pixels suspected to have pneumonia and hence is our output abnormality map. We trained our algorithm on an "unpaired" data set of abnormal and normal CXRs and did not require any location annotations such as bounding boxes/segmentations of abnormal regions. Furthermore, we incorporated additional prior knowledge/constraints into the model and showed that they help improve localization performance. We validated the model on a data set consisting of 14 184 CXRs from the Radiological Society of North America pneumonia detection challenge. We evaluated our methods by comparing the generated abnormality maps with radiologist annotated bounding boxes using receiver operating characteristic (ROC) analysis, image similarity metrics such as normalized cross-correlation/mutual information, and abnormality detection rate.We also present visual examples of the abnormality maps, covering various scenarios of abnormality occurrence. Results demonstrate the ability to highlight regions of abnormality with the best method achieving an ROC area under the curve (AUC) of 0.77 and a detection rate of 85%.The GAN tended to perform better as prior knowledge/constraints were incorporated into the model. We presented a novel GAN based approach for localizing pneumonia on CXRs that (1) does not require expensive hand annotated location ground truth; and (2) was trained to produce abnormality maps at the pixel level as opposed to bounding boxes. We demonstrated the efficacy of our methods via quantitative and qualitative results.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
7154-7171Subventions
Organisme : NCI NIH HHS
ID : P30 CA008748
Pays : United States
Organisme : NCI NIH HHS
ID : P30 CA008748
Pays : United States
Informations de copyright
© 2021 American Association of Physicists in Medicine.
Références
Centers for disease control and preventionhttps://www.cdc.gov/pneumonia/prevention.html. Accessed on 9th Sept. 2021.
WHO. Standardization of interpretation of chest radiographs for the diagnosis of pneumonia in children. Accessed on 9th Sept. 2021. 2001.
Delrue L, Gosselin R, Ilsen B, Van Landeghem A, Mey dJ, Duyck P. Difficulties in the interpretation of chest radiography. In: Coche EE, Ghaye B, Mey J, Duyck P, eds. Comparative Interpretation of CT and Standard Radiography of the Chest. Springer; 2011:27-49.
Ropp A, Waite S, Reede D, Patel J. Did i miss that: subtle and commonly missed findings on chest radiographs. Curr Prob Diagn Radiol. 2015;44:277-289.
Bruno MA, Walker EA, Abujudeh HH. Understanding and confronting our mistakes: the epidemiology of error in radiology and strategies for error reduction. Radiographics. 2015;35:1668-1676.
Berlin L. Radiologic errors, past, present and future. Diagnosis. 2014;1:79-84.
Rao B, Zohrabian V, Cedeno P, Saha A, Pahade J, Davis MA. Utility of artificial intelligence tool as a prospective radiology peer reviewer-detection of unreported intracranial hemorrhage. Acad Radiol. 2021;28:85-93.
Gao Y, Geras KJ, Lewin AA, Moy L. New frontiers: an update on computer-aided diagnosis for breast imaging in the age of artificial intelligence. Am J Roentgenol. 2019;212:300-307.
Bley TA, Baumann T, Saueressig U, et al. Comparison of radiologist and cad performance in the detection of ct-confirmed subtle pulmonary nodules on digital chest radiographs. Invest Radiol. 2008;43:343-348.
Franquet T. Imaging of community-acquired pneumonia. J Thoracic Imaging. 2018;33:282-294.
Barry K. The chest radiograph. Ulster Med J. 2012;81:143-148.
Novack V, Avnon LS, Smolyakov A, Barnea R, Jotkowitz A, Schlaeffer F. Disagreement in the interpretation of chest radiographs among specialists and clinical outcomes of patients hospitalized with suspected pneumonia. Eur J Internal Med. 2006;17:43-47.
Neuman MI, Lee EY, Bixby S, et al. Variability in the interpretation of chest radiographs for the diagnosis of pneumonia in children. J Hosp Med. 2012;7:294-298.
Albaum MN, Hill LC, Murphy M, et al. Interobserver reliability of the chest radiograph in community-acquired pneumonia. Chest. 1996;110:343-350.
Young M, Marrie TJ. Interobserver variability in the interpretation of chest roentgenograms of patients with possible pneumonia. Arch Internal Med. 1994;154:2729-2732.
Xia Y, Ying Y, Wang S, Li W, Shen H. Effectiveness of lung ultrasonography for diagnosis of pneumonia in adults: a systematic review and meta-analysis. J Thoracic Dis. 2016;8:2822-2831.
Claessens YE, Debray MP, Tubach F, et al. Early chest computed tomography scan to assist diagnosis and guide treatment decision for suspected community-acquired pneumonia. Am J Respir Crit Care Med. 2015;192:974-982. PMID: 26168322.
Maughan BC, Asselin N, Carey JL, Sucov A, Valente JH. False-negative chest radiographs in emergency department diagnosis of pneumonia. Rhode Isl Med J. 2014;97:20-23.
Hashmi MF, Katiyar S, Keskar AG, Bokde ND, Geem ZW. Efficient pneumonia detection in chest x-ray images using deep transfer learning. Diagnostics. 2020;10:417.
Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016:2921-2929.
Irvin J, Rajpurkar P, Ko M, et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019.
Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. Chest x-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017:3462-3471.
Baumgartner CF, Koch LM, Can Tezcan K, Xi Ang J, Konukoglu E. Visual feature attribution using wasserstein gans. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Li Z, Wang C, Han M, et al. Thoracic Disease Identification and Localization with Limited Supervision. Springer International Publishing; 2019:139-161.
Taghanaki SA, Havaei M, Berthier T, et al. Infomask: masked variational latent representation to localize chest disease. In: Shen D, Liu T, Peters TM, Staib LH, Essert C, Zhou S, Yap P-T, Khan A, eds. Medical Image Computing and Computer Assisted Intervention - MICCAI 2019. Springer International Publishing; 2019:739-747.
Sirazitdinov I, Kholiavchenko M, Mustafaev T, Yixuan Y, Kuleev R, Ibragimov B. Deep neural network ensemble for pneumonia localization from a large-scale chest x-ray database. Comput Electr Eng. 2019;78:388-399.
Babenko B. Multiple Instance Learning: Algorithms and Applications. Springer; 2008.
Rsna pneumonia detection challenge. https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/overview. Accessed on 9th Sept. 2021.
Pan I, Cadrin-Chênevert A, Cheng PM. Tackling the radiological society of North America pneumonia detection challenge. Am J Roentgenol. 2019;213:568-574.
Zhang K, Zuo W, Chen Y, Meng D, Zhang L. Beyond a Gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Trans Image Process. 2017;26:3142-3155.
Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ, eds. Advances in Neural Information Processing Systems 27. Curran Associates, Inc.; 2014:2672-2680.
Isola P, Zhu JY, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Zhu JY, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
Wolterink JM, Leiner T, Viergever MA, Išgum I. Generative adversarial networks for noise reduction in low-dose ct. IEEE Trans Med Imaging. 2017;36:2536-2545.
Mahapatra D, Bozorgtabar B, Hewavitharanage S, Garnavi R. Image super resolution using generative adversarial networks and local saliency maps for retinal image analysis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2017:382-390.
Kim KH, Do WJ, Park SH. Improving resolution of mr images with an adversarial network incorporating images with different contrast. Med Phys. 2018;45:3120-3131.
Nie D, Trullo R, Lian J, et al. Medical image synthesis with context-aware generative adversarial networks. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S, eds. Medical Image Computing and Computer Assisted Intervention - MICCAI 2017. Springer International Publishing; 2017:417-425.
Wolterink JM, Dinkla AM, Savenije MHF, Seevinck PR, van den Berg CAT, Igum I. Deep MR to CT synthesis using unpaired data. In: SASHIMI@MICCAI, 2017.
Xue Y, Xu T, Zhang H, Long LR, Huang X. Segan: adversarial network with multi-scale l1 loss for medical image segmentation. Neuroinformatics. 2018;16:383-392.
Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks. In: Precup D, Teh YW, eds. Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research. International Convention Centre, Sydney, Australia; 2017: 214-223.
Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell. 2017;39:640-651.
Dosovitskiy A, Fischer P, Ilg E, et al. Flownet: learning optical flow with convolutional networks. In: 2015 IEEE International Conference on Computer Vision (ICCV); 2015:2758-2766.
Theano deep learning documentation. http://deeplearning.net/tutorial/fcn_2D_segm.html. Accessed on 1st Oct. 2020.
Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF, eds. Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015. Springer International Publishing; 2015:234-241.
Rubner Y, Tomasi C, Guibas LJ. A metric for distributions with applications to image databases. In: ICCV '98: Proceedings of the Sixth International Conference on Computer Vision. IEEE Computer Society; 1998:59.
Hitchcock FL. The distribution of a product from several sources to numerous localities. J Math Phys. 1941;20:224-230.
Earth mover's distance. http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/RUBNER/emd.htm. Accessed on 9th Sept. 2021.
Rachev ST, Ruschendorf L. Mass transportation problems. Volume I: Theory, Probability and its Applications, Vol. 1. Springer-Verlag; 1998.
Sohrab HH. Basic Real Analysis, Vol. 231. 2nd ed. Springer; 2003.
Villani C. Optimal Transport: Old and New, Vol. 338. Springer Science & Business Media; 2008.
Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC. Improved training of wasserstein gans. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, eds. Advances in Neural Information Processing Systems 30. Curran Associates, Inc.; 2017:5767-5777.
Zitová B, Flusser J. Image registration methods: a survey. Image Vision Comput. 2003;21:977-1000.
Kingma DP, Ba J. Adam: a method for stochastic optimization, 2014. cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Shih G, Wu CC, Halabi SS, et al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiol Artif Intell. 2019;1:e180041.
Pneumonia dataset annotation methods. https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/64723. Accessed on 9th Sept, 2021.
Gabruseva T, Poplavskiy D, Kalinin A. Deep learning for automatic pneumonia detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
Girosi F, Jones M, Poggio T. Regularization theory and neural networks architectures. Neural Comput. 1995;7:219-269.
Johnson HJ, McCormick MM, Ibanez L, et al. The ITK Software Guide: Design and Functionality. Kitware; 2015.
Lin TY, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell. 2020;42:318-327.
Isensee F, Petersen J, Klein A, et al. Abstract: nnu-net: self-adapting framework for u-net-based medical image segmentation. In Handels H, Deserno TM, Maier A, Maier-Hein KH, Palm C, Tolxdorff T, eds. Bildverarbeitung für die Medizin 2019. Springer Fachmedien Wiesbaden; 2019:22.
Wu Y, Kirillov A, Massa F, Lo WY & Girshick R Detectron2. https://github.com/facebookresearch/detectron2, 2019. Accessed on 9th Sept. 2021
Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32-35.
Kegelmeyer Jr WP, Pruneda JM, Bourland PD, MLNA. Computer-aided mammographic screening for spiculated lesions. Radiology. 1994;191:331-7.
Oliver A. Automatic Mass Segmentation in Mammographic Images. PhD thesis, Universitat De Girona; 2008.
Kallergi M, Carney GM, Gaviria J. Evaluating the performance of detection algorithms in digital mammography. Med Phys. 1999;26:267-275.
Penedo M, Souto M, Tahoces PG, et al. Free-response receiver operating characteristic evaluation of lossy jpeg2000 and object-based set partitioning in hierarchical trees compression of digitized mammograms. Radiology. 2005;237:450-457. PMID: 16244253.
Moor dT, Rodriguez-Ruiz A, Mérida AG, Mann R, Teuwen J. Automated lesion detection and segmentation in digital mammography using a u-net deep learning network. In: Krupinski EA, ed. 14th International Workshop on Breast Imaging (IWBI 2018), Vol. 10718. International Society for Optics and Photonics, SPIE; 2018:23-29.
Jin L, Yang J, Kuang K, et al. Deep-learning-assisted detection and segmentation of rib fractures from ct scans: development and validation of fracnet. EBioMedicine. 2020;62:103106.
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
The viridis color palettes. https://cran.r-project.org/web/packages/viridis/vignettes/intro-to-viridis.html.
Novack V, Avnon LS, Smolyakov A, Barnea R, Jotkowitz A, Schlaeffer F. Disagreement in the interpretation of chest radiographs among specialists and clinical outcomes of patients hospitalized with suspected pneumonia. Eur J Internal Med. 2006;17:43-47.
Albaum MN, Hill LC, Murphy M, et al. Interobserver reliability of the chest radiograph in community-acquired pneumonia. Chest. 1996;110:343-350.
Chakraborty DP. Observer Performance Methods for Diagnostic Imaging: Foundations, Modeling, and Applications with r-Based Examples, CRC Press; 2017.