A medical image classification method based on self-regularized adversarial learning.
adversarial learning
deep learning
medical image classification
Journal
Medical physics
ISSN: 2473-4209
Titre abrégé: Med Phys
Pays: United States
ID NLM: 0425746
Informations de publication
Date de publication:
30 Jul 2024
30 Jul 2024
Historique:
revised:
10
06
2024
received:
30
11
2023
accepted:
20
06
2024
medline:
30
7
2024
pubmed:
30
7
2024
entrez:
30
7
2024
Statut:
aheadofprint
Résumé
Deep learning (DL) techniques have been extensively applied in medical image classification. The unique characteristics of medical imaging data present challenges, including small labeled datasets, severely imbalanced class distribution, and significant variations in imaging quality. Recently, generative adversarial network (GAN)-based classification methods have gained attention for their ability to enhance classification accuracy by incorporating realistic GAN-generated images as data augmentation. However, the performance of these GAN-based methods often relies on high-quality generated images, while large amounts of training data are required to train GAN models to achieve optimal performance. In this study, we propose an adversarial learning-based classification framework to achieve better classification performance. Innovatively, GAN models are employed as supplementary regularization terms to support classification, aiming to address the challenges described above. The proposed classification framework, GAN-DL, consists of a feature extraction network (F-Net), a classifier, and two adversarial networks, specifically a reconstruction network (R-Net) and a discriminator network (D-Net). The F-Net extracts features from input images, and the classifier uses these features for classification tasks. R-Net and D-Net have been designed following the GAN architecture. R-Net employs the extracted feature to reconstruct the original images, while D-Net is tasked with the discrimination between the reconstructed image and the original images. An iterative adversarial learning strategy is designed to guide model training by incorporating multiple network-specific loss functions. These loss functions, serving as supplementary regularization, are automatically derived during the reconstruction process and require no additional data annotation. To verify the model's effectiveness, we performed experiments on two datasets, including a COVID-19 dataset with 13 958 chest x-ray images and an oropharyngeal squamous cell carcinoma (OPSCC) dataset with 3255 positron emission tomography images. Thirteen classic DL-based classification methods were implemented on the same datasets for comparison. Performance metrics included precision, sensitivity, specificity, and Our adversarial-based classification framework leverages GAN-based adversarial networks and an iterative adversarial learning strategy to harness supplementary regularization during training. This design significantly enhances classification accuracy and mitigates overfitting issues in medical image datasets. Moreover, its modular design not only demonstrates flexibility but also indicates its potential applicability to various clinical contexts and medical imaging applications.
Sections du résumé
BACKGROUND
BACKGROUND
Deep learning (DL) techniques have been extensively applied in medical image classification. The unique characteristics of medical imaging data present challenges, including small labeled datasets, severely imbalanced class distribution, and significant variations in imaging quality. Recently, generative adversarial network (GAN)-based classification methods have gained attention for their ability to enhance classification accuracy by incorporating realistic GAN-generated images as data augmentation. However, the performance of these GAN-based methods often relies on high-quality generated images, while large amounts of training data are required to train GAN models to achieve optimal performance.
PURPOSE
OBJECTIVE
In this study, we propose an adversarial learning-based classification framework to achieve better classification performance. Innovatively, GAN models are employed as supplementary regularization terms to support classification, aiming to address the challenges described above.
METHODS
METHODS
The proposed classification framework, GAN-DL, consists of a feature extraction network (F-Net), a classifier, and two adversarial networks, specifically a reconstruction network (R-Net) and a discriminator network (D-Net). The F-Net extracts features from input images, and the classifier uses these features for classification tasks. R-Net and D-Net have been designed following the GAN architecture. R-Net employs the extracted feature to reconstruct the original images, while D-Net is tasked with the discrimination between the reconstructed image and the original images. An iterative adversarial learning strategy is designed to guide model training by incorporating multiple network-specific loss functions. These loss functions, serving as supplementary regularization, are automatically derived during the reconstruction process and require no additional data annotation.
RESULTS
RESULTS
To verify the model's effectiveness, we performed experiments on two datasets, including a COVID-19 dataset with 13 958 chest x-ray images and an oropharyngeal squamous cell carcinoma (OPSCC) dataset with 3255 positron emission tomography images. Thirteen classic DL-based classification methods were implemented on the same datasets for comparison. Performance metrics included precision, sensitivity, specificity, and
CONCLUSION
CONCLUSIONS
Our adversarial-based classification framework leverages GAN-based adversarial networks and an iterative adversarial learning strategy to harness supplementary regularization during training. This design significantly enhances classification accuracy and mitigates overfitting issues in medical image datasets. Moreover, its modular design not only demonstrates flexibility but also indicates its potential applicability to various clinical contexts and medical imaging applications.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : Foundation for the National Institutes of Health
ID : EB031772
Organisme : Foundation for the National Institutes of Health
ID : EB031585
Organisme : Foundation for the National Institutes of Health
ID : CA238191
Organisme : Foundation for the National Institutes of Health
ID : EB034249
Organisme : Foundation for the National Institutes of Health
ID : EB028652
Organisme : Foundation for the National Institutes of Health
ID : CA233873
Organisme : Foundation for the National Institutes of Health
ID : DE033344
Organisme : Foundation for the National Institutes of Health
ID : CA274318
Organisme : Cancer Center at Illinois
Informations de copyright
© 2024 American Association of Physicists in Medicine.
Références
Shen D, Wu G, Suk H‐I. Deep learning in medical image analysis. Annu Rev Biomed Eng. 2017;19:221‐248.
Razzak MI, Naz S, Zaib A. Deep learning for medical image processing: overview, challenges and the future. Classification in BioApps: Automation of Decision Making. Springer; 2018:323‐350.
Zhang X, Landsness EC, Chen W, et al. Automated sleep state classification of wide‐field calcium imaging data via multiplex visibility graphs and deep learning. J Neurosci Methods. 2022;366:109421.
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2016:770‐778.
Saric M, Russo M, Stella M, Sikora M. CNN‐based method for lung cancer detection in whole slide histopathology images. 2019 4th International Conference on Smart and Sustainable Technologies (SpliTech). 2019:1‐4.
Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2015:1‐9.
Kermany DS, Goldbaum M, Cai W, et al. Identifying medical diagnoses and treatable diseases by image‐based deep learning. Cell. 2018;172:1122‐1131.
Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. 2020.
Almalik F, Yaqub M, Nandakumar K. Self‐ensembling vision transformer (sevit) for robust medical image classification. In: International Conference on Medical Image Computing and Computer‐Assisted Intervention. Springer; 2022:376‐386.
Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE; 2021:10012‐10022.
Xie J, Wu Z, Zhu R, Zhu H. Melanoma detection based on swin transformer and SimAM. In: 2021 IEEE 5th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Vol 5. IEEE; 2021:1517‐1521.
Varoquaux G, Cheplygina V. Machine learning for medical imaging: methodological failures and recommendations for the future. npj Digital Med. 2022;5:48.
Johnson J, Khoshgoftaar T. Survey on deep learning with class imbalance. J Big Data. 2019;6:1‐54.
Perez L, Wang J. The effectiveness of data augmentation in image classification using deep learning. ArXiv abs/1712.04621. 2017.
Liu Y, Yang G, Qiao S, et al. Imbalanced data classification: using transfer learning and active sampling. Eng Appl Artif Intell. 2023;117:105621.
Hu F, Chen AA, Horng H, et al. Image harmonization: a review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization. NeuroImage. 2023;274:120125.
Tan M, Le Q. Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning. PMLR; 2019:6105‐6114.
Pan SJ, Yang Q. A survey on transfer learning. IEEE Trans Knowl Data Eng. 2009;22:1345‐1359.
Lin T‐Y, Goyal P, Girshick RB, He K, Dollár P. Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE; 2017:2999‐3007.
Wen Y, Zhang K, Li Z, Qiao Y. A discriminative feature learning approach for deep face recognition. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11‐14, 2016; Proceedings, Part VII 14. Springer; 2016:499‐515.
Tian K, Xu Y, Guan J, Zhou S. Network as regularization for training deep neural networks: framework, model and performance. In: Proceedings of the AAAI Conference on Artificial Intelligence. AAAI; 2020;34:6013‐6020.
Zhu F, Li H, Ouyang W, Yu N, Wang X. Learning spatial regularization with image‐level supervisions for multi‐label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE; 2017:5513‐5522.
Fan Z, Gong P, Tang S, et al. Joint localization and classification of breast masses on ultrasound images using an auxiliary attention‐based framework. Med Image Anal. 2023;90:102960.
Schlemper J, Oktay O, Schaap M, et al. Attention gated networks: learning to leverage salient regions in medical images. Med Image Anal. 2019;53:197‐207.
Yang X, Song Z, King I, Xu Z. A survey on deep semi‐supervised learning. ArXiv abs/2103.00550. 2021.
Guan Q, Huang Y. Multi‐label chest x‐ray image classification via category‐wise residual attention learning. Pattern Recognit Lett. 2020;130:259‐266.
Gonçalves T, Rio‐Torto I, Teixeira LF, Cardoso JS. A survey on attention mechanisms for medical applications: are we moving towards better algorithms? IEEE Access. 2022;22:1‐1.
Hassanin M, Anwar S, Radwan I, Khan FS, Mian A. Visual attention methods in deep learning: an in‐depth survey. arXiv preprint arXiv:2204.07756. 2022.
Goodfellow IJ, Pouget‐Abadie J, Mirza M, et al. Generative adversarial networks. ArXiv abs/1406.2661. 2014.
Gui J, Sun Z, Wen Y, Tao D, Ye J. A review on generative adversarial networks: Algorithms, theory, applications. IEEE Trans Knowl Data Eng. 2021;35:3313‐3332.
Frid‐Adar M, Diamant I, Klang E, Amitai M, Goldberger J, Greenspan H. GAN‐based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing. 2018;321:321‐331.
Yi X, Walia E, Babyn P. Generative adversarial network in medical imaging: a review. Med Image Anal. 2019;58:101552.
Li C, Xu K, Zhu J, Liu J, Zhang B. Triple generative adversarial networks. in IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022;44(12):9629‐9640, doi: 10.1109/TPAMI.2021.3127558
Haque A. EC‐GAN: Low‐sample classification using semi‐supervised algorithms and GANs. ArXiv abs/2012.15864. 2020.
Qin Z, Liu Z, Zhu P, Xue Y. A GAN‐based image synthesis method for skin lesion classification. Comput Methods Programs Biomed. 2020;195:105568.
Ma L, Shuai R, Ran X, Liu W, Ye C. Combining DC‐GAN with ResNet for blood cell image classification. Med Biol Eng Comput. 2020;58:1251‐1264.
Saha S, Sheikh N. Ultrasound image classification using ACGAN with small training dataset. In: Recent Trends in Signal and Image Processing: ISSIP 2020. Springer; 2021:85‐93.
Varghese A, MohammedSafwanK P, Chennamsetty SS, Krishnamurthi G. Generative adversarial networks for brain lesion detection. In: Proceedings, Medical Imaging 2017: Image Processing. SPIE; 2017.
Odena A, Olah C, Shlens J. Conditional image synthesis with auxiliary classifier GANs. In: International Conference on Machine Learning. (vol. 70, pp. 2642‐2651). PMLR. 2017.
Karras T, Aittala M, Hellsten J, Laine S, Lehtinen J, Aila T. Training generative adversarial networks with limited data. Adv Neural Inf Process Syst. 2020;33:12104‐12114.
He K, Zhang X, Ren S, Sun J. Identity mappings in deep residual networks. ArXiv abs/1603.05027. 2016.
Mao X‐J, Shen C, Yang Y‐B. Image restoration using convolutional auto‐encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921. 2016.
Luc P, Couprie C, Chintala S, Verbeek J. Semantic segmentation using adversarial networks. arXiv preprint arXiv:1611.08408. 2016.
Mirza M, Osindero S. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784. 2014.
Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P. InfoGAN: interpretable representation learning by information maximizing generative adversarial nets. 2016.
Isola P, Zhu J‐Y, Zhou T, Efros AA. Image‐to‐image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE; 2017:1125‐1134.
Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M. Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806. 2014.
Wang L, Wong A. COVID‐Net: a tailored deep convolutional neural network design for detection of COVID‐19 cases from chest x‐ray images. arXiv preprint arXiv:2003.09871. 2020.
Saad M, He S, Thorstad W, et al. Learning‐based cancer treatment outcome prognosis using multimodal biomarkers. IEEE Trans Radiat Plasma Med Sci. 2022;6:231‐244.
Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high‐performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, eds. Wallach H, Larochelle H, Beygelzimer A, d'Alché‐Buc F, Fox E, Garnett R, Curran Associates, Inc.; 2019:8024‐8035.
Kingma DP, Ba J. Adam: A method for stochastic optimization. CoRR abs/1412.6980. 2015.
Huang G, Liu Z, Weinberger KQ. Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2016:2261‐2269.
Kubat M, Matwin S. Addressing the curse of imbalanced training sets: one‐sided selection. In: Proceedings of the 14th International Conference on Machine Learning. 1997;179‐186.
Shrivastava A, Gupta A, Girshick RB. Training region‐based object detectors with online hard example mining. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016:761‐769.
Zhao H, Jia J, Koltun V. Exploring self‐attention for image recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2020:10073‐10082.
Karras T, Aila T, Laine S, Lehtinen J. Progressive growing of gans for improved quality, stability, variation. arXiv preprint arXiv:1710.10196. 2017.
Van der Maaten L, Hinton G. Visualizing data using t‐SNE. J Mach Learn Res. 2008;9:2579‐2605.
Rousseeuw PJ. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math. 1987;20:53‐65.
Caliński T, Harabasz J. A dendrite method for cluster analysis. Comm Statist Theory Methods. 1974;3:1‐27.
Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. PMLR; 2010;9:249‐256.
Alzubaidi L, Fadhel MA, Al‐Shamma O, et al. Towards a better understanding of transfer learning for medical imaging: a case study. Appl Sci. 2020;10:4523.
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13:600‐612.
Snell J, Ridgeway K, Liao R, Roads BD, Mozer MC, Zemel RS. Learning to generate images with perceptual similarity metrics. In: 2017 IEEE International Conference on Image Processing (ICIP). IEEE; 2015:4277‐4281.
Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep learning. J of Big Data. 2019;6:1‐48.
Johnson J, Alahi A, Fei‐Fei L. Perceptual losses for real‐time style transfer and super‐resolution. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11‐14, 2016; Proceedings, Part II 14. Springer; 2016:694‐711.
Karras T, Laine S, Aila T. A style‐based generator architecture for generative adversarial networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2019:4396‐4405.
He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE; 2022:16000‐16009.
Caron M, Touvron H, Misra I, et al. Emerging properties in self‐supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. IEEE; 2021:9650‐9660.
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad‐cam: visual explanations from deep networks via gradient‐based localization. In: Proceedings of the IEEE International Conference on Computer Vision. IEEE; 2017:618‐626.
Chen Z, Bei Y, Rudin C. Concept whitening for interpretable image recognition. Nat Mach Intell. 2020;2:772‐782.