Benchmarking robustness of deep neural networks in semantic segmentation of fluorescence microscopy images.
Adversarial robustness
Corruption robustness
Deep neural network
Fluorescence microscopy
Semantic segmentation
Journal
BMC bioinformatics
ISSN: 1471-2105
Titre abrégé: BMC Bioinformatics
Pays: England
ID NLM: 100965194
Informations de publication
Date de publication:
20 Aug 2024
20 Aug 2024
Historique:
received:
14
01
2024
accepted:
07
08
2024
medline:
21
8
2024
pubmed:
21
8
2024
entrez:
20
8
2024
Statut:
epublish
Résumé
Fluorescence microscopy (FM) is an important and widely adopted biological imaging technique. Segmentation is often the first step in quantitative analysis of FM images. Deep neural networks (DNNs) have become the state-of-the-art tools for image segmentation. However, their performance on natural images may collapse under certain image corruptions or adversarial attacks. This poses real risks to their deployment in real-world applications. Although the robustness of DNN models in segmenting natural images has been studied extensively, their robustness in segmenting FM images remains poorly understood RESULTS: To address this deficiency, we have developed an assay that benchmarks robustness of DNN segmentation models using datasets of realistic synthetic 2D FM images with precisely controlled corruptions or adversarial attacks. Using this assay, we have benchmarked robustness of ten representative models such as DeepLab and Vision Transformer. We find that models with good robustness on natural images may perform poorly on FM images. We also find new robustness properties of DNN models and new connections between their corruption robustness and adversarial robustness. To further assess the robustness of the selected models, we have also benchmarked them on real microscopy images of different modalities without using simulated degradation. The results are consistent with those obtained on the realistic synthetic images, confirming the fidelity and reliability of our image synthesis method as well as the effectiveness of our assay. Based on comprehensive benchmarking experiments, we have found distinct robustness properties of deep neural networks in semantic segmentation of FM images. Based on the findings, we have made specific recommendations on selection and design of robust models for FM image segmentation.
Sections du résumé
BACKGROUND
BACKGROUND
Fluorescence microscopy (FM) is an important and widely adopted biological imaging technique. Segmentation is often the first step in quantitative analysis of FM images. Deep neural networks (DNNs) have become the state-of-the-art tools for image segmentation. However, their performance on natural images may collapse under certain image corruptions or adversarial attacks. This poses real risks to their deployment in real-world applications. Although the robustness of DNN models in segmenting natural images has been studied extensively, their robustness in segmenting FM images remains poorly understood RESULTS: To address this deficiency, we have developed an assay that benchmarks robustness of DNN segmentation models using datasets of realistic synthetic 2D FM images with precisely controlled corruptions or adversarial attacks. Using this assay, we have benchmarked robustness of ten representative models such as DeepLab and Vision Transformer. We find that models with good robustness on natural images may perform poorly on FM images. We also find new robustness properties of DNN models and new connections between their corruption robustness and adversarial robustness. To further assess the robustness of the selected models, we have also benchmarked them on real microscopy images of different modalities without using simulated degradation. The results are consistent with those obtained on the realistic synthetic images, confirming the fidelity and reliability of our image synthesis method as well as the effectiveness of our assay.
CONCLUSIONS
CONCLUSIONS
Based on comprehensive benchmarking experiments, we have found distinct robustness properties of deep neural networks in semantic segmentation of FM images. Based on the findings, we have made specific recommendations on selection and design of robust models for FM image segmentation.
Identifiants
pubmed: 39164632
doi: 10.1186/s12859-024-05894-4
pii: 10.1186/s12859-024-05894-4
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
269Subventions
Organisme : National Natural Science Foundation of China
ID : 92354307, 91954201, 31971289, 32101216
Organisme : National Natural Science Foundation of China
ID : 92354307, 91954201, 31971289, 32101216
Organisme : National Natural Science Foundation of China
ID : 92354307, 91954201, 31971289, 32101216
Organisme : Strategic Priority Research Program of the Chinese Academy of Sciences
ID : XDB37040402
Organisme : Strategic Priority Research Program of the Chinese Academy of Sciences
ID : XDB37040402
Organisme : Strategic Priority Research Program of the Chinese Academy of Sciences
ID : XDB37040402
Organisme : Fundamental Research Funds for the Central Universities
ID : E3E45201X2
Organisme : Fundamental Research Funds for the Central Universities
ID : E3E45201X2
Organisme : Fundamental Research Funds for the Central Universities
ID : E3E45201X2
Informations de copyright
© 2024. The Author(s).
Références
Minaee S, Boykov Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D. Image segmentation using deep learning: a survey. IEEE Trans Pattern Anal Mach Intell. 2021;44(7):3523–42.
Garcia-Garcia A, Orts-Escolano S, Oprea S, Villena-Martinez V, Martinez-Gonzalez P, Garcia-Rodriguez J. A survey on deep learning techniques for image and video semantic segmentation. Appl Soft Comput. 2018;70:41–65.
doi: 10.1016/j.asoc.2018.05.018
Sadanandan SK, Ranefall P, Le Guyader S, Wählby C. Automated training of deep convolutional neural networks for cell segmentation. Sci Rep. 2017;7(1):1–7.
doi: 10.1038/s41598-017-07599-6
Kraus OZ, Ba JL, Frey BJ. Classifying and segmenting microscopy images with deep multiple instance learning. Bioinformatics. 2016;32(12):52–9.
doi: 10.1093/bioinformatics/btw252
Xing F, Xie Y, Su H, Liu F, Yang L. Deep learning in microscopy image analysis: a survey. IEEE Trans Neural Netw Learn Syst. 2017;29(10):4550–68.
doi: 10.1109/TNNLS.2017.2766168
Dodge S, Karam L. Understanding how image quality affects deep neural networks. In: International Conference on Quality of Multimedia Experience (QoMEX), 2016:1–6
Dodge S, Karam L. A study and comparison of human and deep learning recognition performance under visual distortions. In: International Conference on Computer Communication and Networks (ICCCN), 2017:1–7
Hendrycks D, Dietterich T. Benchmarking neural network robustness to common corruptions and perturbations. In: International Conference on Learning Representations (ICLR) 2019
Michaelis C, Mitzkus B, Geirhos R, Rusak E, Bringmann O, Ecker AS, Bethge M, Brendel W. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484 2019
Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R. Intriguing properties of neural networks. In: International Conference on Learning Representations (ICLR) 2014
Kamann C, Rother C. Benchmarking the robustness of semantic segmentation models with respect to common corruptions. Int J Comput Vis (IJCV). 2021;129(2):462–83.
doi: 10.1007/s11263-020-01383-2
Arnab A, Miksik O, Torr P. On the robustness of semantic segmentation models to adversarial attacks. IEEE Trans Pattern Anal Mach Intell (TPAMI). 2020;42(12):3040–53.
doi: 10.1109/TPAMI.2019.2919707
Meiniel W, Olivo-Marin J-C, Angelini ED. Denoising of microscopy images: a review of the state-of-the-art, and a new sparsity-based method. IEEE Trans Image Process (TIP). 2018;27(8):3842–56.
doi: 10.1109/TIP.2018.2819821
Zhang Y, Zhu Y, Nichols E, Wang Q, Zhang S, Smith C, Howard S. A Poisson–Gaussian denoising dataset with real fluorescence microscopy images. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. p.11710–11718
Caicedo JC, Roth J, Goodman A, Becker T, Karhohs KW, Broisin M, Molnar C, McQuin C, Singh S, Theis FJ. Evaluation of deep learning strategies for nucleus segmentation in fluorescence images. Cytom A. 2019;95(9):952–65.
doi: 10.1002/cyto.a.23863
Guo Y, Huang J, Zhou Y, Luo Y, Li W, Yang G. Segmentation of intracellular structures in fluorescence microscopy images by fusing low-level features. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), 2021. p. 386–397
Maška M, Ulman V, Delgado-Rodriguez P, Gómez-de-Mariscal E, Nečasová T, Guerrero Peña FA, Ren TI, Meyerowitz EM, Scherr T, Löffler K, Mikut R, Guo T, Wang Y, Allebach JP, Bao R, Al-Shakarji NM, Rahmon G, Toubal IE, Palaniappan K, Lux F, Matula P, Sugawara K, Magnusson KEG, Aho L, Cohen AR, Arbelle A, Ben-Haim T, Raviv TR, Isensee F, Jäger PF, Maier-Hein KH, Zhu Y, Ederra C, Urbiola A, Meijering E, Cunha A, Muñoz-Barrutia A, Kozubek M, Ortiz-de-Solórzano C. The cell tracking challenge: 10 years of objective benchmarking. Nat Methods. 2023;20:1010–20.
doi: 10.1038/s41592-023-01879-y
pubmed: 37202537
pmcid: 10333123
Hendrycks D, Basart S, Mu N, Kadavath S, Wang F, Dorundo E, Desai R, Zhu T, Parajuli S, Guo M. The many faces of robustness: a critical analysis of out-of-distribution generalization. In: International Conference on Computer Vision (ICCV), 2021. p. 8340–8349.
Zheng S, Song Y, Leung T, Goodfellow I. Improving the robustness of deep neural networks via stability training. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. p. 4480–4488.
Vasiljevic I, Chakrabarti A, Shakhnarovich G. Examining the impact of blur on recognition by convolutional networks. arXiv preprint arXiv:1611.05760 2016
Geirhos R, Temme CR, Rauber J, Schütt HH, Bethge M, Wichmann FA. Generalisation in humans and deep neural networks. In: Advances in Neural Information Processing Systems (NeurIPS) 2018.
Rusak E, Schott L, Zimmermann RS, Bitterwolf J, Bringmann O, Bethge M, Brendel W. A simple way to make neural networks robust against diverse image corruptions. In: European Conference on Computer Vision (ECCV), 2020. p. 53–69.
Dodge S, Karam L. Quality resilient deep neural networks. arXiv preprint arXiv:1703.08119 2017
Cubuk ED, Zoph B, Mane D, Vasudevan V, Le QV. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 2018.
Zhang H, Cisse M, Dauphin YN, Lopez-Paz D. mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (ICLR) 2018.
Hendrycks D, Mu N, Cubuk ED, Zoph B, Gilmer J, Lakshminarayanan B. Augmix: A simple data processing method to improve robustness and uncertainty. In: International Conference on Learning Representations (ICLR) 2020.
Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 2015
Kurakin A, Goodfellow IJ, Bengio S. Adversarial examples in the physical world. In: Artificial Intelligence Safety and Security, 2018. p. 99–112.
Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A. Towards deep learning models resistant to adversarial attacks. In: International Conference on Learning Representations (ICLR) 2018.
Dong Y, Liao F, Pang T, Su H, Zhu J, Hu X, Li J. Boosting adversarial attacks with momentum. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. p. 9185–9193.
Zhang H, Yu Y, Jiao J, Xing E, El Ghaoui L, Jordan M. Theoretically principled trade-off between robustness and accuracy. In: International Conference on Machine Learning (ICML), 2019. p. 7472–7482.
Wang Y, Zou D, Yi J, Bailey J, Ma X, Gu Q. Improving adversarial robustness requires revisiting misclassified examples. In: International Conference on Learning Representations (ICLR) 2019.
Kurakin A, Goodfellow I, Bengio S. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236 2016
Rozsa A, Günther M, Boult TE. Are accuracy and robustness correlated. In: IEEE International Conference on Machine Learning and Applications (ICMLA), 2016. p. 227–232.
Liu Y, Chen X, Liu C, Song D. Delving into transferable adversarial examples and black-box attacks. In: International Conference on Learning Representations (ICLR) 2017.
Gilmer J, Ford N, Carlini N, Cubuk E. Adversarial examples are a natural consequence of test error in noise. In: International Conference on Machine Learning (ICML), 2019. p. 2280–2289.
Kireev K, Andriushchenko M, Flammarion N. On the effectiveness of adversarial training against common corruptions. In: Uncertainty in Artificial Intelligence, 2022. p. 1012–1021.
Kang D, Sun Y, Brown T, Hendrycks D, Steinhardt J. Transfer of adversarial robustness between perturbation types. arXiv preprint arXiv:1905.01034 2019
Engstrom L, Tran B, Tsipras D, Schmidt L, Madry A. A rotation and a translation suffice: Fooling CNNS with simple transformations. In: International Conference on Learning Representations (ICLR) 2019.
Chai X, Ba Q, Yang G. Characterizing robustness and sensitivity of convolutional neural networks for quantitative analysis of mitochondrial morphology. Quant Biol. 2018;6(4):344–58.
doi: 10.1007/s40484-018-0156-3
Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. p. 3431–3440.
Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015. p. 234–241.
Zhong L, Li L, Yang G. Robustness benchmark datasets for semantic segmentation of fluorescence images updated. IEEE Dataport 2024 https://doi.org/10.21227/1jk9-nv64
Luo Y, Guo Y, Li W, Liu G, Yang G. Fluorescence microscopy image datasets for deep learning segmentation of intracellular orgenelle networks. IEEE Dataport 2020 https://doi.org/10.21227/t2he-zn97
Caicedo JC, Goodman A, Karhohs KW, Cimini BA, Ackerman J, Haghighi M, Heng C, Becker T, Doan M, McQuin C. Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nat Methods. 2019;16(12):1247–53.
doi: 10.1038/s41592-019-0612-7
pubmed: 31636459
pmcid: 6919559
Feng Y, Chai X, Ba Q, Yang G. Quality assessment of synthetic fluorescence microscopy images for image segmentation. In: IEEE International Conference on Image Processing (ICIP), 2019. p. 814–818.
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial networks. Commun ACM. 2020;63(11):139–44.
doi: 10.1145/3422622
Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 2015
Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks. In: International Conference on Machine Learning (ICML), 2017. p. 214–223.
Zhu J-Y, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision (ICCV), 2017. p. 2223–2232.
Zhong L, Liu G, Yang G. Blind denoising of fluorescence microscopy images using gan-based global noise modeling. In: IEEE International Symposium on Biomedical Imaging (ISBI), 2021. p. 863–867.
Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. p. 1125–1134.
Morris PA, Aspden RS, Bell JE, Boyd RW, Padgett MJ. Imaging with a small number of photons. Nat Commun. 2015;6(1):5913.
doi: 10.1038/ncomms6913
pubmed: 25557090
KuKim S, KiPaik J. Out-of-focus blur estimation and restoration for digital auto-focusing system. Electron Lett. 1998;34(12):1217–9.
doi: 10.1049/el:19980762
Kim SK, Park SR, Paik JK. Simultaneous out-of-focus blur estimation and restoration for digital auto-focusing system. IEEE Trans Consum Electron. 1998;44(3):1071–5.
doi: 10.1109/30.713236
Chen J, Sasaki H, Lai H, Su Y, Liu J, Wu Y, Zhovmer A, Combs CA, Rey-Suarez I, Chang H-Y. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes. Nat Methods. 2021;18(6):678–87.
doi: 10.1038/s41592-021-01155-x
pubmed: 34059829
Edlund C, Jackson TR, Khalid N, Bevan N, Dale T, Dengel A, Ahmed S, Trygg J, Sjögren R. Livecell-a large-scale dataset for label-free live cell segmentation. Nat Methods. 2021;18(9):1038–45.
doi: 10.1038/s41592-021-01249-6
pubmed: 34462594
pmcid: 8440198
Ma J, Xie R, Ayyadhury S, Ge C, Gupta A, Gupta R, Gu S, Zhang Y, Lee G, Kim J, et al. The multimodality cell segmentation challenge: toward universal solutions. Nature Methods, 2024. p. 1–11.
Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell (TPAMI). 2017;39(12):2481–95.
doi: 10.1109/TPAMI.2016.2644615
Chen L-C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 2017
Zhao H, Shi J, Qi X, Wang X, Jia J. Pyramid scene parsing network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. p. 2881–2890.
Zhao H, Qi X, Shen X, Shi J, Jia J. Icnet for real-time semantic segmentation on high-resolution images. In: Proceedings of the European Conference on Computer Vision (ECCV), 2018:405–420
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. p. 770–778.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Advances in Neural Information Processing Systems (NeurIPS) 2017.
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020
Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Lu L, Yuille AL, Zhou Y. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 2021.
Tsipras D, Santurkar S, Engstrom L, Turner A, Madry A. Robustness may be at odds with accuracy. In: International Conference on Learning Representations (ICLR) 2018.
Pinot R, Meunier L, Araujo A, Kashima H, Yger F, Gouy-Pailler C, Atif J. Theoretical evidence for adversarial robustness through randomization. In: Advances in Neural Information Processing Systems (NeurIPS) 2019.
Ilyas A, Santurkar S, Tsipras D, Engstrom L, Tran B, Madry A. Adversarial examples are not bugs, they are features. In: Advances in Neural Information Processing Systems (NeurIPS) 2019.
Cha S-H. Comprehensive survey on distance/similarity measures between probability density functions. Int J Math Models Methods Appl Sci. 2007;1(4):300–7.
Xie C, Tan M, Gong B, Wang J, Yuille AL, Le QV. Adversarial examples improve image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. p. 819–828.
Jordan M, Manoj N, Goel S, Dimakis AG. Quantifying perceptual distortion of adversarial examples. arXiv preprint arXiv:1902.08265 2019.