Human selection bias drives the linear nature of the more ground truth effect in explainable deep learning optical coherence tomography image segmentation.
explainable AI
machine learning
optical coherence tomography
retina
Journal
Journal of biophotonics
ISSN: 1864-0648
Titre abrégé: J Biophotonics
Pays: Germany
ID NLM: 101318567
Informations de publication
Date de publication:
05 Oct 2023
05 Oct 2023
Historique:
revised:
11
09
2023
received:
15
07
2023
accepted:
04
10
2023
pubmed:
5
10
2023
medline:
5
10
2023
entrez:
5
10
2023
Statut:
aheadofprint
Résumé
Supervised deep learning (DL) algorithms are highly dependent on training data for which human graders are assigned, for example, for optical coherence tomography (OCT) image annotation. Despite the tremendous success of DL, due to human judgment, these ground truth labels can be inaccurate and/or ambiguous and cause a human selection bias. We therefore investigated the impact of the size of the ground truth and variable numbers of graders on the predictive performance of the same DL architecture and repeated each experiment three times. The largest training dataset delivered a prediction performance close to that of human experts. All DL systems utilized were highly consistent. Nevertheless, the DL under-performers could not achieve any further autonomous improvement even after repeated training. Furthermore, a quantifiable linear relationship between ground truth ambiguity and the beneficial effect of having a larger amount of ground truth data was detected and marked as the more-ground-truth effect.
Identifiants
pubmed: 37795556
doi: 10.1002/jbio.202300274
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
e202300274Subventions
Organisme : Roche, Basel, Switzerland
Informations de copyright
© 2023 Wiley-VCH GmbH.
Références
M. R. Hee, J. A. Izatt, E. A. Swanson, D. Huang, J. S. Schuman, C. P. Lin, C. A. Puliafito, J. G. Fujimoto, Arch. Ophthalmol. 1995, 113, 325.
J. Fujimoto, E. Swanson, Invest. Ophthalmol. Vis. Sci. 2016, 57, OCT1.
A. Geevarghese, G. Wollstein, H. Ishikawa, J. S. Schuman, Annu. Rev. Vis. Sci. 2021, 7, 693.
K. M. Litts, Y. Zhang, K. B. Freund, C. A. Curcio, Retina 2018, 38, 445.
S. M. Waldstein, W. D. Vogl, H. Bogunovic, A. Sadeghipour, S. Riedl, U. Schmidt-Erfurth, JAMA Ophthalmol. 2020, 138, 740.
Z. Sun, D. Yang, Z. Tang, D. S. Ng, C. Y. Cheung, Eye 2021, 35, 149.
S. K. Johannesen, J. N. Viken, A. S. Vergmann, J. Grauslund, Acta Ophthalmol. 2019, 97, 7.
J. Fujimoto, D. Huang, Invest. Ophthalmol. Vis. Sci. 2016, 57, OCTi.
T. D. Wibbelsman, R. R. Pandit, D. Xu, T. L. Jenkins, P. L. Mellen, R. R. Soares, A. Obeid, H. Levin, J. Hsu, A. C. Ho, Am J. Ophthalmol. 2019, 208, 12.
T. D. Wibbelsman, R. R. Pandit, D. Xu, T. L. Jenkins, P. L. Mellen, R. R. Soares, A. Obeid, H. Levin, J. Hsu, A. C. Ho, Am. J. Ophthalmol. 2020, 211, 229.
M. Wilson, R. Chopra, M. Z. Wilson, C. Cooper, P. MacWilliams, Y. Liu, E. Wulczyn, D. Florea, C. O. Hughes, A. Karthikesalingam, H. Khalid, S. Vermeirsch, L. Nicholson, P. A. Keane, K. Balaskas, C. J. Kelly, JAMA Ophthalmol. 2021, 139, 964.
L. Zhou, Q. Li, G. Huo, Y. Zhou, Comput. Intell. Neurosci. 2017, 2017, 3792805.
B. Liefers, P. Taylor, A. Alsaedi, C. Bailey, K. Balaskas, N. Dhingra, C. A. Egan, F. G. Rodrigues, C. G. Gonzalo, T. F. C. Heeren, A. Lotery, P. L. Müller, A. Olvera-Barrios, B. Paul, R. Schwartz, D. S. Thomas, A. N. Warwick, A. Tufail, C. I. Sánchez, Am. J. Ophthalmol. 2021, 226, 1.
S. Klimscha, S. M. Waldstein, T. Schlegl, H. Bogunovic, A. Sadeghipour, A. M. Philip, D. Podkowinski, E. Pablik, L. Zhang, M. D. Abramoff, M. Sonka, B. S. Gerendas, U. Schmidt-Erfurth, Invest. Ophthalmol. Vis. Sci. 2017, 58, 4039.
P. M. Maloca, A. Y. Lee, E. R. de Carvalho, M. Okada, K. Fasler, I. Leung, B. Hörmann, P. Kaiser, S. Suter, P. W. Hasler, J. Zarranz-Ventura, C. Egan, T. F. C. Heeren, K. Balaskas, A. Tufail, H. P. N. Scholl, PLoS One 2019, 14, e0220063.
O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, Springer International Publishing, Cham 2015.
C. S. Lee, D. M. Baughman, A. Y. Lee, Ophthalmol. Retina 2017, 1, 322.
D. S. W. Ting, L. R. Pasquale, L. Peng, J. P. Campbell, A. Y. Lee, R. Raman, G. S. W. Tan, L. Schmetterer, P. A. Keane, T. Y. Wong, Br. J. Ophthalmol. 2019, 103, 167.
E. Tjoa, C. Guan, IEEE Trans. Neural Netw. Learn. Syst. 2021, 32, 4793.
A. Holzinger, C. Biemann, C. S. Pattichis, D. B. Kell, ArXiv 2017, 1712.09923, abs/1712.09923.
W. Samek, K.-R. Müller, Towards Explainable Artificial Intelligence. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer, Cham, 2019, p. 5.
P. M. Maloca, P. L. Müller, A. Y. Lee, A. Tufail, K. Balaskas, S. Niklaus, P. Kaiser, S. Suter, J. Zarranz-Ventura, C. Egan, H. P. N. Scholl, T. K. Schnitzer, T. Singer, P. W. Hasler, N. Denk, Commun. Biol. 2021, 4, 170.
S. Riedl, W. D. Vogl, S. M. Waldstein, U. Schmidt-Erfurth, H. Bogunović, Ophthalmol. Retina 2022, 6, 291.
M. Baker, Nature 2016, 533, 452.
T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Springer, New York 2009.
C. J. Brady, L. I. Mudie, X. Wang, E. Guallar, D. S. Friedman, J. Med. Internet Res. 2017, 19, e222.
M. Petroff, ArXiv 2021 abs/2107.02270.
J. De Fauw, J. R. Ledsam, B. Romera-Paredes, S. Nikolov, N. Tomasev, S. Blackwell, H. Askham, X. Glorot, B. O'Donoghue, D. Visentin, G. van den Driessche, B. Lakshminarayanan, C. Meyer, F. Mackinder, S. Bouton, K. Ayoub, R. Chopra, D. King, A. Karthikesalingam, C. O. Hughes, R. Raine, J. Hughes, D. A. Sim, C. Egan, A. Tufail, H. Montgomery, D. Hassabis, G. Rees, T. Back, P. T. Khaw, M. Suleyman, J. Cornebise, P. A. Keane, O. Ronneberger, Nat. Med. 2018, 24, 1342.
B. S. Gerendas, H. Bogunovic, U. Schmidt-Erfurth, JAMA Ophthalmol. 2021, 139, 973.
T. Panch, H. Mattie, L. A. Celi, npj Digit. Med. 2019, 2, 77.
A. Grzybowski, P. Brona, J. Clin. Med. 2021, 10, 2352.
F. Renard, S. Guedria, N. Palma, N. Vuillerme, Sci. Rep. 2020, 10, 13724.
D. Gunning, Explainable Artificial Intelligence (xAI). Technical Report, Defense Advanced Research Projects Agency (DARPA), Washington, 2017.
A. Holzinger, P. Kieseberg, E. Weippl, A. M. Tjoa, Springer Lecture Notes in Computer Science LNCS 2018, 11015, 1.
M. E. Matheny, D. Whicher, I. S. Thadaney, JAMA 2020, 323, 509.
J. S. Armstrong, in Principles of Forecasting: A Handbook for Researchers and Practitioners (Ed: J. S. Armstrong), Springer US, Boston, MA 2001, p. 417.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, J. Mach. Learn. Res. 2020, 21, 1.
C. Wang, Y. Tang, X. Ma, A. Wu, D. Okhonko, J. M. Pino, Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq, AACL, Suzhou, 2020.
Y. Jia, R. J. Weiss, F. Biadsy, W. Macherey, M. Johnson, Z. Chen, Y. Wu, Interspeech 2019, arXiv, 1904.06037 2019.
S. Kapoor, A. Narayanan, arXiv, 2022, arXiv, 2207.07048 2022.
National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science, The National Academies Press, Washington, DC 2019.
C. M. Bishop, N. M. Nasrabadi, J. Electron. Imag. 2007, 16, 049901.
T. J. Hastie, R. Tibshirani, J. H. Friedman Eds., The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Springer Series in Statistics, New York, 2001.