Optic Disc Classification by Deep Learning versus Expert Neuro-Ophthalmologists.
Journal
Annals of neurology
ISSN: 1531-8249
Titre abrégé: Ann Neurol
Pays: United States
ID NLM: 7707449
Informations de publication
Date de publication:
10 2020
10 2020
Historique:
received:
27
04
2020
revised:
30
06
2020
accepted:
02
07
2020
pubmed:
6
7
2020
medline:
15
12
2020
entrez:
5
7
2020
Statut:
ppublish
Résumé
To compare the diagnostic performance of an artificial intelligence deep learning system with that of expert neuro-ophthalmologists in classifying optic disc appearance. The deep learning system was previously trained and validated on 14,341 ocular fundus photographs from 19 international centers. The performance of the system was evaluated on 800 new fundus photographs (400 normal optic discs, 201 papilledema [disc edema from elevated intracranial pressure], 199 other optic disc abnormalities) and compared with that of 2 expert neuro-ophthalmologists who independently reviewed the same randomly presented images without clinical information. Area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity were calculated. The system correctly classified 678 of 800 (84.7%) photographs, compared with 675 of 800 (84.4%) for Expert 1 and 641 of 800 (80.1%) for Expert 2. The system yielded areas under the receiver operating characteristic curve of 0.97 (95% confidence interval [CI] = 0.96-0.98), 0.96 (95% CI = 0.94-0.97), and 0.89 (95% CI = 0.87-0.92) for the detection of normal discs, papilledema, and other disc abnormalities, respectively. The accuracy, sensitivity, and specificity of the system's classification of optic discs were similar to or better than the 2 experts. Intergrader agreement at the eye level was 0.71 (95% CI = 0.67-0.76) between Expert 1 and Expert 2, 0.72 (95% CI = 0.68-0.76) between the system and Expert 1, and 0.65 (95% CI = 0.61-0.70) between the system and Expert 2. The performance of this deep learning system at classifying optic disc abnormalities was at least as good as 2 expert neuro-ophthalmologists. Future prospective studies are needed to validate this system as a diagnostic aid in relevant clinical settings. ANN NEUROL 2020;88:785-795.
Types de publication
Journal Article
Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
785-795Subventions
Organisme : NIH/NEI core grant
ID : P30-EY06360
Pays : International
Investigateurs
Barnabé Rondé-Courbis
(B)
Philippe Gohier
(P)
Valérie Biousse
(V)
Nancy J Newman
(NJ)
Caroline Vasseneix
(C)
Neil Miller
(N)
Tanyatuth Padungkiatsagul
(T)
Anuchit Poonyathalang
(A)
Yanin Suwan
(Y)
Kavin Vanikieti
(K)
Leonard B Milea
(LB)
Giulia Amore
(G)
Piero Barboni
(P)
Michele Carbonelli
(M)
Valerio Carelli
(V)
Chiara La Morgia
(C)
Martina Romagnoli
(M)
Marie-Bénédicte Rougier
(MB)
Selvakumar Ambika
(S)
Swetha Komma
(S)
Pedro Fonseca
(P)
Miguel Raimundo
(M)
Steffen Hamann
(S)
Isabelle Karlesand
(I)
Wolf Alexander Lagrèze
(W)
Nicolae Sanda
(N)
Gabriele Thumann
(G)
Florent Aptel
(F)
Christophe Chiquet
(C)
Kaiqun Liu
(K)
Hui Yang
(H)
Carmen Km Chan
(CK)
Noel Cy Chan
(NC)
Carol Y Cheung
(CY)
Thi Ha Chau Tran
(TH)
James Acheson
(J)
Maged S Habib
(MS)
Neringa Jurkute
(N)
Patrick Yu-Wai-Man
(P)
Richard Kho
(R)
Jost B Jonas
(JB)
John J Chen
(JJ)
Nouran Sabbagh
(N)
Catherine Vignal-Clermont
(C)
Rabih Hage
(R)
Raoul K Khanna
(RK)
Jeong-Min Hwang
(JM)
Dong Hyun Kim
(DH)
Hee Kyung Yang
(HK)
Tin Aung
(T)
Ching-Yu Cheng
(CY)
Ecosse Lamoureux
(E)
Jing Liang Loo
(JL)
Dan Milea
(D)
Raymond P Najjar
(RP)
Shweta Singhal
(S)
Daniel Ting
(D)
Sharon Tow
(S)
Caroline Vasseneix
(C)
Tien Yin Wong
(TY)
Yong Liu
(Y)
Xinxing Xu
(X)
Zhubo Jiang
(Z)
Clare L Fraser
(CL)
Luis J Mejico
(LJ)
Masoud Aghsaei Fard
(MA)
Informations de copyright
© 2020 American Neurological Association.
Références
Biousse V, Bruce BB, Newman NJ. Ophthalmoscopy in the 21st century: the 2017 H. Houston Merritt lecture. Neurology 2018;90:167-175.
Stunkel L, Newman-Toker DE, Newman NJ, Biousse V. Diagnostic error of neuro-ophthalmologic conditions-state of the science. J Neuroophthalmol (in press).
Bruce BB, Lamirel C, Wright DW, et al. Nonmydriatic ocular fundus photography in the emergency department. N Engl J Med 2011;364:387-389.
Mackay DD, Garza PS, Bruce BB, et al. The demise of direct ophthalmoscopy: a modern clinical challenge. Neurol Clin Pract 2015;5:150-157.
Golombievski E, Doerrler MW, Ruland SD, et al. Frequency of direct funduscopy upon initial encounters for patients with headaches, altered mental status, and visual changes: a pilot study. Front Neurol 2015;6:233.
Bruce BB, Thulasi P, Fraser CL, et al. Diagnostic accuracy and use of nonmydriatic ocular fundus photography by emergency physicians: phase II of the FOTO-ED study. Ann Emerg Med 2013;62:28-33.
Bruce BB, Bidot S, Hage R, et al. Fundus photography vs. ophthalmoscopy outcomes in the emergency department (FOTO-ED) phase III: web-based, in-service training of emergency providers. Neuroophthalmology 2018;42:269-274.
Toffoli D, Bruce BB, Lamirel C, et al. Feasibility and quality of nonmydriatic fundus photography in children. J AAPOS 2011;15:567-572.
Ivan Y, Ramgopal S, Cardenas-Villa M, et al. Feasibility of the digital retinography system camera in the pediatric emergency department. Pediatr Emerg Care 2018;34:488-491.
Zafar S, Cardenas YM, Leishangthem L, Yaddanapudi S. Opinion and special articles: amateur fundus photography with various new devices: our experience as neurology residents. Neurology 2018;90:897-901.
Irani NK, Bidot S, Peragallo JH, et al. Feasibility of a nonmydriatic ocular fundus camera in an outpatient neurology clinic. Neurologist 2020;25:19-23.
Rathi S, Tsui E, Mehta N, et al. The current state of teleophthalmology in the United States. Ophthalmology 2017;124:1729-1734.
Wedekind L, Sainani K, Pershing S. Supply and perceived demand for teleophthalmology in triage and consultations in California emergency departments. JAMA Ophthalmol 2016;134:537-543.
Woodward MA, Ple-Plakon P, Blachley T, et al. Eye care providers' attitudes towards tele-ophthalmology. Telemed J E Health 2015;21:271-273.
Vasseneix C, Bruce BB, Bidot S, et al. Nonmydriatic fundus photography in patients with acute vision loss. Telemed e-Health 2019;25:911-916.
Ohannessian R, Duong TA, Odone A. Global telemedicine implementation and integration within health systems to fight the COVID-19 pandemic: a call to action. JMIR Public Health Surveill 2020;6:e18810.
Bloem BR, Dorsey ER, Okun MS. The coronavirus disease 2019 crisis as catalyst for telemedicine for chronic neurological disorders. JAMA Neurol (2020). https://doi.org/10.1001/jamaneurol.2020.1452.
Ting DSW, Carin L, Dzau V, Wong TY. Digital technology and COVID-19. Nat Med 2020;26:459-461.
Milea D, Najjar RP, Zhubo J, et al. Artificial intelligence to detect papilledema from ocular fundus photographs. N Engl J Med 2020;382:1687-1695.
Biousse V, Newman NJ. Diagnosis and clinical features of common optic neuropathies. Lancet Neurol 2016;15:1355-1367.
Milea L, Najjar RP. Classif-Eye: a semi-automated image classification application. 2020. GitHub repository. Available at: https://github.com/milealeonard/Classif-Eye/. Accessed April 13, 2020.
McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 1947;12:153-157.
Gu W, Pepe MS. Estimating the capacity for improvement in risk prediction with a marker. Biostatistics 2009;10:172-186.
McHugh ML. Interrater reliability: the kappa statistic. Biochem Med 2012;22:276-282.
Simel D, Samsa G, Matchar D. Likelihood ratios with confidence: sample size estimation for diagnostic test studies. J Clin Epidemiol 1991;44:763-770.
Trobe JD. Papilledema: the vexing issues. J Neuroophthalmol 2011;31:175-186.
Sachdeva V, Vasseneix C, Hage R, et al. Optic nerve head edema among patients presenting to the emergency department. Neurology 2018;90:e373-e379.
Rigi M, Almarzouqi SJ, Morgan ML, Lee AG. Papilledema: epidemiology, etiology, and clinical management. Eye Brain 2015;7:47-57.
Tabatabai RR, Swadron SP. Headache in the emergency department: avoiding misdiagnosis of dangerous secondary causes. Emerg Med Clin 2016;34:695-716.
Poostchi A, Awad M, Wilde C, et al. Spike in neuroimaging requests following the conviction of the optometrist honey rose. Eye 2018;32:489-490.
Medical Advisory Secretariat. Neuroimaging for the evaluation of chronic headaches: an evidence-based analysis. Ont Health Technol Assess Ser 2010;10:1-57.
American College of Radiology. ACR appropriateness criteria. Revised 2019. Available at: https://acsearch.acr.org/docs/69482/Narrative/. Accessed April 13, 2020.
Bruce BB, Lamirel C, Biousse V, et al. Feasibility of nonmydriatic ocular fundus photography in the emergency department: phase I of the FOTO-ED study. Acad Emerg Med 2011;18:928-933.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436-444.
Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med 2019;380:1347-1358.
Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25:44-56.
Liu X, Faes L, Kale AU, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health 2019;1:e271-e297.
Ting DSW, Peng L, Varadarajan AV, et al. Deep learning in ophthalmology: the technical and clinical considerations. Prog Retin Eye Res 2019;72:100759.
Keane PA, Topol EJ. With an eye to AI and autonomous diagnosis. NPJ Digit Med 2018;1:40.
Keane P, Topol EJ. Reinventing the eye exam. Lancet 2019;394:2141.
Milea D, Singhal S, Najjar RP. Artificial intelligence for detection of optic disc abnormalities. Curr Opin Neurol 2020;33:106-110.
Lamirel C, Bruce BB, Wright DW, et al. Quality of nonmydriatic digital fundus photography obtained by nurse practitioners in the emergency department: the FOTO-ED study. Ophthalmology 2012;119:617-624.
Stunkel L, Mackay DD, Bruce BB, et al. Referral patterns in neuro-ophthalmology. J Neuroophthalmol (2011).