Deep Learning System Outperforms Clinicians in Identifying Optic Disc Abnormalities.
Journal
Journal of neuro-ophthalmology : the official journal of the North American Neuro-Ophthalmology Society
ISSN: 1536-5166
Titre abrégé: J Neuroophthalmol
Pays: United States
ID NLM: 9431308
Informations de publication
Date de publication:
01 06 2023
01 06 2023
Historique:
medline:
17
5
2023
pubmed:
1
2
2023
entrez:
31
1
2023
Statut:
ppublish
Résumé
The examination of the optic nerve head (optic disc) is mandatory in patients with headache, hypertension, or any neurological symptoms, yet it is rarely or poorly performed in general clinics. We recently developed a brain and optic nerve study with artificial intelligence-deep learning system (BONSAI-DLS) capable of accurately detecting optic disc abnormalities including papilledema (swelling due to elevated intracranial pressure) on digital fundus photographs with a comparable classification performance to expert neuro-ophthalmologists, but its performance compared to first-line clinicians remains unknown. In this international, cross-sectional multicenter study, the DLS, trained on 14,341 fundus photographs, was tested on a retrospectively collected convenience sample of 800 photographs (400 normal optic discs, 201 papilledema and 199 other abnormalities) from 454 patients with a robust ground truth diagnosis provided by the referring expert neuro-ophthalmologists. The areas under the receiver-operating-characteristic curves were calculated for the BONSAI-DLS. Error rates, accuracy, sensitivity, and specificity of the algorithm were compared with those of 30 clinicians with or without ophthalmic training (6 general ophthalmologists, 6 optometrists, 6 neurologists, 6 internists, 6 emergency department [ED] physicians) who graded the same testing set of images. With an error rate of 15.3%, the DLS outperformed all clinicians (average error rates 24.4%, 24.8%, 38.2%, 44.8%, 47.9% for general ophthalmologists, optometrists, neurologists, internists and ED physicians, respectively) in the overall classification of optic disc appearance. The DLS displayed significantly higher accuracies than 100%, 86.7% and 93.3% of clinicians (n = 30) for the classification of papilledema, normal, and other disc abnormalities, respectively. The performance of the BONSAI-DLS to classify optic discs on fundus photographs was superior to that of clinicians with or without ophthalmic training. A trained DLS may offer valuable diagnostic aid to clinicians from various clinical settings for the screening of optic disc abnormalities harboring potentially sight- or life-threatening neurological conditions.
Sections du résumé
BACKGROUND
The examination of the optic nerve head (optic disc) is mandatory in patients with headache, hypertension, or any neurological symptoms, yet it is rarely or poorly performed in general clinics. We recently developed a brain and optic nerve study with artificial intelligence-deep learning system (BONSAI-DLS) capable of accurately detecting optic disc abnormalities including papilledema (swelling due to elevated intracranial pressure) on digital fundus photographs with a comparable classification performance to expert neuro-ophthalmologists, but its performance compared to first-line clinicians remains unknown.
METHODS
In this international, cross-sectional multicenter study, the DLS, trained on 14,341 fundus photographs, was tested on a retrospectively collected convenience sample of 800 photographs (400 normal optic discs, 201 papilledema and 199 other abnormalities) from 454 patients with a robust ground truth diagnosis provided by the referring expert neuro-ophthalmologists. The areas under the receiver-operating-characteristic curves were calculated for the BONSAI-DLS. Error rates, accuracy, sensitivity, and specificity of the algorithm were compared with those of 30 clinicians with or without ophthalmic training (6 general ophthalmologists, 6 optometrists, 6 neurologists, 6 internists, 6 emergency department [ED] physicians) who graded the same testing set of images.
RESULTS
With an error rate of 15.3%, the DLS outperformed all clinicians (average error rates 24.4%, 24.8%, 38.2%, 44.8%, 47.9% for general ophthalmologists, optometrists, neurologists, internists and ED physicians, respectively) in the overall classification of optic disc appearance. The DLS displayed significantly higher accuracies than 100%, 86.7% and 93.3% of clinicians (n = 30) for the classification of papilledema, normal, and other disc abnormalities, respectively.
CONCLUSIONS
The performance of the BONSAI-DLS to classify optic discs on fundus photographs was superior to that of clinicians with or without ophthalmic training. A trained DLS may offer valuable diagnostic aid to clinicians from various clinical settings for the screening of optic disc abnormalities harboring potentially sight- or life-threatening neurological conditions.
Identifiants
pubmed: 36719740
doi: 10.1097/WNO.0000000000001800
pii: 00041327-202306000-00003
doi:
Types de publication
Multicenter Study
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
159-167Investigateurs
Philippe Gohier
(P)
Neil Miller
(N)
Kavin Vanikieti
(K)
Chiara La Morgia
(C)
Marie-Bénédicte Rougier
(MB)
Selvakumar Ambika
(S)
Pedro Fonseca
(P)
Wolf Alexander Lagrèze
(WA)
Nicolae Sanda
(N)
Christophe Chiquet
(C)
Hui Yang
(H)
Carmen K M Chan
(CKM)
Carol Y Cheung
(CY)
Tran Thi Ha Chau
(TT)
Neringa Jurkute
(N)
Patrick Yu-Wai-Man
(P)
Richard Kho
(R)
Jost B Jonas
(JB)
Catherine Vignal-Clermont
(C)
Dong Hyun Kim
(DH)
Hee Kyung Yang
(HK)
Tin Aung
(T)
Shweta Singhal
(S)
Sharon Tow
(S)
Monisha Esther Nongpiur
(ME)
Shamira Perera
(S)
Arun Narayanaswamy
(A)
Umapathi N Thirugnanam
(UN)
Clare L Fraser
(CL)
Luis J Mejico
(LJ)
Masoud Aghsaei Fard
(MA)
Informations de copyright
Copyright © 2023 by North American Neuro-Ophthalmology Society.
Déclaration de conflit d'intérêts
The authors report no conflicts of interest.
Références
Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, Tan GSW, Schmetterer L, Keane PA, Wong TY. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. 2019;103:167–175.
Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, Mahendiran T, Moraes G, Shamdas M, Kern C, Ledsam JR, Schmid MK, Balaskas K, Topol EJ, Bachmann LM, Keane PA, Denniston AK. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digital Health. 2019;1:e271–e297.
Nagendran M, Chen Y, Lovejoy CA, Gordon AC, Komorowski M, Harvey H, Topol EJ, Ioannidis JPA, Collins GS, Maruthappu M. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ. 2020;368:m689.
Bruce BB, Lamirel C, Wright DW, Ward A, Heilpern KL, Biousse V, Newman NJ. Nonmydriatic ocular fundus photography in the emergency department. N Engl J Med. 2011;364:387–389.
Biousse V, Bruce BB, Newman NJ. Ophthalmoscopy in the 21st century: the 2017 H. Houston Merritt lecture. Neurology. 2018;90:167–175.
Biousse V, Newman NJ. Diagnosis and clinical features of common optic neuropathies. Lancet Neurol. 2016;15:1355–1367.
Bruce BB, Bidot S, Hage R, Clough LC, Fajoles-Vasseneix C, Melomed M, Keadey MT, Wright DW, Newman NJ, Biousse V. Fundus photography vs. ophthalmoscopy outcomes in the emergency department (FOTO-ED) phase III: web-based, in-service training of emergency providers. Neuroophthalmol. 2018;42:269–274.
Rathi S, Tsui E, Mehta N, Zahid S, Schuman JS. The current state of teleophthalmology in the United States. Ophthalmology. 2017;124:1729–1734.
Milea D, Najjar RP, Zhubo J, Ting D, Vasseneix C, Xu X, Aghsaei Fard M, Fonseca P, Vanikieti K, Lagreze WA, La Morgia C, Cheung CY, Hamann S, Chiquet C, Sanda N, Yang H, Mejico LJ, Rougier MB, Kho R, Tran TH, Singhal S, Gohier P, Clermont-Vignal C, Cheng CY, Jonas JB, Yu-Wai-Man P, Fraser CL, Chen JJ, Ambika S, Miller NR, Liu Y, Newman NJ, Wong TY, Biousse V. Artificial intelligence to detect papilledema from ocular fundus photographs. N Engl J Med. 2020;382:1687–1695.
Biousse V, Newman NJ, Najjar RP, Vasseneix C, Xu X, Ting DS, Milea LB, Hwang J, Kim DH, Yang HK, Hamann S, Chen JJ, Liu Y, Wong TY, Milea D, Ronde‐Courbis B, Gohier P, Biousse V, Newman NJ, Vasseneix C, Miller N, Padungkiatsagul T, Poonyathalang A, Suwan Y, Vanikieti K, Milea LB, Amore G, Barboni P, Carbonelli M, Carelli V, La Morgia C, Romagnoli M, Rougier M, Ambika S, Komma S, Fonseca P, Raimundo M, Hamann S, Karlesand I, Alexander Lagreze W, Sanda N, Thumann G, Aptel F, Chiquet C, Liu K, Yang H, Chan CK, Chan NC, Cheung CY, Chau Tran TH, Acheson J, Habib MS, Jurkute N, Yu‐Wai‐Man P, Kho R, Jonas JB, Chen JJ, Sabbagh N, Vignal‐Clermont C, Hage R, Khanna RK, Hwang J, Kim DH, Yang HK, Aung T, Cheng C, Lamoureux E, Loo JL, Milea D, Najjar RP, Singhal S, Ting D, Tow S, Vasseneix C, Wong TY, Liu Y, Xu X, Jiang Z, Fraser CL, Mejico LJ, Fard MA; for the BONSAI Brain and Optic Nerve Study with Artificial Intelligence Study Group. Optic disc classification by deep learning versus expert neuro‐ophthalmologists. Ann Neurol. 2020;88:785–795.
Milea D, Singhal S, Najjar RP. Artificial intelligence for detection of optic disc abnormalities. Curr Opin Neurol. 2020;33:106–110.
Milea L, Najjar RP. Classif-Eye: A Semi-automated Image Classification Application, 2020. GitHub repository. Available at: https://github.com/milealeonard/Classif-Eye/ . Accessed April 13, 2020.
McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 1947;12:153–157.
Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digital Med. 2018;1:39.
Rim TH, Lee G, Kim Y, Tham YC, Lee CJ, Baik SJ, Kim YA, Yu M, Deshmukh M, Lee BK, Park S, Kim HC, Sabayanagam C, Ting DSW, Wang YX, Jonas JB, Kim SS, Wong TY, Cheng CY. Prediction of systemic biomarkers from retinal photographs: development and validation of deep-learning algorithms. Lancet Digital Health. 2020;2:e526–e536.
Sachdeva V, Vasseneix C, Hage R, Bidot S, Clough LC, Wright DW, Newman NJ, Biousse V, Bruce BB. Optic nerve head edema among patients presenting to the emergency department. Neurology. 2018;90:e373–e379.
Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44–56.
Jammal AA, Thompson AC, Mariottoni EB, Berchuck SI, Urata CN, Estrela T, Wakil SM, Costa VP, Medeiros FA. Human versus machine: comparing a deep learning algorithm to human gradings for detecting glaucoma on fundus photographs. Am J Ophthalmol. 2020;211:123–131.
De Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, Askham H, Glorot X, O'Donoghue B, Visentin D, van den Driessche G, Lakshminarayanan B, Meyer C, Mackinder F, Bouton S, Ayoub K, Chopra R, King D, Karthikesalingam A, Hughes CO, Raine R, Hughes J, Sim DA, Egan C, Tufail A, Montgomery H, Hassabis D, Rees G, Back T, Khaw PT, Suleyman M, Cornebise J, Keane PA, Ronneberger O. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018;24:1342–1350.
Brown JM, Campbell JP, Beers A, Chang K, Ostmo S, Chan RVP, Dy J, Erdogmus D, Ioannidis S, Kalpathy-Cramer J, Chiang MF; for the Imaging and Informatics in Retinopathy of Prematurity i-ROP Research Consortium. Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmol. 2018;136:803–810.