FDA-approved deep learning software application versus radiologists with different levels of expertise: detection of intracranial hemorrhage in a retrospective single-center study.
Artificial intelligence
Computed tomography
Deep learning
Diagnostic accuracy
Intracranial hemorrhage
Journal
Neuroradiology
ISSN: 1432-1920
Titre abrégé: Neuroradiology
Pays: Germany
ID NLM: 1302751
Informations de publication
Date de publication:
May 2022
May 2022
Historique:
received:
26
08
2021
accepted:
01
12
2021
pubmed:
7
1
2022
medline:
15
4
2022
entrez:
6
1
2022
Statut:
ppublish
Résumé
To assess an FDA-approved and CE-certified deep learning (DL) software application compared to the performance of human radiologists in detecting intracranial hemorrhages (ICH). Within a 20-week trial from January to May 2020, 2210 adult non-contrast head CT scans were performed in a single center and automatically analyzed by an artificial intelligence (AI) solution with workflow integration. After excluding 22 scans due to severe motion artifacts, images were retrospectively assessed for the presence of ICHs by a second-year resident and a certified radiologist under simulated time pressure. Disagreements were resolved by a subspecialized neuroradiologist serving as the reference standard. We calculated interrater agreement and diagnostic performance parameters, including the Breslow-Day and Cochran-Mantel-Haenszel tests. An ICH was present in 214 out of 2188 scans. The interrater agreement between the resident and the certified radiologist was very high (κ = 0.89) and even higher (κ = 0.93) between the resident and the reference standard. The software has delivered 64 false-positive and 68 false-negative results giving an overall sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of 68.2%, 96.8%, 69.5%, 96.6%, and 94.0%, respectively. Corresponding values for the resident were 94.9%, 99.2%, 93.1%, 99.4%, and 98.8%. The accuracy of the DL application was inferior (p < 0.001) to that of both the resident and the certified neuroradiologist. A resident under time pressure outperformed an FDA-approved DL program in detecting ICH in CT scans. Our results underline the importance of thoughtful workflow integration and post-approval validation of AI applications in various clinical environments.
Identifiants
pubmed: 34988593
doi: 10.1007/s00234-021-02874-w
pii: 10.1007/s00234-021-02874-w
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
981-990Informations de copyright
© 2021. The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.
Références
Zaharchuk G, Gong E, Wintermark M et al (2018) Am J Neuroradiol 39:1776–1784
doi: 10.3174/ajnr.A5543
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444
doi: 10.1038/nature14539
Chan HP, Samala RK, Hadjiiski LM et al (2020) Deep learning in medical image analysis. Adv Exp Med Biol 1213:3–21
doi: 10.1007/978-3-030-33128-3_1
Banja J (2020) AI hype and radiology: a plea for realism and accuracy. Radiol Artif Intell 2:e190223
doi: 10.1148/ryai.2020190223
Desai AN (2020) Artificial intelligence: promise, pitfalls, and perspective. JAMA 323:2448–2449
doi: 10.1001/jama.2020.8737
Rockall A (2020) From hype to hope to hard work: developing responsible AI for radiology. Clin Radiol 75:1–2
doi: 10.1016/j.crad.2019.09.123
Bluemke DA, Moy L, Bredella MA et al (2020) Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers – from the Radiology editorial board. Radiology 294:487–489
doi: 10.1148/radiol.2019192515
Nagendran M, Chen Y, Lovejoy CA et al (2020) Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ 368:m689
doi: 10.1136/bmj.m689
Bruls RJM, Kwee RM (2020) Workload for radiologists during on-call hours: dramatic increase in the past 15 years. Insights Imaging 11:121
doi: 10.1186/s13244-020-00925-z
Bruno MA, Duncan JR, Bierhals AJ, Tappouni R (2018) Overnight resident versus 24-hour attending radiologist coverage in academic medical centers. Radiology 289:809–813
doi: 10.1148/radiol.2018180690
Cooper VF, Goodhartz LA, Nemcek AA Jr, Ryu RK (2008) Radiology resident interpretations of on-call imaging studies: the incidence of major discrepancies. Acad Radiol 15:1198–1204
doi: 10.1016/j.acra.2008.02.011
Mellnick V, Raptis C, McWilliams S, Picus D, Wahl R (2016) On-call radiology resident discrepancies: categorization by patient location and severity. J Am Coll Radiol 13:1233–1238
doi: 10.1016/j.jacr.2016.04.020
Terreblanche OD, Andronikou S, Hlabangana LT, Brown T, Boshoff PE (2012) Should registrars be reporting after-hours CT scans? A calculation of error rate and the influencing factors in South Africa. Acta Radiol 53:61–68
doi: 10.1258/ar.2011.110103
Ruutiainen AT, Durand DJ, Scanlon MH, Itri JN (2013) Increased error rates in preliminary reports issued by radiology residents working more than 10 consecutive hours overnight. Acad Radiol 20:305–311
doi: 10.1016/j.acra.2012.09.028
van Asch CJJ, Luitse MJA, Rinkel GJE, van der Tweel I, Algra A, Klijn CJM (2010) Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis. Lancet Neurol 9:167–176
doi: 10.1016/S1474-4422(09)70340-0
Steiner T, Al-Shahi Salman R, Beer R, Christensen H et al (2014) European Stroke Organisation (ESO) guidelines for the management of spontaneous intracerebral hemorrhage. Int J Stroke 9:840–855
doi: 10.1111/ijs.12309
Chilamkurthy S, Ghosh R, Tanamala S et al (2018) Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet 392:2388–2396
doi: 10.1016/S0140-6736(18)31645-3
Ginat DT (2020) Analysis of head CT scans flagged by deep learning software for acute intracranial hemorrhage. Neuroradiology 62:335–340
doi: 10.1007/s00234-019-02330-w
O’Neill TJ, Xi Y, Stehel E et al (2020) Active reprioritization of the reading worklist using artificial intelligence has a beneficial effect on the turnaround time for interpretation of head CT with intracranial hemorrhage. Radiol Artif Intell 3(2):e200024. https://doi.org/10.1148/ryai.2020200024
doi: 10.1148/ryai.2020200024
pubmed: 33937858
pmcid: 8043365
Ojeda P, Zawaideh M, Mossa-Basha M, Haynor D (2019) The utility of deep learning: evaluation of a convolutional neural network for detection of intracranial bleeds on non-contrast head computed tomography studies. Proc. SPIE 10949, Medical Imaging 2019: Image Processing, 109493J. https://doi.org/10.1117/12.2513167
Arbabshirani MR, Fornwalt BK, Mongelluzzo GJ et al (2018) Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. NPJ Digit Med 1:9
doi: 10.1038/s41746-017-0015-z
Voter AF, Meram E, Garrett JW, Yu JJ (2021) Diagnostic accuracy and failure mode analysis of a deep learning algorithm for the detection of intracranial hemorrhage. J Am Coll Radiol S1546–1440(21):00227–00231
Bossuyt PM, Reitsma JB, Bruns DE et al (2015) STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology 277:826–832
doi: 10.1148/radiol.2015151516
Parizel PM, Makkat S, Van Miert E, Van Goethem JW, van den Hauwe L, De Schepper AM (2021) Intracranial hemorrhage: principles of CT and MRI interpretation. Eur Radiol 11:1770–1783
doi: 10.1007/s003300000800
US FDA. Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD): discussion paper and request for feedback. Available at: https://www.fda.gov/media/122535/download . Published April 2, 2019. Accessed 15 June 2021
Tariq A, Purkayastha S, Padmanaban GP et al (2020) Current clinical applications of artificial intelligence in radiology and their best supporting evidence. J Am Coll Radiol 17:1371–1381
doi: 10.1016/j.jacr.2020.08.018
Simpson SA, Cook TS (2020) Artificial intelligence and the trainee experience in radiology. J Am Coll Radiol 17:1388–1393
doi: 10.1016/j.jacr.2020.09.028