A quality assessment tool for focused abdominal sonography for trauma examinations using artificial intelligence.
Journal
The journal of trauma and acute care surgery
ISSN: 2163-0763
Titre abrégé: J Trauma Acute Care Surg
Pays: United States
ID NLM: 101570622
Informations de publication
Date de publication:
27 Sep 2024
27 Sep 2024
Historique:
medline:
27
9
2024
pubmed:
27
9
2024
entrez:
27
9
2024
Statut:
aheadofprint
Résumé
Current tools to review focused abdominal sonography for trauma (FAST) images for quality have poorly defined grading criteria or are developed to grade the skills of the sonographer and not the examination. The purpose of this study is to establish a grading system with substantial agreement among coders, thereby enabling the development of an automated assessment tool for FAST examinations using artificial intelligence (AI). Five coders labeled a set of FAST clips. Each coder was responsible for a different subset of clips (10% of the clips were labeled in triplicate to evaluate intercoder reliability). The clips were labeled with a quality score from 1 (lowest quality) to 5 (highest quality). Clips of 3 or greater were considered passing. An AI training model was developed to score the quality of the FAST examination. The clips were split into a training set, a validation set, and a test set. The predicted scores were rounded to the nearest quality level to distinguish passing from failing clips. A total of 1,514 qualified clips (1,399 passing and 115 failing clips) were evaluated in the final data set. This final data set had a 94% agreement between pairs of coders on the pass/fail prediction, and the set had a Krippendorff α of 66%. The decision threshold can be tuned to achieve the desired tradeoff between precision and sensitivity. Without using the AI model, a reviewer would, on average, examine roughly 25 clips for every 1 failing clip identified. In contrast, using our model with a decision threshold of 0.015, a reviewer would examine roughly five clips for every one failing clip - a fivefold reduction in clips reviewed while still correctly identifying 85% of passing clips. Integration of AI holds significant promise in improving the accurate evaluation of FAST images while simultaneously alleviating the workload burden on expert physicians. Diagnostic Test/Criteria; Level II.
Sections du résumé
BACKGROUND
BACKGROUND
Current tools to review focused abdominal sonography for trauma (FAST) images for quality have poorly defined grading criteria or are developed to grade the skills of the sonographer and not the examination. The purpose of this study is to establish a grading system with substantial agreement among coders, thereby enabling the development of an automated assessment tool for FAST examinations using artificial intelligence (AI).
METHODS
METHODS
Five coders labeled a set of FAST clips. Each coder was responsible for a different subset of clips (10% of the clips were labeled in triplicate to evaluate intercoder reliability). The clips were labeled with a quality score from 1 (lowest quality) to 5 (highest quality). Clips of 3 or greater were considered passing. An AI training model was developed to score the quality of the FAST examination. The clips were split into a training set, a validation set, and a test set. The predicted scores were rounded to the nearest quality level to distinguish passing from failing clips.
RESULTS
RESULTS
A total of 1,514 qualified clips (1,399 passing and 115 failing clips) were evaluated in the final data set. This final data set had a 94% agreement between pairs of coders on the pass/fail prediction, and the set had a Krippendorff α of 66%. The decision threshold can be tuned to achieve the desired tradeoff between precision and sensitivity. Without using the AI model, a reviewer would, on average, examine roughly 25 clips for every 1 failing clip identified. In contrast, using our model with a decision threshold of 0.015, a reviewer would examine roughly five clips for every one failing clip - a fivefold reduction in clips reviewed while still correctly identifying 85% of passing clips.
CONCLUSION
CONCLUSIONS
Integration of AI holds significant promise in improving the accurate evaluation of FAST images while simultaneously alleviating the workload burden on expert physicians.
LEVEL OF EVIDENCE
METHODS
Diagnostic Test/Criteria; Level II.
Identifiants
pubmed: 39327643
doi: 10.1097/TA.0000000000004425
pii: 01586154-990000000-00814
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
Copyright © 2024 Wolters Kluwer Health, Inc. All rights reserved.
Références
Ollerton JE, Sugrue M, Balogh Z, D'Amours SK, Giles A, Wyllie P. Prospective study to evaluate the influence of FAST on trauma patient management. J Trauma. 2006;60:785–791.
Teixeira PGR, Inaba K, Hadjizacharia P, Brown C, Salim A, Rhee P, et al. Preventable or potentially preventable mortality at a mature trauma center. J Trauma. 2007;63:1338–1337; discussion 1346-7.
Planquart F, Marcaggi E, Blondonnet R, Clovet O, Bobbia X, Boussat B, et al. Appropriateness of initial course of action in the management of blunt trauma based on a diagnostic workup including an extended ultrasonography scan. JAMA Netw Open. 2022;5:e2245432.
Melniker LA, Leibner E, McKenney MG, Lopez P, Briggs WM, Mancuso CA. Randomized controlled clinical trial of point-of-care, limited ultrasonography for trauma in the emergency department: the first sonography outcomes assessment program trial. Ann Emerg Med. 2006;48:227–235.
Liu R, Blaivas M, Moore C, Sivitz A, Flannigan M, Tirado A, et al. Emergency ultrasound standard reporting guidelines. 2018.
Jang T, Kryder G, Sineff S, Naunheim R, Aubin C, Kaji AH. The technical errors of physicians learning to perform focused assessment with sonography in trauma. Acad Emerg Med. 2012;19:98–101.
Ziesmann MT, Park J, Unger BJ, Kirkpatrick AW, Vergis A, Logsetty S, et al. Validation of the quality of ultrasound imaging and competence (QUICk) score as an objective assessment tool for the FAST examination. J Trauma Acute Care Surg. 2015;78:1008–1013.
Smith DH, Lineberger JP, Baker GH. On the relevance of temporal features for medical ultrasound video recognition. Lecture Notes in Computer Science, pages 744-753, 2023;arXiv:2310.10453
Olczak J, Pavlopoulos J, Prijs J, Ijpma FFA, Doornberg JN, Lundstrom C, Hedlund J, et al. Presenting artificial intelligence, deep learning, and machine learning studies to clinicians and healthcare stakeholders: an introductory reference with a guideline and a clinical AI research (CAIR) checklist proposal. Acta Orthop. 2021;92:513–525.
Anonymous 7th Edition of the Resources for Optimal Care of the Injured Patient (2022 Standards).
Anonymous ultrasound guidelines: emergency, point-of-care and clinical ultrasound guidelines in medicine. Ann Emerg Med. 2017;69:e27–e54.
Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273–278.
Erickson BJ, Korfiatis P, Akkus Z, Kline TL. Machine learning for medical imaging. Radiographics. 2017;37:505–515.
Ratner M. FDA backs clinician-free AI imaging diagnostic tools. Nat Biotechnol. 2018;36:673–674.
Ghorbani A, Ouyang D, Abid A, He B, Chen JH, Harrington RA, et al. Deep learning interpretation of echocardiograms. NPJ Digit Med. 2020;3:10.
Madani A, Arnaout R, Mofrad M, Arnaout R. Fast and accurate view classification of echocardiograms using deep learning. NPJ Digit Med. 2018;1:6. doi:10.1038/s41746-1 Epub 2018 Mar 21.
doi: 10.1038/s41746-1
Brattain LJ, Telfer BA, Dhyani M, Grajo JR, Samir AE. Machine learning for medical ultrasound: status, methods, and future opportunities. Abdom Radiol (NY). 2018;43:786–799.
Sjogren AR, Leo MM, Feldman J, Gwin JT. Image segmentation and machine learning for detection of abdominal free fluid in focused assessment with sonography for trauma examinations: a pilot study. J Ultrasound Med. 2016;35:2501–2509.
Kornblith AE, Addo N, Dong R, Rogers R, Grupp-Phelan J, Butte A, et al. Development and validation of a deep learning model for automated view classification of pediatric focused assessment with sonography for trauma (FAST). J Ultrasound Med. 2022;41:1915–1924.
Yu F, Sun J, Li A, Cheng J, Wan C, Liu J. Image quality classification for DR screening using deep learning. Annu Int Conf IEEE Eng Med Biol Soc. 2017;2017:664–667.
Zago GT, Andreão RV, Dorizzi B, Teatini Salles EO. Retinal image quality assessment using deep learning. Comput Biol Med. 2018;103:64–70.
Lin Z, Li S, Ni D, Liao Y, Wen H, Du J, et al. Multi-task learning for quality assessment of fetal head ultrasound images. Med Image Anal. 2019;58:101548.
Taye M, Morrow D, Cull J, Smith DH, Hagan M. Deep learning for FAST quality assessment. J Ultrasound Med. 2023;42:71–79.