A User Interface for Optimizing Radiologist Engagement in Image Data Curation for Artificial Intelligence.

Journal

Radiology. Artificial intelligence
ISSN: 2638-6100
Titre abrégé: Radiol Artif Intell
Pays: United States
ID NLM: 101746556

Informations de publication

Date de publication:
Nov 2019
Historique:
received: 28 12 2018
revised: 14 06 2019
accepted: 25 06 2019
entrez: 3 5 2021
pubmed: 27 11 2019
medline: 27 11 2019
Statut: epublish

Résumé

To delineate image data curation needs and describe a locally designed graphical user interface (GUI) to aid radiologists in image annotation for artificial intelligence (AI) applications in medical imaging. GUI components support image analysis toolboxes, picture archiving and communication system integration, third-party applications, processing of scripting languages, and integration of deep learning libraries. For clinical AI applications, GUI components included two-dimensional segmentation and classification; three-dimensional segmentation and quantification; and three-dimensional segmentation, quantification, and classification. To assess radiologist engagement and performance efficiency associated with GUI-related capabilities, image annotation rate (studies per day) and speed (minutes per case) were evaluated in two clinical scenarios of varying complexity: hip fracture detection and coronary atherosclerotic plaque demarcation and stenosis grading. For hip fracture, 1050 radiographs were annotated over 7 days (150 studies per day; median speed: 10 seconds per study [interquartile range, 3-21 seconds per study]). A total of 294 coronary CT angiographic studies with 1843 arteries and branches were annotated for atherosclerotic plaque over 23 days (15.2 studies [80.1 vessels] per day; median speed: 6.08 minutes per study [interquartile range, 2.8-10.6 minutes per study] and 73 seconds per vessel [interquartile range, 20.9-155 seconds per vessel]). GUI-component compatibility with common image analysis tools facilitates radiologist engagement in image data curation, including image annotation, supporting AI application development and evolution for medical imaging. When complemented by other GUI elements, a continuous integrated workflow supporting formation of an agile deep neural network life cycle results.Supplemental material is available for this article.© RSNA, 2019.

Identifiants

DOI: 10.1148/ryai.2019180095 PMID: 33937804 PMC: PMC8017380
pubmed: 33937804
doi: 10.1148/ryai.2019180095
pmc: PMC8017380
doi:

Types de publication

Journal Article

Langues

eng

Pagination

e180095

Informations de copyright

2019 by the Radiological Society of North America, Inc.

Déclaration de conflit d'intérêts

Disclosures of Conflicts of Interest: M.D. Activities related to the present article: institution receives time-limited loaner access by OSU AI Lab to NVIDIA GPUs via Master Research Agreement between Ohio State University and NVIDIA; time-limited access by OSU AI lab to WIP postprocessing software via Master Research Agreement between Ohio State University and Siemens Healthineers. No money transferred; unrestrictive support of OSH AI lab from DeBartolo Family Funds. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. S.C. disclosed no relevant relationships. M.T.B. Activities related to the present article: institution receives time-limited loaner access by OSU AI Lab to NVIDIA GPUs via Master Research Agreement between Ohio State University and NVIDIA, no money transferred; time-limited access by OSU AI lab to WIP postprocessing software via Master Research Agreement between Ohio State University and Siemens Healthineers, no money transferred; unrestrictive support of OSH AI lab from DeBartolo Family Funds. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. S.M.Y. disclosed no relevant relationships. V.G. disclosed no relevant relationships. L.M.P. Activities related to the present article: institution receives time-limited loaner access by OSU AI Lab to NVIDIA GPUs via Master Research Agreement between Ohio State University and NVIDIA, no money transferred; time-limited access by OSU AI lab to WIP postprocessing software via Master Research Agreement between Ohio State University and Siemens Healthineers, no money transferred; unrestrictive support of OSH AI lab from DeBartolo Family Funds. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. R.D.W. Activities related to the present article: institution receives time-limited loaner access by OSU AI Lab to NVIDIA GPUs via Master Research Agreement between Ohio State University and NVIDIA, no money transferred; time-limited access by OSU AI lab to WIP postprocessing software via Master Research Agreement between Ohio State University and Siemens Healthineers, no money transferred; unrestrictive support of OSH AI lab from DeBartolo Family Funds. Activities not related to the present article: institution receives grant from NIH, Co-I on several unrelated R01 active or proposed grants. Other relationships: disclosed no relevant relationships. J.S.Y. disclosed no relevant relationships. R.G. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: employed by Siemens Healthcare. Other relationships: disclosed no relevant relationships. M.W. disclosed no relevant relationships. A.W. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: disclosed no relevant relationships. Other relationships: employed by Siemens Healthineers. A.H.H. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: employee of NVIDIA. Other relationships: disclosed no relevant relationships. A.I. disclosed no relevant relationships. T.P.O. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: employee of Siemens Healthineers; has stock in Siemens Healthineers. Other relationships: disclosed no relevant relationships. B.S.E. Activities related to the present article: institution receives time-limited loaner access by OSU AI Lab to NVIDIA GPUs via Master Research Agreement between Ohio State University and NVIDIA, no money transferred; time-limited access by OSU AI lab to WIP postprocessing software via Master Research Agreement between Ohio State University and Siemens Healthineers, no money transferred; unrestrictive support of OSH AI lab from DeBartolo Family Funds. Activities not related to the present article: institution receives NIH grants; author is co-I on several unrelated NIH grants. Other relationships: disclosed no relevant relationships.

Références

Heart. 2002 Jul;88(1):91-6
pubmed: 12067962
Stud Health Technol Inform. 2012;180:574-8
pubmed: 22874256
Am J Public Health. 2017 Aug;107(8):1283-1289
pubmed: 28640681
Can Assoc Radiol J. 2018 May;69(2):120-135
pubmed: 29655580
Int J Comput Assist Radiol Surg. 2016 Jan;11(1):99-106
pubmed: 26092662
Radiology. 2017 Dec;285(3):923-931
pubmed: 28678669
Med Image Comput Comput Assist Interv. 2013;16(Pt 3):74-81
pubmed: 24505746
Med Image Anal. 2017 Dec;42:60-88
pubmed: 28778026
IEEE J Biomed Health Inform. 2017 Jan;21(1):4-21
pubmed: 28055930
J Am Coll Radiol. 2018 Mar;15(3 Pt B):521-526
pubmed: 29396120
IEEE Pulse. 2011 Nov;2(6):60-70
pubmed: 22147070
Comput Med Imaging Graph. 2006 Mar;30(2):75-87
pubmed: 16584976
Radiology. 2016 May;279(2):329-43
pubmed: 27089187
J Digit Imaging. 2018 Jun;31(3):341-352
pubmed: 29725964

Auteurs

Mutlu Demirer (M)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0002-6842-282X

Sema Candemir (S)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).

Matthew T Bigelow (MT)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0002-4733-5911

Sarah M Yu (SM)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).

Vikash Gupta (V)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0002-6786-9865

Luciano M Prevedello (LM)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0002-6768-6452

Richard D White (RD)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).

Joseph S Yu (JS)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0001-6832-7243

Rainer Grimmer (R)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).

Michael Wels (M)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).

Andreas Wimmer (A)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0003-1009-4306

Abdul H Halabi (AH)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).

Alvin Ihsani (A)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).

Thomas P O'Donnell (TP)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0003-3553-1306

Barbaros S Erdal (BS)

Department of Radiology, Laboratory for Augmented Intelligence in Imaging-Division of Medical Imaging Informatics, Ohio State University College of Medicine, OSU Wexner Medical Center, 395 W 12th Ave, Suite 452, Columbus, OH 43210 (M.D., S.C., M.T.B., S.M.Y., V.G., L.M.P., R.D.W., J.S.Y., B.S.E.); Siemens Healthineers, Erlangen, Germany (R.G., M.W., A.W.); NVIDIA, Santa Clara, Calif (A.H.H., A.I.); and Siemens Healthineers, Malvern, Pa (T.P.O.).
ORCID: 0000-0003-3637-0102

Classifications MeSH