Accuracy of ChatGPT responses on tracheotomy for patient education.

Accuracy Artificial intelligence ChatGPT Head neck Information Intensive care LLM Otolaryngology Surgery Tracheotomy

Journal

European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery
ISSN: 1434-4726
Titre abrégé: Eur Arch Otorhinolaryngol
Pays: Germany
ID NLM: 9002937

Informations de publication

Date de publication:
02 Oct 2024
Historique:
received: 17 07 2024
accepted: 18 07 2024
medline: 2 10 2024
pubmed: 2 10 2024
entrez: 2 10 2024
Statut: aheadofprint

Résumé

To investigate the accuracy of information provided by ChatGPT-4o to patients about tracheotomy. Twenty common questions of patients about tracheotomy were presented to ChatGPT-4o twice (7-day intervals). The accuracy, clarity, relevance, completeness, referencing, and usefulness of responses were assessed by a board-certified otolaryngologist and a board-certified intensive care unit practitioner with the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool. The interrater reliability and the stability of the ChatGPT-4o responses were evaluated with intraclass correlation coefficient (ICC) and Pearson correlation analysis. The total scores of QAMAI were 22.85 ± 4.75 for the intensive care practitioner and 21.45 ± 3.95 for the otolaryngologist, which consists of moderate-to-high accuracy. The otolaryngologist and the ICU practitioner reported high ICC (0.807; 95%CI: 0.655-0.911). The highest QAMAI scores have been found for clarity and completeness of explanations. The QAMAI scores for the accuracy of the information and the referencing were the lowest. The information related to the post-laryngectomy tracheostomy remains incomplete or erroneous. ChatGPT-4o did not provide references for their responses. The stability analysis reported high stability in regenerated questions. The accuracy of ChatGPT-4o is moderate-to-high in providing information related to the tracheotomy. However, patients using ChatGPT-4o need to be cautious about the information related to tracheotomy care, steps, and the differences between temporary and permanent tracheotomies.

Identifiants

DOI: 10.1007/s00405-024-08859-8 PMID: 39356355
pubmed: 39356355
doi: 10.1007/s00405-024-08859-8
pii: 10.1007/s00405-024-08859-8
doi:

Types de publication

Journal Article

Langues

eng

Sous-ensembles de citation

IM

Informations de copyright

© 2024. The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.

Références

Lechien JR, Rameau A (2024) Applications of ChatGPT in Otolaryngology-Head Neck surgery: a state of the Art Review. Otolaryngol Head Neck Surg. https://doi.org/10.1002/ohn.807
doi: 10.1002/ohn.807 pubmed: 39350512
Lechien JR (2024) Generative AI and Otolaryngology-Head & Neck Surgery. Otolaryngol Clin North Am. : S0030-6665(24)00064 – 1. https://doi.org/10.1016/j.otc.2024.04.006
Abou-Abdallah M, Dar T, Mahmudzade Y, Michaels J, Talwar R, Tornari C (2024) The quality and readability of patient information provided by ChatGPT: can AI reliably explain common ENT operations? Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-024-08598-w
doi: 10.1007/s00405-024-08598-w pubmed: 38530460
Tessler I, Wolfovitz A, Alon EE, Gecel NA, Livneh N, Zimlichman E, Klang E (2024) ChatGPT’s adherence to otolaryngology clinical practice guidelines. Eur Arch Otorhinolaryngol 281(7):3829–3834. https://doi.org/10.1007/s00405-024-08634-9
doi: 10.1007/s00405-024-08634-9 pubmed: 38647684
Maniaci A, Lazzeroni M, Cozzi A, Fraccaroli F, Gaffuri M, Chiesa-Estomba C, Capaccio P (2024) Can chatbots enhance the management of pediatric sialadenitis in clinical practice? Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-024-08798-4
doi: 10.1007/s00405-024-08798-4 pubmed: 39284943
Bellinger JR, De La Chapa JS, Kwak MW, Ramos GA, Morrison D, Kesser BW (2023) BPPV Information on Google Versus AI (ChatGPT). Otolaryngol Head Neck Surg. https://doi.org/10.1002/ohn.506
doi: 10.1002/ohn.506 pubmed: 37622581
Riestra-Ayora J, Vaduva C, Esteban-Sánchez J, Garrote-Garrote M, Fernández-Navarro C, Sánchez-Rodríguez C, Martin-Sanz E (2024) ChatGPT as an information tool in rhinology. Can we trust each other today? Eur Arch Otorhinolaryngol 281(6):3253–3259. https://doi.org/10.1007/s00405-024-08581-5
doi: 10.1007/s00405-024-08581-5 pubmed: 38436756
Campbell DJ, Estephan LE, Sina E, Mastrolonardo EV, Alapati R, Amin DR, Cottrill E (2023) Evaluating ChatGPT responses on thyroid nodules for patient education. Thyroid. https://doi.org/10.1089/thy.2023.0491
doi: 10.1089/thy.2023.0491 pubmed: 38010917
Mira FA, Favier V, Dos Santos Sobreira Nunes H, de Castro JV, Carsuzaa F, Meccariello G, Vicini C, De Vito A, Lechien JR, Estomba CC, Maniaci A, Iannella G, Rojas EP, Cornejo JB, Cammaroto G (2023) Chat GPT for the management of obstructive sleep apnea: do we have a polar star? Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-023-08270-9
doi: 10.1007/s00405-023-08270-9 pubmed: 37980605
Cheong RCT, Unadkat S, Mcneillis V, Williamson A, Joseph J, Randhawa P, Andrews P, Paleri V (2024) Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard. Eur Arch Otorhinolaryngol 281(2):985–993. https://doi.org/10.1007/s00405-023-08319-9
doi: 10.1007/s00405-023-08319-9 pubmed: 37917165
Lechien JR, Carroll TL, Huston MN, Naunheim MR (2024) ChatGPT-4 accuracy for patient education in laryngopharyngeal reflux. Eur Arch Otorhinolaryngol 281(5):2547–2552. https://doi.org/10.1007/s00405-024-08560-w
doi: 10.1007/s00405-024-08560-w pubmed: 38492008
Ostrowska M, Kacała P, Onolememen D, Vaughan-Lane K, Sisily Joseph A, Ostrowski A, Pietruszewska W, Banaszewski J, Wróbel MJ (2024) To trust or not to trust: evaluating the reliability and safety of AI responses to laryngeal cancer queries. Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-024-08643-8
doi: 10.1007/s00405-024-08643-8 pubmed: 38652298
13.Vaira LA, Lechien JR, Abbate V, Allevi F, Audino G, Beltramini GA, Bergonzani M, Boscolo-Rizzo P, Califano G, Cammaroto G, Chiesa-Estomba CM, Committeri U, Crimi S, Curran NR, di Bello F, di Stadio A, Frosolini A, Gabriele G, Gengler IM, Lonardi F, Maglitto F, Mayo-Yáñez M, Petrocelli M, Pucci R, Saibene AM, Saponaro G, Tel A, Trabalzini F, Trecca EMC, Vellone V, Salzano G, De Riu G (2024) Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms. Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-024-08710-0
doi: 10.1007/s00405-024-08710-0 pubmed: 38795148 pmcid: 11392976
Charnock D, Shepperd S, Needham G et al (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 53:105–111
doi: 10.1136/jech.53.2.105 pubmed: 10396471 pmcid: 1756830
Brenner MJ, Pandian V, Milliren CE, Graham DA, Zaga C, Morris LL, Bedwell JR, Das P, Zhu H, Lee Y, Allen J, Peltz A, Chin K, Schiff BA, Randall DM, Swords C, French D, Ward E, Sweeney JM, Warrillow SJ, Arora A, Narula A, McGrath BA, Cameron TS, Roberson DW, Lee YJ, Grimm D, Divi V (2020) Global Tracheostomy Collaborative: data-driven improvements in patient safety through multidisciplinary teamwork, standardisation, education, and patient partnership. Br J Anaesth. ; 125(1):e104-e118. https://doi.org/10.1016/j.bja.2020.04.054 .
Nielsen JPS, von Buchwald C, Grønhøj C. Validity of the large language model ChatGPT (GPT4) as a patient information source in otolaryngology by a variety of doctors in a tertiary otorhinolaryngology department. Acta Otolaryngol. 2023; 143(9):779–782. https://doi.org/10.1080/00016489.2023.2254809 .
Ayoub NF, Lee YJ, Grimm D, Divi V Head-to-Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition. Otolaryngol Head Neck Surg. 2023. https://doi.org/10.1002/ohn.465 .
Shen SA, Perez-Heydrich CA, Xie DX, Nellis JC (2024) ChatGPT vs. web search for patient questions: what does ChatGPT do better? Eur Arch Otorhinolaryngol 281(6):3219–3225. https://doi.org/10.1007/s00405-024-08524-0
doi: 10.1007/s00405-024-08524-0 pubmed: 38416195 pmcid: 11410109
Langlie J, Kamrava B, Pasick LJ, Mei C, Hoffer ME (2024) Artificial intelligence and ChatGPT: an otolaryngology patient’s ally or foe? Am J Otolaryngol 45(3):104220. https://doi.org/10.1016/j.amjoto.2024.104220
doi: 10.1016/j.amjoto.2024.104220 pubmed: 38219629
Vaira LA, Lechien JR, Abbate V, Allevi F, Audino G, Beltramini GA, Bergonzani M, Bolzoni A, Committeri U, Crimi S, Gabriele G, Lonardi F, Maglitto F, Petrocelli M, Pucci R, Saponaro G, Tel A, Vellone V, Chiesa-Estomba CM, Boscolo-Rizzo P, Salzano G, De Riu G (2024) Accuracy of ChatGPT-Generated information on Head and Neck and Oromaxillofacial surgery: a Multicenter Collaborative Analysis. Otolaryngol Head Neck Surg 170(6):1492–1503. https://doi.org/10.1002/ohn.489
doi: 10.1002/ohn.489 pubmed: 37595113
Kuşcu O, Pamuk AE, Sütay Süslü N, Hosal S (2023) Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer? Front Oncol 13:1256459. https://doi.org/10.3389/fonc.2023.1256459
doi: 10.3389/fonc.2023.1256459 pubmed: 38107064 pmcid: 10722294

Auteurs

Amina Khaldi (A)

Intensive Care Unit, EpiCURA Hospital, Hornu, Belgium.

Shahram Machayekhi (S)

Intensive Care Unit, EpiCURA Hospital, Hornu, Belgium.

Michele Salvagno (M)

Intensive Care Unit, Erasme Hospital, Brussels, Belgium.

Antonino Maniaci (A)

Faculty of Medicine and Surgery, University of Enna Kore, Enna, 94100, Italy.

Luigi A Vaira (LA)

Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Viale San Pietro 43/B, Sassari, 07100, Italy.

Luigi La Via (L)

Department of Anesthesia and Intensive Care, University Hospital Policlinico "G.Rodolico-San Marco", Catania, Italy.

Fabio S Taccone (FS)

Intensive Care Unit, Erasme Hospital, Brussels, Belgium.

Jerome R Lechien (JR)

Department of Surgery, UMONS Research Institute for Health Sciences and Technology, University of Mons (UMons), Mons, Belgium. Jerome.Lechien@umons.ac.be.
Department of Otolaryngology, Elsan Polyclinic of Poitiers, Poitiers, France. Jerome.Lechien@umons.ac.be.
Department of Otolaryngology-Head Neck Surgery, School of Medicine, UFR Simone Veil, Foch Hospital, Université Versailles Saint-Quentin-en-Yvelines (Paris Saclay University), Paris, France. Jerome.Lechien@umons.ac.be.

Classifications MeSH