Accuracy of ChatGPT-Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative Analysis.
ChatGPT
artificial intelligence
maxillofacial surgery
otorhinolaryngology
Journal
Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery
ISSN: 1097-6817
Titre abrégé: Otolaryngol Head Neck Surg
Pays: England
ID NLM: 8508176
Informations de publication
Date de publication:
18 Aug 2023
18 Aug 2023
Historique:
revised:
16
06
2023
received:
27
04
2023
accepted:
14
07
2023
medline:
18
8
2023
pubmed:
18
8
2023
entrez:
18
8
2023
Statut:
aheadofprint
Résumé
To investigate the accuracy of Chat-Based Generative Pre-trained Transformer (ChatGPT) in answering questions and solving clinical scenarios of head and neck surgery. Observational and valuative study. Eighteen surgeons from 14 Italian head and neck surgery units. A total of 144 clinical questions encompassing different subspecialities of head and neck surgery and 15 comprehensive clinical scenarios were developed. Questions and scenarios were inputted into ChatGPT4, and the resulting answers were evaluated by the researchers using accuracy (range 1-6), completeness (range 1-3), and references' quality Likert scales. The overall median score of open-ended questions was 6 (interquartile range[IQR]: 5-6) for accuracy and 3 (IQR: 2-3) for completeness. Overall, the reviewers rated the answer as entirely or nearly entirely correct in 87.2% of cases and as comprehensive and covering all aspects of the question in 73% of cases. The artificial intelligence (AI) model achieved a correct response in 84.7% of the closed-ended questions (11 wrong answers). As for the clinical scenarios, ChatGPT provided a fully or nearly fully correct diagnosis in 81.7% of cases. The proposed diagnostic or therapeutic procedure was judged to be complete in 56.7% of cases. The overall quality of the bibliographic references was poor, and sources were nonexistent in 46.4% of the cases. The results generally demonstrate a good level of accuracy in the AI's answers. The AI's ability to resolve complex clinical scenarios is promising, but it still falls short of being considered a reliable support for the decision-making process of specialists in head-neck surgery.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2023 The Authors. Otolaryngology-Head and Neck Surgery published by Wiley Periodicals LLC on behalf of American Academy of Otolaryngology-Head and Neck Surgery Foundation.
Références
OpenAI. ChatGPT. 2023. Accessed March 28, 2023. https://openai.com/blog/chatgpt
Exploding Topics. Number of ChatGPT users 2023. 2023. Accessed March 30, 2023. https://explodingtopics.com/blog/chatgpt-users
Barat M, Soyer P, Dohan A. Appropriateness of recommendations provided by ChatGPT to interventional radiologists. Can Assoc Radiol J. Published online April 13, 2023. doi:10.1177/08465371231170133
Cheng K, Sun Z, He Y, Gu S, Wu H. The potential impact of ChatGPT/GPT-4 on surgery: will it topple the profession of surgeons? Int J Surg. 2023;109:1545-1547. doi:10.1097/JS9.0000000000000388
Strong E, DiGiammarino A, Weng Y, et al. Performance of ChatGPT on free-response, clinical reasoning exams. medRxiv. Published online March 29, 2023. doi:10.1101/2023.03.24.23287731
Zimmerman A. A Ghostwriter for the masses: ChatGPT and the future of writing. Ann Surg Oncol. 2023;30:3170-3173. doi:10.1245/s10434-023-13436-0
Gupta R, Herzog I, Weisberger J, Chao J, Chaiyasate K, Lee Es. Utilization of ChatGPT for plastic surgery research: friend or foe? J Plast Reconstr Aesthet Surg. 2023;80:145-147.
Ariyaratne S, Iyengar KP, Nischal N, Chitti Babu N, Botchu R. A comparison of ChatGPT-generated articles with human-written articles. Skeletal Radiol. 2023;52:1755-1758. doi:10.1007/s00256-023-04340-5
Lee H. The rise of ChatGPT: exploring its potential in medical education. Anat Sci Educ. Published online March 14, 2023. doi:10.1002/ase.2270
Eysenbach G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers. JMIR Med Educ. 2023;9:e46885.
Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT-reshaping medical education and clinical management. Pak J Med Sci. 2023;39:605-607.
Kahambing JG. ChatGPT, public health communication and ‘intelligent patient companionship’. J Public Health. Published online April 8, 2023. doi:10.1093/pubmed/fdad028
Cox A, Seth I, Xie Y, Hunter-Smith DJ, Rozen WM. Utilizing ChatGPT-4 for providing medical information on blepharoplasties to patients. Aesthet Surg J. 2023;43:NP658-NP662. doi:10.1093/asj/sjad096
Potapenko I, Boberg-Ans LC, Stormly Hansen M, Klefter ON, van Dijk EHC, Subhi Y. Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT. Acta Ophthalmol. Published online March 13, 2023. doi:10.1111/aos.15661
Al Ghamdi KM, Moussa NA. Internet use by the public to search for health-related information. Int J Med Inform. 2012;81:363-373.
Tonsaker T, Bartlett G, Trpkov C. Health information on the internet: gold mine or minefield? Can Fam Physician. 2014;60:407-408.
Van Dis EAM, Bollen J, Zuidema W, van Rooij R, Bockting CL. ChatGPT: five priorities for research. Nature. 2023;614:224-226.
Thorp HH. ChatGPT is fun, but not an author. Science. 2023;379:313.
Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD. Evaluating ChatGPT as an adjunct for radiologic decision-making. medRxiv. Published online February 7, 2023. doi:10.1101/2023.02.02.23285399
Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47:33.
Grünebaum A, Chervenak J, Pollet SL, Katz A, Chervenak FA. The exciting potential for ChatGPT in obstetrics and gynecology. Am J Obstet Gynecol. 2023;228:696-705. doi:10.1016/j.ajog.2023.03.009
Rao A, Pang M, Kim J, et al. Assessing the utility of ChatGPT throughout the entire clinical workflow. medRxiv. Published online February 26, 2023. doi:10.1101/2023.02.21.23285886
Johnson D, Goodman R, Patrinely J, et al. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Res Sq. Published online February 28, 2023. doi:10.21203/rs.3.rs-2566942/v1
Hooshafza S, Mc Quaid L, Stephens G, Flynn R, OConnor L. Development of a framework to assess the quality of data sources in healthcare settings. J Am Med Inform Assoc. 2022;29:944-952.
The jamovi project. Jamovi. (version 2.3) [Computer Software]. 2022. Accessed january 24, 2023. https://www.jamovi.org
Blease C, Kaptchuk TJ, Bernstein MH, Mandl KD, Halamka JD, DesRoches CM. Artificial intelligence and the future of primary care: exploratory qualitative study of UK general practitioners' view. J Med Internet Res. 2020;22:e16775.
Laranjo L, Dunn AG, Tong HL, et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc. 2018;25:1248-1258.
Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare. 2023;11:887.
Hopkins AM, Logan JM, Kichenadasse G, Sorich MJ. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectrum. 2023;7:pkad010.
Goodman RS, Patrinely, Jr. JR, Osterman T, Wheless L, Johnson DB. On the cusp: considering the impact of artificial intelligence language models in healthcare. Med. 2023;4:139-140.
Zhang J, Zhang Z. Ethics and governance of trustworthy medical artificial intelligence. BMC Med Inform Decis Mak. 2023;23:7.
Masters K. Ethical use of artificial intelligence in health professions education: AMEE Guide No. 158. Med Teach. 2023;45:574-584. doi:10.1080/0142159X.2023.2186203
Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595. doi:10.3389/frai.2023.1169595
Cheng K, Li Z, He Y, et al. Potential use of artificial intelligence in infectious disease: take ChatGPT as an example. Ann Biomed Eng. 2023;51:1130-1135. doi:10.1007/s10439-023-03203-3
Cheng K, Li Z, Guo Q, Sun Z, Wu H, Li C. Emergency surgery in the era of artificial intelligence: ChatGPT could be the doctor's right-hand man. Int J Surg. 2023;109:1816-1818. doi:10.1097/JS9.0000000000000410
Qureshi R, Shaughnessy D, Gill KAR, Robinson KA, Li T, Agai E. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? Syst Rev. 2023;12:72.
Frosolini A, Gennaro P, Cascino F, Gabriele G. In reference to “role of Chat GPT in public health”, to highlight the AI's incorrect reference generation. Ann Biomed Eng. Published online 2023 May 22, 2023. doi:10.1007/s10439-023-03248-4
Wagner MW, Ertl-Wagner BB. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information. Can Assoc Radiol J. 2023. doi:10.1177/08465371231171125