Reliability of artificial intelligence chatbot responses to frequently asked questions in breast surgical oncology.
ChatGPT
artificial intelligence
breast cancer
education
surgery
Journal
Journal of surgical oncology
ISSN: 1096-9098
Titre abrégé: J Surg Oncol
Pays: United States
ID NLM: 0222643
Informations de publication
Date de publication:
04 Jun 2024
04 Jun 2024
Historique:
received:
08
04
2024
accepted:
21
05
2024
medline:
5
6
2024
pubmed:
5
6
2024
entrez:
5
6
2024
Statut:
aheadofprint
Résumé
Artificial intelligence (AI)-driven chatbots, capable of simulating human-like conversations, are becoming more prevalent in healthcare. While this technology offers potential benefits in patient engagement and information accessibility, it raises concerns about potential misuse, misinformation, inaccuracies, and ethical challenges. This study evaluated a publicly available AI chatbot, ChatGPT, in its responses to nine questions related to breast cancer surgery selected from the American Society of Breast Surgeons' frequently asked questions (FAQ) patient education website. Four breast surgical oncologists assessed the responses for accuracy and reliability using a five-point Likert scale and the Patient Education Materials Assessment (PEMAT) Tool. The average reliability score for ChatGPT in answering breast cancer surgery questions was 3.98 out of 5.00. Surgeons unanimously found the responses understandable and actionable per the PEMAT criteria. The consensus found ChatGPT's overall performance was appropriate, with minor or no inaccuracies. ChatGPT demonstrates good reliability in responding to breast cancer surgery queries, with minor, nonharmful inaccuracies. Its answers are accurate, clear, and easy to comprehend. Notably, ChatGPT acknowledged its informational role and did not attempt to replace medical advice or discourage users from seeking input from a healthcare professional.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2024 Wiley Periodicals LLC.
Références
Computer and Internet use. American Community Survey, U.S. Census Bureau. Available August 21, 2023. https://www.census.gov/acs/www/about/why-we-ask-each-question/computer/
Caiata‐Zufferey M, Abraham A, Sommerhalder K, Schulz PJ. Online health information seeking in the context of the medical consultation in Switzerland. Qual Health Res. 2010;20(8):1050‐1061.
Langford AT, Roberts T, Gupta J, Orellana KT, Loeb S. Impact of the Internet on patient‐physician communication. Eur Urol Focus. 2020;6(3):440‐444.
Potapenko I, Boberg‐Ans LC, Stormly Hansen M, Klefter ON, van Dijk EHC, Subhi Y. Artificial intelligence‐based chatbot patient information on common retinal diseases using ChatGPT. Acta Ophthalmol. 2023;101(7):829‐831.
Introducing ChatGPT. OpenAI. Available August 21, 2023. https://openai.com/blog/chatgpt
Uprety D, Zhu D, West H. ChatGPT‐A promising generative AI tool and its implications for cancer care. Cancer. 2023;129(15):2284‐2289.
Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber‐Physical Systems. 2023;3:121‐154.
Johnson D, Goodman R, Patrinely J, et al. Assessing the accuracy and reliability of AI‐generated medical responses: an evaluation of the Chat‐GPT model. Res Sq. 2023;rs.3.rs-2566942. doi:10.21203/rs.3.rs-2566942/v1
Liu J, Wang C, Liu S. Utility of ChatGPT in clinical practice. J Med Internet Res. 2023;25:e48568.
Sallam M. Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023;11(6):887.
Lukac S, Dayan D, Fink V, et al. Evaluating ChatGPT as an adjunct for the multidisciplinary tumor board decision‐making in primary breast cancer cases. Arch Gynecol Obstet. 2023;308:1831‐1844.
Yeo YH, Samaan JS, Ng WH, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023;29(3):721‐732.
The Patient Education Materials Assessment Tool (PEMAT) and user's guide. Agency for Healthcare Research and Quality. Available August 22, 2023. https://www.ahrq.gov/health-literacy/patient-education/pemat.html
Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47(1):33.
Vaira LA, Lechien JR, Abbate V, et al. Accuracy of ChatGPT‐generated information on head and neck and oromaxillofacial surgery: a multicenter collaborative analysis. Otolaryngol Head Neck Surg. 2024;170(6):1492‐1503. doi:10.1002/ohn.489
Frosolini A, Gennaro P, Cascino F, Gabriele G. In reference to “role of chat GPT in public health”, to highlight the AI's incorrect reference generation. Ann Biomed Eng. 2023;51:2120‐2122.
Wagner MW, Ertl‐Wagner BB. Accuracy of information and references using ChatGPT‐3 for retrieval of clinical radiological information. Can Assoc Radiol J. 2024;75(1):69‐73.
Milligan F. Suicide and women living with and beyond a breast cancer diagnosis. Br J Nurs. 2022;31(18):954‐960.