Artificial intelligence versus human touch: can artificial intelligence accurately generate a literature review on laser technologies?
Artificial intelligence
Laser
Lithotripsy
Machine learning
Neural network
Urinary stones
Journal
World journal of urology
ISSN: 1433-8726
Titre abrégé: World J Urol
Pays: Germany
ID NLM: 8307716
Informations de publication
Date de publication:
28 Oct 2024
28 Oct 2024
Historique:
received:
10
06
2024
accepted:
04
10
2024
medline:
28
10
2024
pubmed:
28
10
2024
entrez:
28
10
2024
Statut:
epublish
Résumé
To compare the accuracy of open-source Artificial Intelligence (AI) Large Language Models (LLM) against human authors to generate a systematic review (SR) on the new pulsed-Thulium:YAG (p-Tm:YAG) laser. Five manuscripts were compared. The Human-SR on p-Tm:YAG (considered to be the "ground truth") was written by independent certified endourologists with expertise in lasers, accepted in a peer-review pubmed-indexed journal (but not yet available online, and therefore not accessible to the LLMs). The query to the AI LLMs was: "write a systematic review on pulsed-Thulium:YAG laser for lithotripsy" which was submitted to four LLMs (ChatGPT3.5/Vercel/Claude/Mistral-7b). The LLM-SR were uniformed and Human-SR reformatted to fit the general output appearance, to ensure blindness. Nine participants with various levels of endourological expertise (three Clinical Nurse Specialist nurses, Urology Trainees and Consultants) objectively assessed the accuracy of the five SRs using a bespoke 10 "checkpoint" proforma. A subjective assessment was recorded using a composite score including quality (0-10), clarity (0-10) and overall manuscript rank (1-5). The Human-SR was objectively and subjectively more accurate than LLM-SRs (96 ± 7% and 86.8 ± 8.2% respectively; p < 0.001). The LLM-SRs did not significantly differ but ChatGPT3.5 presented greater subjective and objective accuracy scores (62.4 ± 15% and 29 ± 28% respectively; p > 0.05). Quality and clarity assessments were significantly impacted by SR type but not the expertise level (p < 0.001 and > 0.05, respectively). LLM generated data on highly technical topics present a lower accuracy than Key Opinion Leaders. LLMs, especially ChatGPT3.5, with human supervision could improve our practice.
Identifiants
pubmed: 39466443
doi: 10.1007/s00345-024-05311-8
pii: 10.1007/s00345-024-05311-8
doi:
Types de publication
Journal Article
Comparative Study
Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
598Subventions
Organisme : Association Française d'Urologie
ID : 2023
Organisme : European Association of Urology
ID : 2023-2024
Informations de copyright
© 2024. The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.
Références
Turing AM. I.—Computing machinery and intelligence. Mind. 1950;LIX(236):433‑60.
Yu P, Xu H, Hu X, Deng C (2023) Leveraging generative AI and large language models: a comprehensive roadmap for healthcare integration. Healthc Basel Switz 11(20):2776
Stamatelou K, Goldfarb DS (2023) Epidemiology of kidney stones. Healthc Basel Switz. 11(3):424
Brikowski TH, Lotan Y, Pearle MS (2008) Climate-related increase in the prevalence of urolithiasis in the United States. Proc Natl Acad Sci 105(28):9841–9846
doi: 10.1073/pnas.0709652105
pubmed: 18626008
pmcid: 2474527
Stamatelou K, Goldfarb DS (2023) Epidemiology of Kidney Stones. Healthcare 11(3):424
doi: 10.3390/healthcare11030424
pubmed: 36766999
pmcid: 9914194
Türk C, Petřík A, Sarica K, Seitz C, Skolarikos A, Straub M et al (2016) EAU guidelines on interventional treatment for urolithiasis. Eur Urol 69(3):475–482
doi: 10.1016/j.eururo.2015.07.041
pubmed: 26344917
Assimos D, Krambeck A, Miller NL, Monga M, Murad MH, Nelson CP et al (2016) Surgical management of stones: American Urological Association/Endourological Society Guideline, part I. J Urol 196(4):1153–1160
doi: 10.1016/j.juro.2016.05.090
pubmed: 27238616
Johnson DE, Cromeens DM, Price RE (1992) Use of the holmium:YAG laser in urology. Lasers Surg Med 12(4):353–363
doi: 10.1002/lsm.1900120402
pubmed: 1386643
Keller EX, De Coninck V, Doizi S, Daudon M, Traxer O (2021) Thulium fiber laser: ready to dust all urinary stone composition types? World J Urol 39(6):1693–1698
doi: 10.1007/s00345-020-03217-9
pubmed: 32363450
Chicaud M, Corrales M, Kutchukian S, Solano C, Candela L, Doizi S et al (2023) Thulium:YAG laser: a good compromise between holmium:YAG and thulium fiber laser for endoscopic lithotripsy? A narrative review. World J Urol 41(12):3437–3447
doi: 10.1007/s00345-023-04679-3
pubmed: 37932561
Ventimiglia E, Robesti D, Bevilacqua L, Tondelli E, Oliva I, Orecchia L et al (2023) What to expect from the novel pulsed thulium:YAG laser? A systematic review of endourological applications. World J Urol 41:3301–3308
doi: 10.1007/s00345-023-04580-z
pubmed: 37682286
ChatGPT [Internet]. Disponible sur: https://chat.openai.com . Accessed 31 Oct 2023
Eppler M, Ganjavi C, Ramacciotti LS, Piazza P, Rodler S, Checcucci E et al (2024) Awareness and use of ChatGPT and large language models: a prospective cross-sectional global survey in urology. Eur Urol 85(2):146–153
doi: 10.1016/j.eururo.2023.10.014
pubmed: 37926642
Jeblick K, Schachtner B, Dexl J, Mittermeier A, Stüber AT, Topalis J et al (2023) ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur Radiol 34:2817–2825
doi: 10.1007/s00330-023-10213-1
pubmed: 37794249
pmcid: 11126432
Panthier C, Gatinel D (2023) Success of ChatGPT, an AI language model, in taking the French language version of the European Board of Ophthalmology examination: a novel approach to medical knowledge assessment. J Fr Ophtalmol 46(7):706–711
doi: 10.1016/j.jfo.2023.05.006
pubmed: 37537126
AI M. Mistral 7B [Internet]. 2023. Disponible sur: https://mistral.ai/fr/news/announcing-mistral-7b/ . Accessed 25 Mar 2024
Vercel [Internet]. Vercel: Build and deploy the best Web experiences with The Frontend Cloud. Disponible sur: https://vercel.com/home . Accessed 25 Mar 2024
Claude [Internet]. Disponible sur: https://claude.ai/login . Accessed 25 Mar 2024
Mesnard B, Schirmann A, Branchereau J, Perrot O, Bogaert G, Neuzillet Y et al (2024) Artificial intelligence: ready to pass the European board examinations in urology? Eur Urol Open Sci 60:44–46
doi: 10.1016/j.euros.2024.01.002
pubmed: 38321995
pmcid: 10845241
Kollitsch L, Eredics K, Marszalek M, Rauchenwald M, Brookman-May SD, Burger M et al (2024) How does artificial intelligence master urological board examinations? A comparative analysis of different Large Language Models’ accuracy and reliability in the 2022 In-Service Assessment of the European Board of Urology. World J Urol 42(1):20
doi: 10.1007/s00345-023-04749-6
pubmed: 38197996
GitHub: Let’s build from here · GitHub [Internet]. Disponible sur: https://github.com/ . Accessed 27 Mar 2024
Falahkheirkhah K, Mukherjee SS, Gupta S, Herrera-Hernandez L, McCarthy MR, Jimenez RE et al (2023) Accelerating cancer histopathology workflows with chemical imaging and machine learning. Cancer Res Commun. 3(9):1875–1887
doi: 10.1158/2767-9764.CRC-23-0226
pubmed: 37772992
pmcid: 10506535
Cacciamani GE, Sanford DI, Chu TN, Kaneko M, De Castro Abreu AL, Duddalwar V et al (2023) Is artificial intelligence replacing our radiology stars? Not Yet! Eur Urol Open Sci 48:14–16
doi: 10.1016/j.euros.2022.09.024
pubmed: 36588775
Rodler S, Kopliku R, Ulrich D, Kaltenhauser A, Casuscelli J, Eismann L et al (2023) Patients’ trust in artificial intelligence-based decision-making for localized prostate cancer: results from a prospective trial. Eur Urol Focus. https://doi.org/10.1016/j.euf.2023.10.020
doi: 10.1016/j.euf.2023.10.020
pubmed: 38036339
Cacciamani GE, Chen A, Gill IS, Hung AJ (2023) Artificial intelligence and urology: ethical considerations for urologists and patients. Nat Rev Urol 21:50–59
doi: 10.1038/s41585-023-00796-1
pubmed: 37524914