Evaluating the scientific reliability of ChatGPT as a source of information on asthma.

Background

This study assessed the reliability of ChatGPT as a source of information on asthma, given the increasing use of artificial intelligence-driven models for medical information. Prior concerns about misinformation on atopic diseases in various digital platforms underline the importance of this evaluation.

Objective

We aimed to evaluate the scientific reliability of ChatGPT as a source of information on asthma.

Methods

The study involved analyzing ChatGPT's responses to 26 asthma-related questions, each followed by a follow-up question. These encompassed definition/risk factors, diagnosis, treatment, lifestyle factors, and specific clinical inquiries. Medical professionals specialized in allergic and respiratory diseases independently assessed the responses using a 1-to-5 accuracy scale.

Results

Approximately 81% of the responses scored 4 or higher, suggesting a generally high accuracy level. However, 5 responses scored >3, indicating minor potentially harmful inaccuracies. The overall median score was 4. Fleiss multirater kappa value showed moderate agreement among raters.

Conclusion

ChatGPT generally provides reliable asthma-related information, but its limitations, such as lack of depth in certain responses and inability to cite sources or update in real time, were noted. It shows promise as an educational tool, but it should not be a substitute for professional medical advice. Future studies should explore its applicability for different user demographics and compare it with newer artificial intelligence models.

Informations de copyright

© 2024 The Author(s).

Déclaration de conflit d'intérêts

Disclosure of potential conflict of interest: H. Meteran reports receipt outside the present study of honoraria for lectures or advisory board meetings from GSK, Teva, Novartis, Sanofi-Aventis, Airsonett AB, and ALK-Abelló Nordic A/S; and research grants from ALK-Abelló A/S. S. F. Thomsen reports, outside the present study, acting as speaker and/or advisor for 10.13039/100004339Sanofi, 10.13039/100006483AbbVie, 10.13039/501100023331LEO Pharma, 10.13039/100004319Pfizer, Eli Lilly, Novartis, UCB Pharma, Almirall, Union Therapeutics, and Janssen Pharmaceuticals; and receiving research support from Sanofi, AbbVie, LEO Pharma, Novartis, UCB Pharma, and 10.13039/100008897Janssen Pharmaceuticals. The rest of the authors declare that they have no relevant conflicts of interest.