Algorithmic Identification of Treatment-Emergent Adverse Events From Clinical Notes Using Large Language Models: A Pilot Study in Inflammatory Bowel Disease.
Journal
Clinical pharmacology and therapeutics
ISSN: 1532-6535
Titre abrégé: Clin Pharmacol Ther
Pays: United States
ID NLM: 0372741
Informations de publication
Date de publication:
08 Mar 2024
08 Mar 2024
Historique:
received:
12
09
2023
accepted:
13
02
2024
medline:
9
3
2024
pubmed:
9
3
2024
entrez:
9
3
2024
Statut:
aheadofprint
Résumé
Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California - San Francisco (UCSF)-BERT, to identify serious AEs (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF-BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF-BERT was significantly superior at identifying hospitalization events emergent to medication use (P < 0.01). LLMs like UCSF-BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre-trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : NIH HHS
ID : K99LM014099
Pays : United States
Commentaires et corrections
Type : UpdateOf
Informations de copyright
© 2024 The Authors. Clinical Pharmacology & Therapeutics © 2024 American Society for Clinical Pharmacology and Therapeutics.
Références
Questions and Answers on FDA's Adverse Event Reporting System (FAERS) <https://www.fda.gov/drugs/surveillance/questions-and-answers-fdas-adverse-event-reporting-system-faers#:~:text=The%20FDA%20Adverse%20Event%20Reporting,that%20were%20submitted%20to%20FDA> Accessed May 19, 2023.
Thein, D., Egeberg, A., Skov, L. & Loft, N. Absolute and relative risk of new-onset psoriasis associated with tumor necrosis factor-α inhibitor treatment in patients with immune-mediated inflammatory diseases: a Danish Nationwide cohort study. JAMA Dermatol. 158, 997-1004 (2022).
Chaparro, M. et al. Short and long-term effectiveness and safety of vedolizumab in inflammatory bowel disease: results from the ENEIDA registry. Aliment. Pharmacol. Ther. 48, 839-851 (2018).
Hazell, L. & Shakir, S.A. Under-reporting of adverse drug reactions. Drug Saf. 29, 385-396 (2006).
Varallo, F.R., Guimarães, S.O.P., Abjaude, S.A.R. & Mastroianni, P.C. Causes for the underreporting of adverse drug events by health professionals: a systematic review. Rev. Esc. Enferm. USP 48, 739-747 (2014).
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805 (2018).
Sushil, M., Ludwig, D., Butte, A.J. & Rudrapatna, V.A. Developing a general-purpose clinical language inference model from a large corpus of clinical notes. arXiv preprint arXiv:221006566 (2022).
Henry, S., Buchan, K., Filannino, M., Stubbs, A. & Uzuner, O. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J. Am. Med. Inform. Assoc. 27, 3-12 (2019).
Code of Federal Regulations (21CFR312.32). Title 21, Chapter I, SubChapter D, Part 312.32. https://www.ecfr.gov/current/title-21/chapter-I/subchapter-D/part-312.
Agency EM. ICH Topic E9 Statistical Principles for Clinical Trials. https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e-9-statistical-principles-clinical-trials-step-5_en.pdf.
Norgeot, B. et al. Protected Health Information Filter (philter): accurately and securely de-identifying free-text clinical notes. NPJ Digit Med. 3, 57 (2020).
Savova, G.K. et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J. Am. Med. Inform. Assoc. 17, 507-513 (2010).
Bodenreider, O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 32(Suppl 1), D267-D270 (2004).
Label Studio: Data Labeling Software [computer program]. 2020-2022.
Pedregosa, F. et al. Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825-2830 (2011).
Erickson, N. et al. Autogluon-tabular: robust and accurate automl for structured data. arXiv preprint arXiv:200306505 (2020).
Lecun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278-2324 (1998).
LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. Paper presented at: Advances in Neural Information Processing Systems 21989.
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735-1780 (1997).
Ji, S., Hölttä, M. & Marttinen, P. Does the magic of BERT apply to medical code assignment? A quantitative study. Comput. Biol. Med. 139, 104998 (2021).
He, J. & Cheng, M.X. Weighting methods for rare event identification from imbalanced datasets. Front. Big Data 4, 715320 (2021).
Medical Dictionary for Regulatory Activities [computer program].
Quezada, S.M., McLean, L.P. & Cross, R.K. Adverse events in IBD therapy: the 2018 update. Expert Rev. Gastroenterol. Hepatol. 12, 1183-1191 (2018).
Anderson, K., Moss, K., Campbell, B., Moote, D., Kakazu, K. & Hyams, J.S. Follicular dendritic cell sarcoma in a patient with adolescent-onset Crohn's disease exposed to multiple immunomodulator and biologic therapies. JPGN Rep. 3, e231 (2022).
Smith, M. Anti-TNFα therapy did not increase short-or medium-term risk for cancer in patients with rheumatoid arthritis. Ann. Inter. Med. 152, JC5-JC13 (2010).
Conti, F. et al. The influence of comorbidities on the efficacy of tumour necrosis factor inhibitors, and the effect of tumour necrosis factor inhibitors on comorbidities in rheumatoid arthritis: report from a National Consensus Conference. Rheumatology 57(Suppl 7), vii11-vii22 (2018).
Statsmodels [computer program]. 2009-2022.