Development, Implementation, and Evaluation of a Personalized Machine Learning Algorithm for Clinical Decision Support: Case Study With Shingles Vaccination.
alert fatigue
clinical decision support
implementation science
machine learning
Journal
Journal of medical Internet research
ISSN: 1438-8871
Titre abrégé: J Med Internet Res
Pays: Canada
ID NLM: 100959882
Informations de publication
Date de publication:
29 04 2020
29 04 2020
Historique:
received:
30
10
2019
accepted:
21
02
2020
revised:
19
02
2020
entrez:
30
4
2020
pubmed:
30
4
2020
medline:
6
11
2020
Statut:
epublish
Résumé
Although clinical decision support (CDS) alerts are effective reminders of best practices, their effectiveness is blunted by clinicians who fail to respond to an overabundance of inappropriate alerts. An electronic health record (EHR)-integrated machine learning (ML) algorithm is a potentially powerful tool to increase the signal-to-noise ratio of CDS alerts and positively impact the clinician's interaction with these alerts in general. This study aimed to describe the development and implementation of an ML-based signal-to-noise optimization system (SmartCDS) to increase the signal of alerts by decreasing the volume of low-value herpes zoster (shingles) vaccination alerts. We built and deployed SmartCDS, which builds personalized user activity profiles to suppress shingles vaccination alerts unlikely to yield a clinician's interaction. We extracted all records of shingles alerts from January 2017 to March 2019 from our EHR system, including 327,737 encounters, 780 providers, and 144,438 patients. During the 6 weeks of pilot deployment, the SmartCDS system suppressed an average of 43.67% (15,425/35,315) potential shingles alerts (appointments) and maintained stable counts of weekly shingles vaccination orders (326.3 with system active vs 331.3 in the control group; P=.38) and weekly user-alert interactions (1118.3 with system active vs 1166.3 in the control group; P=.20). All key statistics remained stable while the system was turned on. Although the results are promising, the characteristics of the system can be subject to future data shifts, which require automated logging and monitoring. We demonstrated that an automated, ML-based method and data architecture to suppress alerts are feasible without detriment to overall order rates. This work is the first alert suppression ML-based model deployed in practice and serves as foundational work in encounter-level customization of alert display to maximize effectiveness.
Sections du résumé
BACKGROUND
Although clinical decision support (CDS) alerts are effective reminders of best practices, their effectiveness is blunted by clinicians who fail to respond to an overabundance of inappropriate alerts. An electronic health record (EHR)-integrated machine learning (ML) algorithm is a potentially powerful tool to increase the signal-to-noise ratio of CDS alerts and positively impact the clinician's interaction with these alerts in general.
OBJECTIVE
This study aimed to describe the development and implementation of an ML-based signal-to-noise optimization system (SmartCDS) to increase the signal of alerts by decreasing the volume of low-value herpes zoster (shingles) vaccination alerts.
METHODS
We built and deployed SmartCDS, which builds personalized user activity profiles to suppress shingles vaccination alerts unlikely to yield a clinician's interaction. We extracted all records of shingles alerts from January 2017 to March 2019 from our EHR system, including 327,737 encounters, 780 providers, and 144,438 patients.
RESULTS
During the 6 weeks of pilot deployment, the SmartCDS system suppressed an average of 43.67% (15,425/35,315) potential shingles alerts (appointments) and maintained stable counts of weekly shingles vaccination orders (326.3 with system active vs 331.3 in the control group; P=.38) and weekly user-alert interactions (1118.3 with system active vs 1166.3 in the control group; P=.20).
CONCLUSIONS
All key statistics remained stable while the system was turned on. Although the results are promising, the characteristics of the system can be subject to future data shifts, which require automated logging and monitoring. We demonstrated that an automated, ML-based method and data architecture to suppress alerts are feasible without detriment to overall order rates. This work is the first alert suppression ML-based model deployed in practice and serves as foundational work in encounter-level customization of alert display to maximize effectiveness.
Identifiants
pubmed: 32347813
pii: v22i4e16848
doi: 10.2196/16848
pmc: PMC7221637
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
e16848Informations de copyright
©Ji Chen, Sara Chokshi, Roshini Hegde, Javier Gonzalez, Eduardo Iturrate, Yin Aphinyanaphongs, Devin Mann. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 29.04.2020.
Références
J Am Med Inform Assoc. 2012 Jan-Feb;19(1):66-71
pubmed: 21890873
Implement Sci. 2018 Aug 20;13(1):114
pubmed: 30126421
MD Comput. 1992 Sep-Oct;9(5):304-12
pubmed: 1522792
Comput Math Methods Med. 2011;2011:143480
pubmed: 21461385
J Am Med Inform Assoc. 2015 Mar;22(2):361-9
pubmed: 25318641
Transplantation. 2004 Oct 15;78(7):1042-7
pubmed: 15480172
J Crit Care. 2012 Jun;27(3):242-9
pubmed: 22520497
J Am Med Inform Assoc. 2006 Jan-Feb;13(1):5-11
pubmed: 16221941
Comput Cardiol (2010). 2015 Sep;2015:273-276
pubmed: 27331073
J Biomed Inform. 2015 Aug;56:229-38
pubmed: 26044081
J Am Med Inform Assoc. 1997 Sep-Oct;4(5):364-75
pubmed: 9292842
Appl Clin Inform. 2010 Sep 29;1(3):346-62
pubmed: 23616845
Ann Emerg Med. 2016 Feb;67(2):240-248.e3
pubmed: 26553282
Qual Saf Health Care. 2010 Oct;19(5):e15
pubmed: 20427312
J Hosp Med. 2015 Jun;10(6):345-51
pubmed: 25873486
J Clin Monit Comput. 2017 Feb;31(1):213-219
pubmed: 26621389
Arch Intern Med. 2006 May 22;166(10):1098-104
pubmed: 16717172
Am J Crit Care. 2008 Jan;17(1):36-41
pubmed: 18158387
CMAJ. 2003 Sep 16;169(6):549-56
pubmed: 12975221
AMIA Annu Symp Proc. 2011;2011:1036-44
pubmed: 22195164
Physiol Meas. 2013 Apr;34(4):465-78
pubmed: 23524637
Crit Care Med. 2016 Jul;44(7):e456-63
pubmed: 26992068
Lancet. 1995 Aug 5;346(8971):341-6
pubmed: 7623532
Acad Emerg Med. 2005 Mar;12(3):225-31
pubmed: 15741585
Ann Intern Med. 2018 Feb 6;168(3):210-220
pubmed: 29404596
Pharmacoepidemiol Drug Saf. 2011 Sep;20(9):930-8
pubmed: 21774031
Arch Intern Med. 2009 Feb 9;169(3):305-11
pubmed: 19204222
Proc Annu Symp Comput Appl Med Care. 1993;:224-8
pubmed: 8130466
J Am Med Inform Assoc. 2019 Oct 1;26(10):1141-1149
pubmed: 31206159
J Am Med Inform Assoc. 2012 Jun;19(e1):e145-8
pubmed: 22534081
N Engl J Med. 2001 Sep 27;345(13):965-70
pubmed: 11575289
Anaesthesia. 2009 Feb;64(2):131-5
pubmed: 19143688
J Am Med Inform Assoc. 2007 Mar-Apr;14(2):141-5
pubmed: 17213487
Ochsner J. 2014 Summer;14(2):195-202
pubmed: 24940129
J Am Med Inform Assoc. 2009 Jan-Feb;16(1):40-6
pubmed: 18952941
J Am Med Inform Assoc. 2008 May-Jun;15(3):311-20
pubmed: 18308989
J Am Med Inform Assoc. 2006 Mar-Apr;13(2):138-47
pubmed: 16357358
J Biomed Inform. 2012 Oct;45(5):913-21
pubmed: 22465785
N Engl J Med. 1990 May 24;322(21):1499-504
pubmed: 2186274
JAMA. 1998 Oct 21;280(15):1339-46
pubmed: 9794315
MD Comput. 1996 Jan-Feb;13(1):46-54, 63
pubmed: 8569464
BMJ Open. 2018 Jan 26;8(1):e017833
pubmed: 29374661
J Am Med Inform Assoc. 2015 Jul;22(4):881-7
pubmed: 25911673
Pediatrics. 2013 Jun;131(6):e1970-3
pubmed: 23713099
Health Informatics J. 2007 Sep;13(3):163-77
pubmed: 17711879
J Am Geriatr Soc. 2006 Jun;54(6):963-8
pubmed: 16776793
Int J Med Inform. 2015 Sep;84(9):617-29
pubmed: 26051616
Int J Nurs Stud. 2013 Oct;50(10):1351-8
pubmed: 23474081
J Biomed Inform. 2008 Apr;41(2):387-92
pubmed: 18029232