Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials.
AB testing
alert fatigue
clinical decision support
clinical informatics
randomized controlled trials
usability
Journal
Journal of medical Internet research
ISSN: 1438-8871
Titre abrégé: J Med Internet Res
Pays: Canada
ID NLM: 100959882
Informations de publication
Date de publication:
09 04 2021
09 04 2021
Historique:
received:
14
10
2019
accepted:
11
03
2021
revised:
14
08
2020
entrez:
9
4
2021
pubmed:
10
4
2021
medline:
30
9
2021
Statut:
epublish
Résumé
Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools. This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care. A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior. To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images. These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS. Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.
Sections du résumé
BACKGROUND
Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools.
OBJECTIVE
This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care.
METHODS
A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior.
RESULTS
To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images.
CONCLUSIONS
These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS.
TRIAL REGISTRATION
Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.
Identifiants
pubmed: 33835035
pii: v23i4e16651
doi: 10.2196/16651
pmc: PMC8065554
doi:
Banques de données
ClinicalTrials.gov
['NCT03415425', 'NCT03714191']
Types de publication
Journal Article
Randomized Controlled Trial
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
e16651Informations de copyright
©Jonathan Austrian, Felicia Mendoza, Adam Szerencsy, Lucille Fenelon, Leora I Horwitz, Simon Jones, Masha Kuznetsova, Devin M Mann. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 09.04.2021.
Références
N Engl J Med. 2019 Sep 19;381(12):1175-1179
pubmed: 31532967
JAAPA. 2015 Jul;28(7):41-2
pubmed: 26107794
Proc AMIA Symp. 2002;:400-4
pubmed: 12463855
Transplantation. 2004 Oct 15;78(7):1042-7
pubmed: 15480172
JMIR Hum Factors. 2018 Nov 28;5(4):e10721
pubmed: 30487119
Proc Annu Symp Comput Appl Med Care. 1993;:224-8
pubmed: 8130466
J Am Med Inform Assoc. 2018 May 1;25(5):523-529
pubmed: 29025165
Ann Intern Med. 2018 Jul 3;169(1):50-51
pubmed: 29801050
Pilot Feasibility Stud. 2019 Feb 20;5:28
pubmed: 30820339
N Engl J Med. 1990 May 24;322(21):1499-504
pubmed: 2186274
JAMA. 1999 Jul 7;282(1):67-74
pubmed: 10404914
Qual Saf Health Care. 2010 Oct;19(5):e15
pubmed: 20427312
J Am Med Inform Assoc. 2003 Nov-Dec;10(6):523-30
pubmed: 12925543
J Am Med Inform Assoc. 2014 Oct;21(e2):e287-96
pubmed: 24668841
Arch Intern Med. 2006 May 22;166(10):1098-104
pubmed: 16717172
CMAJ. 2003 Sep 16;169(6):549-56
pubmed: 12975221
Evid Based Med. 2016 Dec;21(6):203-207
pubmed: 27664174
Lancet. 1995 Aug 5;346(8971):341-6
pubmed: 7623532
Acad Emerg Med. 2005 Mar;12(3):225-31
pubmed: 15741585
JAMA. 2005 Mar 9;293(10):1223-38
pubmed: 15755945
J Am Med Inform Assoc. 2006 Jan-Feb;13(1):5-11
pubmed: 16221941
Stud Health Technol Inform. 2016;226:51-4
pubmed: 27350464
J Am Med Inform Assoc. 2011 Nov-Dec;18(6):783-8
pubmed: 21712374
J Am Med Inform Assoc. 1997 Sep-Oct;4(5):364-75
pubmed: 9292842
JAMA. 1998 Oct 21;280(15):1339-46
pubmed: 9794315
Int J Med Inform. 2013 Jun;82(6):492-503
pubmed: 23490305
MD Comput. 1996 Jan-Feb;13(1):46-54, 63
pubmed: 8569464
Health Informatics J. 2007 Sep;13(3):163-77
pubmed: 17711879
J Am Geriatr Soc. 2006 Jun;54(6):963-8
pubmed: 16776793
Appl Clin Inform. 2014 Dec 17;5(4):1015-25
pubmed: 25589914
JMIR Hum Factors. 2018 Dec 19;5(4):e11048
pubmed: 30567688
J Am Med Inform Assoc. 2015 Nov;22(6):1243-50
pubmed: 25829460