Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation.
Journal
Health care management review
ISSN: 1550-5030
Titre abrégé: Health Care Manage Rev
Pays: United States
ID NLM: 7611530
Informations de publication
Date de publication:
Historique:
pubmed:
14
9
2021
medline:
19
4
2022
entrez:
13
9
2021
Statut:
ppublish
Résumé
Health care organizations are integrating a variety of machine learning (ML)-based clinical decision support (CDS) tools into their operations, but practitioners lack clear guidance regarding how to implement these tools so that they assist end users in their work. We designed this study to identify how health care organizations can facilitate collaborative development of ML-based CDS tools to enhance their value for health care delivery in real-world settings. We utilized qualitative methods, including 37 interviews in a large, multispecialty health system that developed and implemented two operational ML-based CDS tools in two of its hospital sites. We performed thematic analyses to inform presentation of an explanatory framework and recommendations. We found that ML-based CDS tool development and implementation into clinical workflows proceeded in four phases: iterative solution coidentification, iterative coengagement, iterative coapplication, and iterative corefinement. Each phase is characterized by a collaborative back-and-forth process between the technology's developers and users, through which both users' activities and the technology itself are transformed. Health care organizations that anticipate iterative collaboration to be an integral aspect of their ML-based CDS tools' development and implementation process may have more success in deploying ML-based CDS tools that assist end users in their work than organizations that expect a traditional technology innovation process. Managers developing and implementing ML-based CDS tools should frame the work as a collaborative learning opportunity for both users and the technology itself and should solicit constructive feedback from users on potential changes to the technology, in addition to potential changes to user workflows, in an ongoing, iterative manner.
Sections du résumé
BACKGROUND
Health care organizations are integrating a variety of machine learning (ML)-based clinical decision support (CDS) tools into their operations, but practitioners lack clear guidance regarding how to implement these tools so that they assist end users in their work.
PURPOSE
We designed this study to identify how health care organizations can facilitate collaborative development of ML-based CDS tools to enhance their value for health care delivery in real-world settings.
METHODOLOGY/APPROACH
We utilized qualitative methods, including 37 interviews in a large, multispecialty health system that developed and implemented two operational ML-based CDS tools in two of its hospital sites. We performed thematic analyses to inform presentation of an explanatory framework and recommendations.
RESULTS
We found that ML-based CDS tool development and implementation into clinical workflows proceeded in four phases: iterative solution coidentification, iterative coengagement, iterative coapplication, and iterative corefinement. Each phase is characterized by a collaborative back-and-forth process between the technology's developers and users, through which both users' activities and the technology itself are transformed.
CONCLUSION
Health care organizations that anticipate iterative collaboration to be an integral aspect of their ML-based CDS tools' development and implementation process may have more success in deploying ML-based CDS tools that assist end users in their work than organizations that expect a traditional technology innovation process.
PRACTICE IMPLICATIONS
Managers developing and implementing ML-based CDS tools should frame the work as a collaborative learning opportunity for both users and the technology itself and should solicit constructive feedback from users on potential changes to the technology, in addition to potential changes to user workflows, in an ongoing, iterative manner.
Identifiants
pubmed: 34516438
doi: 10.1097/HMR.0000000000000324
pii: 00004010-900000000-99601
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
E21-E31Informations de copyright
Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.
Références
Amarasingham R., Moore B. J., Tabak Y. P., Drazner M. H., Clark C. A., Zhang S., Reed W. G., Swanson T. S., Ma Y., Halm E. A. (2010). An automated model to identify heart failure patients at risk for 30-day readmission or death using electronic medical record data. Medical Care, 48, 981–988.
Angelo S. A., Arruda E. F., Goldwasser R., Lobo M. S., Salles A. (2017). Demand forecast and optimal planning of intensive care unit (ICU) capacity. Pesquisa Operacional, 37(2), 229–245.
Barnes S., Hamrock E., Toerper M., Siddiqui S., Levin S. (2016). Real-time prediction of inpatient length of stay for discharge prioritization. Journal of the American Medical Informatics Association, 23(e1), e2–e10.
Barrett M., Oborn E., Orlikowski W. J., Yates J. (2012). Reconfiguring boundary relations: Robotic innovations in pharmacy work. Organization Science, 23(5), 1448–1466.
Becker H. S. (1998). Tricks of the trade: How to think about your research while you're doing it. University of Chicago Press.
Brajer N., Cozzi B., Gao M., Nichols M., Revoir M., Balu S., Futoma J., Bae J., Setji N., Hernandez A., Sendak M. (2020). Prospective and external evaluation of a machine learning model to predict in-hospital mortality of adults at time of admission. JAMA Network Open, 3(2), e1920733.
Chalfin D. B., Trzeciak S., Likourezos A., Baumann B. M., Dellinger R. P.; DELAY-ED Study Group (2007). Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit. Critical Care Medicine, 35(6), 1477–1483.
Cheng F.-Y., Joshi H., Tandon P., Freeman R., Reich D. L., Mazumdar M., Kohli-Seth R., Levin M., Timsina P., Kia A. (2020). Using machine learning to predict ICU transfer in hospitalized COVID-19 patients. Journal of Clinical Medicine, 9(6), 1668.
Elish M. C. (2018). The stakes of uncertainty: Developing and integrating machine learning in clinical care. Paper presented at the Ethnographic Praxis in Industry Conference Proceedings.
Elish M. C., Watkins E. (2020). Repairing innovation: A study of integrating AI in clinical care[Unpublished manuscript].
Gerhardt G., Yemane A., Hickman P., Oelschlaeger A., Rollins E., Brennan N. (2013). Data shows reduction in Medicare hospital readmission rates during 2012. Medicare & Medicaid Research Review, 3(2), E1–E11.
Glaser B., Strauss A. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine Publishing.
Goodman D., Fisher E., Chang C. (2013). The revolving door (RWJF404178). https://www.rwjf.org/en/library/research/2013/02/the-revolving-door--a-report-on-u-s--hospital-readmissions.html
He D., Mathews S. C., Kalloo A. N., Hutfless S. (2014). Mining high-dimensional administrative claims data to predict early hospital readmissions. Journal of the American Medical Informatics Association, 21(2), 272–279.
Jamei M., Nisnevich A., Wetchler E., Sudat S., Liu E. (2017). Predicting all-cause risk of 30-day hospital readmission using artificial neural networks. PLOS ONE, 12(7), e0181173.
Jamoom E., Yang N., Hing E. (2016). Adoption of certified electronic health record systems and electronic information sharing in physician offices: United States, 2013 and 2014. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention.
Kansagara D., Englander H., Salanitro A., Kagen D., Theobald C., Freeman M., Kripalani S. (2011). Risk prediction models for hospital readmission: A systematic review. JAMA, 306(15), 1688–1698.
Kong C. W., Wilkinson T. M. A. (2020). Predicting and preventing hospital readmission for exacerbations of COPD. ERJ Open Research, 6(2), 00325-2019.
Lai H.-J., Tan T.-H., Lin C.-S., Chen Y.-F., Lin H.-H. (2020). Designing a clinical decision support system to predict readmissions for patients admitted with all-cause conditions. Journal of Ambient Intelligence and Humanized Computing, 1–10.
Leonardi P. M. (2011). When flexible routines meet flexible technologies: Affordance, constraint, and the imbrication of human and material agencies. MIS Quarterly, 35(1), 147–167.
Li R. C., Asch S. M., Shah N. H. (2020). Developing a delivery science for artificial intelligence in healthcare. NPJ Digital Medicine, 3, 107.
Litvak N., Van Rijsbergen M., Boucherie R. J., Van Houdenhoven M. (2008). Managing the overflow of intensive care patients. European Journal of Operational Research, 185(3), 998–1010.
Lobach D., Sanders G. D., Bright T. J., Wong A., Dhurjati R., Bristow E., Bastian L., Coeytaux R., Samsa G., Hasselblad V., Williams J. W., Wing L., Musty M., Kendrick A. S. (2012). Enabling health care decisionmaking through clinical decision support and knowledge management. Evidence Report/Technology Assessment, (203), 1–784.
McManus M. L., Long M. C., Cooper A., Litvak E. (2004). Queuing theory accurately models the need for critical care resources. Anesthesiology, 100(5), 1271–1276.
McWilliams C. J., Lawson D. J., Santos-Rodriguez R., Gilchrist I. D., Champneys A., Gould T. H., Thomas M. J., Bourdeaux C. P. (2019). Towards a decision support tool for intensive care discharge: Machine learning algorithm development using electronic healthcare data from MIMIC-III and Bristol, UK. BMJ Open, 9(3), e025925.
Miles M. B., Huberman A. M. (1984). Drawing valid meaning from qualitative data: Toward a shared craft. Educational Researcher, 13(5), 20–30.
Moore B. J., White S., Washington R., Coenen N., Elixhauser A. (2017). Identifying increased risk of readmission and in-hospital mortality using hospital administrative data. Medical Care, 55(7), 698–705.
Morgan D. J., Bame B., Zimand P., Dooley P., Thom K. A., Harris A. D., Bentzen S., Ettinger W., Garrett-Ray S. D., Tracy J. K., Liang Y. (2019). Assessment of machine learning vs. standard prediction rules for predicting hospital readmissions. JAMA Network Open, 2(3), e190348.
Mostashari F. (2012). Foreward. In Improving outcomes with clinical decision support: An implementer's guide (pp. xvii–xviii). CRC Press.
Mulholland M. W., Abrahamse P., Bahl V. (2005). Linear programming to optimize performance in a department of surgery. Journal of the American College of Surgeons, 200(6), 861–868.
Pickering A. (1995). The Mangle of Practice: Time, Agency, and Science. University of Chicago Press.
Rojas J. C., Carey K. A., Edelson D. P., Venable L. R., Howell M. D., Churpek M. M. (2018). Predicting intensive care unit readmission with machine learning using electronic health record data. Annals of the American Thoracic Society, 15(7), 846–853.
Romero-Brufau S., Wyatt K. D., Boyum P., Mickelson M., Moore M., Cognetta-Rieke C. (2020). Implementation of artificial intelligence-based clinical decision support to reduce hospital readmissions at a regional hospital. Applied Clinical Informatics, 11(4), 570–577.
Ryckman F. C., Yelton P. A., Anneken A. M., Kiessling P. E., Schoettker P. J., Kotagal U. R. (2009). Redesigning intensive care unit flow using variability management to improve access and safety. The Joint Commission Journal on Quality and Patient Safety, 35(11), 535–543.
Sandhu S., Lin A. L., Brajer N., Sperling J., Ratliff W., Bedoya A. D., Balu S., O'Brien C., Sendak M. P. (2020). Integrating a machine learning system into clinical workflows: Qualitative study. Journal of Medical Internet Research, 22(11), e22421.
Sendak M., Elish M. C., Gao M., Futoma J., Ratliff W., Nichols M., Bedoya A., Balu S., O'Brien C. (2020). “The human body is a black box” supporting clinical decision-making with deep learning. Paper presented at the Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency.
Sendak M., Gao M., Nichols M., Lin A., Balu S. (2019). Machine learning in health care: A critical appraisal of challenges and opportunities. eGEMs (Generating Evidence & Methods to Improve Patient Outcomes), 7(1), 1. 10.5334/egems.287/
doi: 10.5334/egems.287/
Shmueli A., Sprung C. L., Kaplan E. H. (2003). Optimizing admissions to an intensive care unit. Health Care Management Science, 6(3), 131–136.
Tcheng J. E. (2017). Optimizing strategies for clinical decision support: Summary of a meeting series. National Academy of Medicine.
Van Houdenhoven M., van Oostrum J. M., Wullink G., Hans E., Hurink J. L., Bakker J., Kazemier G. (2008). Fewer intensive care unit refusals and a higher capacity utilization by using a cyclic surgical case schedule. Journal of Critical Care, 23(2), 222–226.
Zhu Z., Hen B. H., Teow K. L. (2012). Estimating ICU bed capacity using discrete event simulation. International Journal of Health Care Quality Assurance., 25, 134–144.