User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications.
Journal
Anesthesia and analgesia
ISSN: 1526-7598
Titre abrégé: Anesth Analg
Pays: United States
ID NLM: 1310650
Informations de publication
Date de publication:
01 Apr 2024
01 Apr 2024
Historique:
pmc-release:
20
12
2024
medline:
18
3
2024
pubmed:
20
6
2023
entrez:
20
6
2023
Statut:
ppublish
Résumé
Machine learning models can help anesthesiology clinicians assess patients and make clinical and operational decisions, but well-designed human-computer interfaces are necessary for machine learning model predictions to result in clinician actions that help patients. Therefore, the goal of this study was to apply a user-centered design framework to create a user interface for displaying machine learning model predictions of postoperative complications to anesthesiology clinicians. Twenty-five anesthesiology clinicians (attending anesthesiologists, resident physicians, and certified registered nurse anesthetists) participated in a 3-phase study that included (phase 1) semistructured focus group interviews and a card sorting activity to characterize user workflows and needs; (phase 2) simulated patient evaluation incorporating a low-fidelity static prototype display interface followed by a semistructured interview; and (phase 3) simulated patient evaluation with concurrent think-aloud incorporating a high-fidelity prototype display interface in the electronic health record. In each phase, data analysis included open coding of session transcripts and thematic analysis. During the needs assessment phase (phase 1), participants voiced that (a) identifying preventable risk related to modifiable risk factors is more important than nonpreventable risk, (b) comprehensive patient evaluation follows a systematic approach that relies heavily on the electronic health record, and (c) an easy-to-use display interface should have a simple layout that uses color and graphs to minimize time and energy spent reading it. When performing simulations using the low-fidelity prototype (phase 2), participants reported that (a) the machine learning predictions helped them to evaluate patient risk, (b) additional information about how to act on the risk estimate would be useful, and (c) correctable problems related to textual content existed. When performing simulations using the high-fidelity prototype (phase 3), usability problems predominantly related to the presentation of information and functionality. Despite the usability problems, participants rated the system highly on the System Usability Scale (mean score, 82.5; standard deviation, 10.5). Incorporating user needs and preferences into the design of a machine learning dashboard results in a display interface that clinicians rate as highly usable. Because the system demonstrates usability, evaluation of the effects of implementation on both process and clinical outcomes is warranted.
Sections du résumé
BACKGROUND
BACKGROUND
Machine learning models can help anesthesiology clinicians assess patients and make clinical and operational decisions, but well-designed human-computer interfaces are necessary for machine learning model predictions to result in clinician actions that help patients. Therefore, the goal of this study was to apply a user-centered design framework to create a user interface for displaying machine learning model predictions of postoperative complications to anesthesiology clinicians.
METHODS
METHODS
Twenty-five anesthesiology clinicians (attending anesthesiologists, resident physicians, and certified registered nurse anesthetists) participated in a 3-phase study that included (phase 1) semistructured focus group interviews and a card sorting activity to characterize user workflows and needs; (phase 2) simulated patient evaluation incorporating a low-fidelity static prototype display interface followed by a semistructured interview; and (phase 3) simulated patient evaluation with concurrent think-aloud incorporating a high-fidelity prototype display interface in the electronic health record. In each phase, data analysis included open coding of session transcripts and thematic analysis.
RESULTS
RESULTS
During the needs assessment phase (phase 1), participants voiced that (a) identifying preventable risk related to modifiable risk factors is more important than nonpreventable risk, (b) comprehensive patient evaluation follows a systematic approach that relies heavily on the electronic health record, and (c) an easy-to-use display interface should have a simple layout that uses color and graphs to minimize time and energy spent reading it. When performing simulations using the low-fidelity prototype (phase 2), participants reported that (a) the machine learning predictions helped them to evaluate patient risk, (b) additional information about how to act on the risk estimate would be useful, and (c) correctable problems related to textual content existed. When performing simulations using the high-fidelity prototype (phase 3), usability problems predominantly related to the presentation of information and functionality. Despite the usability problems, participants rated the system highly on the System Usability Scale (mean score, 82.5; standard deviation, 10.5).
CONCLUSIONS
CONCLUSIONS
Incorporating user needs and preferences into the design of a machine learning dashboard results in a display interface that clinicians rate as highly usable. Because the system demonstrates usability, evaluation of the effects of implementation on both process and clinical outcomes is warranted.
Identifiants
pubmed: 37339083
doi: 10.1213/ANE.0000000000006577
pii: 00000539-990000000-00600
pmc: PMC10730770
mid: NIHMS1899575
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
804-813Subventions
Organisme : NCATS NIH HHS
ID : KL2 TR002346
Pays : United States
Organisme : NINR NIH HHS
ID : R01 NR017916
Pays : United States
Organisme : NIGMS NIH HHS
ID : T32 GM108539
Pays : United States
Informations de copyright
Copyright © 2023 International Anesthesia Research Society.
Déclaration de conflit d'intérêts
The authors declare no conflicts of interest.
Références
Char DS, Burgart A. Machine-learning implementation in clinical anesthesia: opportunities and challenges. Anesth Analg. 2020;130:1709–1712.
Hofer IS, Burns M, Kendale S, Wanderer JP. Realistically integrating machine learning into clinical practice: a road map of opportunities, challenges, and a potential future. Anesth Analg. 2020;130:1115–1118.
Buschman TJ, Siegel M, Roy JE, Miller EK. Neural substrates of cognitive capacity limitations. Proc Natl Acad Sci U S A. 2011;108:11252–11255.
Stiegler MP, Tung A. Cognitive processes in anesthesiology decision making. Anesthesiology. 2014;120:204–217.
Ehrmann DE, Gallant SN, Nagaraj S, et al. Evaluating and reducing cognitive load should be a priority for machine learning in healthcare. Nat Med. 2022;28:1331–1333.
Bellini V, Guzzon M, Bigliardi B, Mordonini M, Filippelli S, Bignami E. Artificial intelligence: a new tool in operating room management. Role of machine learning models in operating room optimization. J Med Syst. 2019;44:20.
Meier JM, Tschoellitsch T. Artificial intelligence and machine learning in patient blood management: a scoping review. Anesth Analg. 2022;135:524–531.
Palla K, Hyland SL, Posner K, et al. Intraoperative prediction of postanaesthesia care unit hypotension. Br J Anaesth. 2022;128:623–635.
Wijnberge M, Geerts BF, Hol L, et al. Effect of a machine learning–derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial. JAMA. 2020;323:1052–1060.
Lundberg SM, Nair B, Vavilala MS, et al. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nat Biomed Eng. 2018;2:749–760.
Kang AR, Lee J, Jung W, et al. Development of a prediction model for hypotension after induction of anesthesia using machine learning. PLoS One. 2020;15:e0231172.
Wright MC, Borbolla D, Waller RG, et al. Critical care information display approaches and design frameworks: a systematic review and meta-analysis. J Biomed Inform. 2019;100:100041.
Fritz BA, Cui Z, Zhang M, et al. Deep-learning model for predicting 30-day postoperative mortality. Br J Anaesth. 2019;123:688–695.
Cui Z, Fritz BA, King CR, Avidan MS, Chen Y. A factored generalized additive model for clinical decision support in the operating room. AMIA Annu Symp Proc. 2019;2019:343–352.
Xue B, Li D, Lu C, et al. Use of machine learning to develop and evaluate models using preoperative and intraoperative data to identify risks of postoperative complications. JAMA Netw Open. 2021;4:e212240–e212240.
Ericsson A. Valid and non-reactive verbalization of thoughts during performance of tasks towards a solution to the central problems of introspection as a source of scientific data. J Conscious Stud. 2003;10:1–18.
O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89:1245–1251.
Gregory S, Murray-Torres TM, Fritz BA, et al.; ACTFAST Study Group. Study protocol for the anesthesiology control tower-feedback alerts to supplement treatments (ACTFAST-3) trial: a pilot randomized controlled trial in intraoperative telemedicine. F1000Res. 2018;7:623.
King CR, Abraham J, Kannampallil TG, et al.; TECTONICS Research Group. Protocol for the effectiveness of an anesthesiology control tower system in improving perioperative quality metrics and clinical outcomes: the TECTONICS randomized, pragmatic trial. F1000Res. 2019;8:2032.
Brooke J. SUS: a quick and dirty usability scale. Usability Evaluation in Industry. 1995:189.
Keenan SL, Hartson HR, Kafura DG, Schulman RS. The usability problem taxonomy: a framework for classification and analysis. Empir Softw Eng. 1999;4:71–104.
Diprose WK, Buist N, Hua N, Thurier Q, Shand G, Robinson R. Physician understanding, explainability, and trust in a hypothetical machine learning risk calculator. J Am Med Inform Assoc. 2020;27:592–600.
Matthiesen S, Diederichsen SZ, Hansen MKH, et al. Clinician preimplementation perspectives of a decision-support tool for the prediction of cardiac arrhythmia based on machine learning: near-live feasibility and qualitative study. JMIR Human Factors. 2021;8:e26964.
Schwartz JM, George M, Rossetti SC, et al. Factors Influencing Clinician Trust in Predictive Clinical Decision Support Systems for In-Hospital Deterioration: Qualitative Descriptive Study. JMIR Human Factors. 2022;9:e33960.
Abraham J, Bartek B, Meng A, et al. Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool. J Biomed Inform. 2023;137:104270.
Barda AJ, Horvat CM, Hochheiser H. A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare. BMC Med Inform Decis Mak. 2020;20:257.
Helman S, Terry MA, Pellathy T, et al. Engaging clinicians early during the development of a graphical user display of an intelligent alerting system at the bedside. Int J Med Inform. 2022;159:104643.
Magunia H, Lederer S, Verbuecheln R, et al. Machine learning identifies ICU outcome predictors in a multicenter COVID-19 cohort. Crit Care. 2021;25:295.
Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health. 2021;3:e745–e750.
Springer A, Whittaker S. Progressive disclosure: empirically motivated approaches to designing effective transparency. Paper presented at: Proceedings of the 24th International Conference on Intelligent User Interfaces. 2019. Marina del Rey, CA, March 17-20, 2019. https://doi.org/10.1145/3301275.3302322
Kheterpal S, Shanks A, Tremper KK. Impact of a novel multiparameter decision support system on intraoperative processes of care and postoperative outcomes. Anesthesiology. 2018;128:272–282.
Chi EA, Chi G, Tsui CT, et al. Development and validation of an artificial intelligence system to optimize clinician review of patient records. JAMA Netw Open. 2021;4:e2117391.