Covid-19 triage in the emergency department 2.0: how analytics and AI transform a human-made algorithm for the prediction of clinical pathways.
Artificial intelligence
Clinical decision making
Covid-19 triage
Machine learning
Predictive analytics
Journal
Health care management science
ISSN: 1572-9389
Titre abrégé: Health Care Manag Sci
Pays: Netherlands
ID NLM: 9815649
Informations de publication
Date de publication:
Sep 2023
Sep 2023
Historique:
received:
08
10
2021
accepted:
01
06
2023
medline:
8
9
2023
pubmed:
10
7
2023
entrez:
10
7
2023
Statut:
ppublish
Résumé
The Covid-19 pandemic has pushed many hospitals to their capacity limits. Therefore, a triage of patients has been discussed controversially primarily through an ethical perspective. The term triage contains many aspects such as urgency of treatment, severity of the disease and pre-existing conditions, access to critical care, or the classification of patients regarding subsequent clinical pathways starting from the emergency department. The determination of the pathways is important not only for patient care, but also for capacity planning in hospitals. We examine the performance of a human-made triage algorithm for clinical pathways which is considered a guideline for emergency departments in Germany based on a large multicenter dataset with over 4,000 European Covid-19 patients from the LEOSS registry. We find an accuracy of 28 percent and approximately 15 percent sensitivity for the ward class. The results serve as a benchmark for our extensions including an additional category of palliative care as a new label, analytics, AI, XAI, and interactive techniques. We find significant potential of analytics and AI in Covid-19 triage regarding accuracy, sensitivity, and other performance metrics whilst our interactive human-AI algorithm shows superior performance with approximately 73 percent accuracy and up to 76 percent sensitivity. The results are independent of the data preparation process regarding the imputation of missing values or grouping of comorbidities. In addition, we find that the consideration of an additional label palliative care does not improve the results.
Identifiants
pubmed: 37428304
doi: 10.1007/s10729-023-09647-2
pii: 10.1007/s10729-023-09647-2
pmc: PMC10485125
doi:
Types de publication
Multicenter Study
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
412-429Investigateurs
Christoph Spinner
(C)
Maria Madeleine Ruethrich
(MM)
Julia Lanznaster
(J)
Christoph Römmele
(C)
Kai Wille
(K)
Lukas Tometten
(L)
Sebastian Dolff
(S)
Michael von Bergwelt-Baildon
(M)
Uta Merle
(U)
Katja Rothfuss
(K)
Nora Isberner
(N)
Norma Jung
(N)
Siri Göpel
(S)
Juergen Vom Dahl
(J)
Christian Degenhardt
(C)
Richard Strauss
(R)
Beate Gruener
(B)
Lukas Eberwein
(L)
Kerstin Hellwig
(K)
Dominic Rauschning
(D)
Mark Neufang
(M)
Timm Westhoff
(T)
Claudia Raichle
(C)
Murat Akova
(M)
Bjoern-Erik Jensen
(BE)
Joerg Schubert
(J)
Stephan Grunwald
(S)
Anette Friedrichs
(A)
Janina Trauth
(J)
Katja de With
(K)
Wolfgang Guggemos
(W)
Jan Kielstein
(J)
David Heigener
(D)
Philipp Markart
(P)
Robert Bals
(R)
Sven Stieglitz
(S)
Ingo Voigt
(I)
Jorg Taubel
(J)
Milena Milovanovic
(M)
Informations de copyright
© 2023. The Author(s).
Références
Arballa N, Al-Turaiki I (2021) Machine learning approaches in COVID-19 diagnosis, mortality, and severity risk prediction: a review. Informatics in Medicine Unlocked. Online First
Ardabili SF, Mosavi A, Ghamisi P, Ferdinand F, Varkonyi-Koczy AR, Reuter U, Rabczuk T, Atkinson PM (2020) COVID-19 outbreak prediction with machine learning. Algorithms 13(10):1–36
doi: 10.3390/a13100249
Arrieta AB, Díaz-Rodríguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garcia S, Gil-Lopez S, Molina D, Benjamins R, Chatila R, Herrera F (2020) Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 58:82–115
doi: 10.1016/j.inffus.2019.12.012
Azad-Khaneghah P, Neubauer N, Cruz AM, Liu L (2021) Mobile health app usability and quality rating scales: a systematic review. Disabil Rehabil Assist Technol 16(7):712–721
doi: 10.1080/17483107.2019.1701103
Bartenschlager CC, Ebel SS, Kling S, Vehreschild J, Zabel LT, Spinner CD, Schuler A, Heller AR, Borgmann S, Hoffmann R, Rieg S, Messmann H, Hower M, Brunner JO, Hanses F, Römmele C (2022) COVIDAL: a machine learning classifier for digital COVID-19 diagnosis in German hospitals, Working paper, University of Augsburg
Bartenschlager CC, Gassner UM, Römmele C, Brunner JO, Schlögl-Flierl K (2022) The Practical Ethics of Digital COVID-19 Diagnosis and their Legal, Medical, Operational, and Technological Implications, Working Paper, University of Augsburg
Bertsimas D, Borenstein A, Mingardi L et al (2021) Personalized prescription of ACEI/ARBs for hypertensive COVID-19 patients. Health Care Manag Sci 24:339–355
doi: 10.1007/s10729-021-09545-5
Bertsimas D, Boussioux L, Cory-Wright R et al (2021) From predictions to prescriptions: a data-driven response to COVID-19. Health Care Manag Sci 24:253–272
doi: 10.1007/s10729-020-09542-0
Bhagat RC, Patil SS (2015) Enhanced SMOTE algorithm for classification of imbalanced big-data using Random Forest. IEEE Int Advance Comput Conf (IACC) 2015:403–408
doi: 10.1109/IADCC.2015.7154739
Bouttell J, Hawkins N (2021) Evaluation of Triage Tests When Existing Test Capacity Is Constrained: Application to Rapid Diagnostic Testing in COVID-19. Medical Decision Making. Online First
Burdick H, Lam C, Mataraso S, Siefkas A, Braden G, Dellinger RP, McCoy A, Vincent J-L, Green-Saxena A, Barnes G, Hoffman J, Calvert J, Pellegrini E, Das R (2020) Prediction of respiratory decompensation in Covid-19 patients using machine learning: the READY trial. Comput Biol Med 124:103949
doi: 10.1016/j.compbiomed.2020.103949
Carrington AM, Fieguth PW, Qazi H, Holzinger A, Chen HH, Mayr F, Manuel DG (2020) A new concordant partial AUC and partial C statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Med Infor Decis Making 20(4):1–12
Charlson ME, Pompei P, Ales KL, MacKenzie CR (1987) A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 40(5):373–383
doi: 10.1016/0021-9681(87)90171-8
Chen J, Wu L, Zhang J, Zhang L, Gong D, Zhao Y, Chen Q, Huang S, Yang M, Yang X, Hu S, Wang Y, Hu X, Zheng B, Zhang K, Wu H, Dong Z, Xu Y, Zhu Y, Chen X, Zhang M, Yu L, Cheng F, Yu H (2020) Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography. Sci Rep 10(1):19196
Chen T, Wu D, Chen H, Yan W, Yang D, Chen G, Ma K, Xu D, Yu H, Wang H, Wang T, Guo W, Chen J, Ding C, Zhang X, Huang J, Han M, Li S, Luo X, Zhao J, Ning Q (2020) Clinical characteristics of 113 deceased patients with coronavirus disease 2019; retrospective study. BMJ 386:1–12
Dong Y, Peng CYJ (2013) Principled missing data methods for researchers. Springerplus 2:222
doi: 10.1186/2193-1801-2-222
FitzGerald G, Jelinek GA, Scott D et al (2010) Emergency department triage revisited. Emerg Med J 27:86–92
doi: 10.1136/emj.2009.077081
Fuhrman JD, Gorre N, Hu Q, Li H, El Naqa I, Giger ML (2022) A review of explainable and interpretable AI with applications in COVID-19 imaging. Med Phys 49:1–14
doi: 10.1002/mp.15359
Goodfellow I, Begio Y, Courville A (2016) Deep Learning. The MIT Press, Cambridge, Massachusetts
Harrison R, Flood D, Duce D (2013) Usability of mobile applications: literature review and rationale for a new usability model. J Interaction Sci 1:1–16
doi: 10.1186/2194-0827-1-1
Heller AR, Salvador N, Frank M, Schiffner J, Kipke R, Kleber C (2019) Diagnostic precision of triage algorithms for mass casualty incidents, English version. Anaesthesist 68:15–24
doi: 10.1007/s00101-017-0352-y
Holzinger A, Müller H (2021) Toward Human–AI interfaces to support explainability and causability in medical AI. IEEE Comput 54(10):78–86
doi: 10.1109/MC.2021.3092610
ISO 9241–11: Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts, Geneva 2018
Jakob CEM, Mahajan UM, Oswald M et al (2021) Prediction of COVID-19 deterioration in high-risk patients at diagnosis: an early warning score for advanced COVID-19 developed by machine learning. Infection 50(2):359–370
doi: 10.1007/s15010-021-01656-z
Jakob CEM, Kohlmayer F, Meurers T et al (2020) Design and evaluation of a data anonymization pipeline to promote Open Science on COVID-19. Sci Data 7:435
doi: 10.1038/s41597-020-00773-y
Jakob CEM, Borgmann S, Duygu F et al (2021) First results of the “Lean European Open Survey on SARS-CoV-2-Infected Patients (LEOSS).” Infection 49:63–73
doi: 10.1007/s15010-020-01499-0
Jakobsen JC, Gluud C, Wetterslev J et al (2017) When and how should multiple imputation be used for handling missing data in randomised clinical trials – a practical guide with flowcharts. BMC Med Res Methodol 17:162
doi: 10.1186/s12874-017-0442-1
Jaziri R, Alnahdi S (2020) Choosing which COVID-19 patient to save? The ethical triage and rationing dilemma. Ethics Med Public Health 15:100570
Jiang X, Coffee M, Bari A, Wang J, Jiang X, Huang J, Shi J, Dai J, Cai J, Zhang T, Wu Z, He G, Huang Y (2020) Towards an artificial intelligence framework for data-driven prediction of coronavirus clinical severity. Comput Mater Continua 63(1):537–551
doi: 10.32604/cmc.2020.010691
Joebges S, Biller-Andorno N (2020) Ethics guidelines on COVID-19 triage—an emerging international consensus. Crit Care 24(1):1–5
doi: 10.1186/s13054-020-02927-1
Lalmuanawma S, Hussain J, Chhakchhuak L (2020) Applications of machine learning and artificial intelligence for Covid-19 (SARS-CoV-2) pandemic: a review. Chaos Solitons Fractals 139:110059
doi: 10.1016/j.chaos.2020.110059
Lepenioti K, Bousdekis A, Apostolou D, Mentzas G (2020) Prescriptive analytics: literature review and research challenges. Int J Inf Manag 50:57–70
doi: 10.1016/j.ijinfomgt.2019.04.003
Li X, Xu S, Yu M, Wang K, Tao Y, Zhou Y, Shi J, Zhou M, Wu B, Yang Z, Zhang C, Yue J, Zhang Z, Renz H, Liu X, Xie J, Xie M, Zhao J (2020) Risk factors for severity and mortality in adult COVID-19 inpatients in Wuhan. J Allergy Clin Immunol 146(1):110–118
doi: 10.1016/j.jaci.2020.04.006
Liang W, Liang H, Ou L, Chen B, Chen A, Li C, Li Y, Guan W, Sang L, Lu J, Xu Y, Chen G, Guo H, Guo J, Chen Z, Zhao Y, Li S, Zhang N, Zhong N, He J (2020) Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med 180(8):1081–1089
doi: 10.1001/jamainternmed.2020.2033
Mei X, Lee HC, Diao KY, Huang M, Lin B, Liu C, Xie Z, Ma Y, Robson PM, Chung M, Bernheim A, Mani V, Calcagno C, Li K, Li S, Shan H, Lv J, Zhao T, Xia J, Long Q, Steinberger S, Jacobi A, Deyer T, Luksza M, Liu F, Little BP, Fayad ZA, Yang Y (2020) Artificial intelligence-enabled rapid diagnosis of patients with COVID-19. Nat Med 26(8):1224–1228
doi: 10.1038/s41591-020-0931-3
Müller H, Mayrhofer M, Van Veen E, Holzinger A (2021) The ten commandments of ethical medical AI. IEEE Comput 54(7):119–123
doi: 10.1109/MC.2021.3074263
Neidel T, Heller AR (2018) Triage beim Massenanfall von Verletzten (MANV). Notfallmedizin Up2date 13(2):135–149
doi: 10.1055/a-0607-1895
Neidel T, Salvador N, Heller AR (2017) Impact of systolic blood pressure limits on the diagnostic value of triage algorithms. Scand J Trauma Resusc Emerg Med 25(1):118
doi: 10.1186/s13049-017-0461-2
Pin M, Künstler C, Dodt C, Jerusalem K (2020) Behandlung Covid-19 Verdachtsfälle in der Notaufnahme, DGINA Notfallcampus V1.03, 2020, modified version according to K. Weber, Klinikum Kassel: COVID-19 Abklärungsalgorithmus Erwachsene (according to UCSF COVID-19 ID Clinical Working Group) and Zhang et al.: Therapeutic and triage strategies for 2019 novel coronavirus disease in fever clinics. Lanc Resp Med 8(3):e11–e12
Ryan L, Lam C, Mataraso S, Allen A, Green-Saxena A, Pellegrini E, Hoffman J, Barton C, McCoy A, Das R (2020) Mortality prediction model for the triage of COVID-19, pneumonia, and mechanically ventilated ICU patients: a retrospective study. Ann Med Surg 59:207–216
doi: 10.1016/j.amsu.2020.09.044
Tjoa E, Guan C (2021) A survey on explainable artificial intelligence (XAI): toward medical XAI. IEEE Trans Neural Netw Learn Syst 32(11):4793–4813
doi: 10.1109/TNNLS.2020.3027314
van Berkel N, Skov MB, Kjeldskov J (2021) Human-AI interaction: intermittent, continuous, and proactive. Interactions 28(6):67–71
doi: 10.1145/3486941
Wang D, Hu B, Hu C, Zhu F, Liu X, Zhang J, Wang B, Xiang H, Cheng Z, Xiong Y, Zhao Y, Li Y, Wang X, Peng Z (2020) Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus-infected pneumonia in Wuhan, China. JAMA 323(11):1061–1069
doi: 10.1001/jama.2020.1585
Wood RM, Pratt AC, Kenward C, McWilliams CJ, Booton RD, Thomas MJ, Bourdeaux CP, Vasilakis C (2021) The value of triage during periods of intense COVID-19 demand: Simulation modeling study. Med Decis Making 41(4):393–407
doi: 10.1177/0272989X21994035
World Health Organization (2021) Ethics and governance of artificial intelligence for health: WHO guidance. WHO, Geneva
Wu G, Yang P, Xie Y, Woodruff HC, Rao X, Guiot J, Frix AN, Louis R, Moutschen M, Li J, Li J, Yan C, Du D, Zhao S, Ding Y, Liu B, Sun W, Albarello F, D'Abramo A, Schininà V, Lambin P (2020) Development of a clinical decision support system for severity risk prediction and triage of COVID-19 patients at hospital admission: an international multicentre study. Eur Respir J 56(2):2001104
Wynants L, van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, Bonten MMJ, Dahly DL, Damen JAA, Debray TPA, de Jong VMT, de Vos M, Dhiman P, Haller MC, Harhay MO, Henckaerts L, Heus P, Kammer M, Kreuzberger N, Lohmann A, Luijken K, Ma J, Martin GP, McLernon DJ, Andaur CL, Reitsma JB, Sergeant JC, Shi C, Skoetz N, Smits LJM, Snell KIE, Sperrin M, Spijker R, Steyerberg EW, Takada T, Tzoulaki I, van Kuijk SMJ, van Bussel B, van Royen FS, Verbakel JY, Wallisch C, Wilkinson J, Wolff R, Hooft L, Moons KGM, van Smeden M (2020) Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ 369:1–16
Xie X, Zhong Z, Zhao W, Zheng C, Wang F, Liu J (2020) Chest CT for typical Coronavirus Disease 2019 (COVID-19) pneumonia: relationship to negative RT-PCR testing. Radiology 296(2):41.45
doi: 10.1148/radiol.2020200343
Zhang K, Liu X, Shen J, Li Z, Sang Y, Wu X, Zha Y, Liang W, Wang C, Wang K, Ye L, Gao M, Zhou Z, Li L, Wang J, Yang Z, Cai H, Xu J, Yang L, Cai W, Xu W, Wu S, Zhang W, Jiang S, Zheng L, Zhang X, Wang L, Lu L, Li J, Yin H, Wang W, Li O, Zhang C, Liang L, Wu T, Deng R, Wei K, Zhou Y, Chen T, Lau JYN, Fok M, He J, Lin T, Li W, Wang G (2020) Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell 181(6):1423–1433
doi: 10.1016/j.cell.2020.04.045