Assessing the reliability of medical resource demand models in the context of COVID-19.
Epidemiology
Mathematical modeling
Medical device
Validation
Journal
BMC medical informatics and decision making
ISSN: 1472-6947
Titre abrégé: BMC Med Inform Decis Mak
Pays: England
ID NLM: 101088682
Informations de publication
Date de publication:
31 Oct 2024
31 Oct 2024
Historique:
received:
22
11
2023
accepted:
16
10
2024
medline:
1
11
2024
pubmed:
1
11
2024
entrez:
1
11
2024
Statut:
epublish
Résumé
Numerous medical resource demand models have been created as tools for governments or hospitals, aiming to predict the need for crucial resources like ventilators, hospital beds, personal protective equipment (PPE), and diagnostic kits during crises such as the COVID-19 pandemic. However, the reliability of these demand models remains uncertain. Demand models typically consist of two main components: hospital use epidemiological models that predict hospitalizations or daily admissions, and a demand calculator that translates the outputs of the epidemiological model into predictions for resource usage. We conducted separate analyses to evaluate each of these components. In the first analysis, we validated various hospital use epidemiological models using a recent validation framework designed for epidemiological models. This allowed us to quantify the accuracy of the models in predicting critical aspects such as the date and magnitude of local COVID-19 peaks, among other factors. In the second analysis, we evaluated a range of demand calculators for ventilators, medical gowns, and COVID-19 test kits. To achieve this, we decoupled these demand calculators from the underlying epidemiological models and provided ground truth data for their inputs. This approach enabled a direct comparison of the demand calculators, comparing them against each other and actual usage data when available. The code is available at https://doi.org/10.5281/zenodo.13712387 . Performance varied greatly across the epidemiological models, with greater variability in COVID-19 hospital use predictions than for COVID-19 deaths as analyzed previously. Some models did not have any peaks. Among those that did, the models under-estimated date of peak approximately as often as they over-estimated, but were more likely to under-estimate magnitude of peak, with typical relative errors around 50%. Regarding demand calculator predictions, there was significant variability, including five-fold differences in predictions for gown models. Validation against actual or surrogate usage data illustrated the potential value of demand models while demonstrating their limitations. The emerging field of demand modeling holds promise in averting medical resource shortages during future public health emergencies. However, achieving this potential necessitates focused efforts on standardization, transparency, and rigorous model validation before placing reliance on demand models in critical public health decision-making.
Sections du résumé
BACKGROUND
BACKGROUND
Numerous medical resource demand models have been created as tools for governments or hospitals, aiming to predict the need for crucial resources like ventilators, hospital beds, personal protective equipment (PPE), and diagnostic kits during crises such as the COVID-19 pandemic. However, the reliability of these demand models remains uncertain.
METHODS
METHODS
Demand models typically consist of two main components: hospital use epidemiological models that predict hospitalizations or daily admissions, and a demand calculator that translates the outputs of the epidemiological model into predictions for resource usage. We conducted separate analyses to evaluate each of these components. In the first analysis, we validated various hospital use epidemiological models using a recent validation framework designed for epidemiological models. This allowed us to quantify the accuracy of the models in predicting critical aspects such as the date and magnitude of local COVID-19 peaks, among other factors. In the second analysis, we evaluated a range of demand calculators for ventilators, medical gowns, and COVID-19 test kits. To achieve this, we decoupled these demand calculators from the underlying epidemiological models and provided ground truth data for their inputs. This approach enabled a direct comparison of the demand calculators, comparing them against each other and actual usage data when available. The code is available at https://doi.org/10.5281/zenodo.13712387 .
RESULTS
RESULTS
Performance varied greatly across the epidemiological models, with greater variability in COVID-19 hospital use predictions than for COVID-19 deaths as analyzed previously. Some models did not have any peaks. Among those that did, the models under-estimated date of peak approximately as often as they over-estimated, but were more likely to under-estimate magnitude of peak, with typical relative errors around 50%. Regarding demand calculator predictions, there was significant variability, including five-fold differences in predictions for gown models. Validation against actual or surrogate usage data illustrated the potential value of demand models while demonstrating their limitations.
CONCLUSIONS
CONCLUSIONS
The emerging field of demand modeling holds promise in averting medical resource shortages during future public health emergencies. However, achieving this potential necessitates focused efforts on standardization, transparency, and rigorous model validation before placing reliance on demand models in critical public health decision-making.
Identifiants
pubmed: 39482697
doi: 10.1186/s12911-024-02726-6
pii: 10.1186/s12911-024-02726-6
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
322Informations de copyright
© 2024. This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply.
Références
FDA. Resilient Supply Chain Program for Medical Devices. https://www.fda.gov/about-fda/cdrh-offices/resilient-supply-chain-program-medical-devices . Accessed Sep 2022.
Health Information and Quality Authority. Identification of demand models for estimating the quantities of personal protective equipment (PPE) required for optimal patient care in the context of COVID-19. https://www.hiqa.ie/sites/default/files/2021-09/PPE-modelling-report.pdf . Accessed Sep 2022.
WHO. The WHO COVID-19 Essential Supplies Forecasting Tool. https://www.who.int/publications/i/item/WHO-2019-nCoV-Tools-Essential_forecasting-2022.1 . Accessed Sep 2022.
U Penn . COVID-19 Hospital Impact Model for Epidemics (CHIME). https://penn-chime.phl.io/ . Accessed Sep 2022.
MEMA. MEMA COVID-19 Burn Rate Projection Planning Tool. https://mdem.maryland.gov/Documents/MEMA_COVID-19-Burn-Rate-Projection-Tool-Report.pdf . Accessed Sep 2022.
Locey KJ, Webb TA, Khan J, Antony AK, Hota B. An interactive tool to forecast US hospital needs in the coronavirus 2019 pandemic. JAMIA Open. 2020;3(4):506–12.
doi: 10.1093/jamiaopen/ooaa045
pubmed: 33619466
pmcid: 7543612
Wells CR, Fitzpatrick MC, Sah P, Shoukat A, Pandey A, El-Sayed AM, et al. Projecting the demand for ventilators at the peak of the COVID-19 outbreak in the USA. Lancet Infect Dis. 2020;20(10):1123–5.
doi: 10.1016/S1473-3099(20)30315-7
pubmed: 32325039
pmcid: 7172723
Moghadas SM, Shoukat A, Fitzpatrick MC, Wells CR, Sah P, Pandey A, et al. Projecting hospital utilization during the COVID-19 outbreaks in the United States. Proc Natl Acad Sci. 2020;117(16):9122–6.
doi: 10.1073/pnas.2004064117
pubmed: 32245814
pmcid: 7183199
IHME. Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months. medRxiv. 2020. https://doi.org/10.1101/2020.03.27.20043752 .
JHU. Worksheet for calcuating national PPE need for COVID-19. https://www.centerforhealthsecurity.org/resources/COVID-19/PPE/PPE-assumptions . Accessed Sep 2022.
CDC. Personal Protective Equipment (PPE) Burn Rate Calculator. https://www.cdc.gov/coronavirus/2019-ncov/hcp/ppe-strategy/burn-calculator.html . Accessed Sep 2022.
Raja S, Patolia HH, Baffoe-Bonnie AW. Calculating an institutional personal protective equipment (PPE) burn rate to project future usage patterns during the 2020 COVID-19 pandemic. Infect Control Hosp Epidemiol. 2020;41(12):1474–5.
doi: 10.1017/ice.2020.190
pubmed: 32362291
COVID Staffing. Daily PPE Calculator. https://www.covidstaffing.org/modules/daily-ppe-calculator/ . Accessed Sep 2022.
Rickard KNZ, McKinley KW. Comparing PPE Conservation Strategies. https://emscimprovement.center/education-and-resources/toolkits/pediatric-disaster-preparedness-toolbox/covid-19-coronavirus/domainspreparednesscoronavirus-covid-19ppe/ . Accessed Sep 2022.
CDC. US Centers for Disease Control and Prevention (CDC) - Ebola PPE Calculator. https://www.cdc.gov/vhf/ebola/healthcare-us/ppe/calculator.html . Accessed Sep 2022.
ASPR. ASPR TRACIE. (2018). Hospital Personal Protective Equipment Planning Tool. https://files.asprtracie.hhs.gov/documents/aspr-tracie-hospital-ppe-planning-tool.pdf . Accessed Sep 2022.
Walker PG, Whittaker C, Watson OJ, Baguelin M, Winskill P, Hamlet A, et al. The impact of COVID-19 and strategies for mitigation and suppression in low-and middle-income countries. Science. 2020;369(6502):413–22.
doi: 10.1126/science.abc0035
pubmed: 32532802
pmcid: 7292504
McCabe R, Kont MD, Schmit N, Whittaker C, Løchen A, Baguelin M, et al. Modelling intensive care unit capacity under different epidemiological scenarios of the COVID-19 pandemic in three Western European countries. Int J Epidemiol. 2021;50(3):753–67.
doi: 10.1093/ije/dyab034
pubmed: 33837401
pmcid: 8083295
Pilz K, Shakfeh N, Perry M, Yadav P. Data Science to Forecast Demand for Supplies at Health Clinics. https://www.cgdev.org/blog/data-science-forecast-demand-supplies-health-clinics . Accessed Aug 2024.
GCCD. Top forecasting methods for effective healthcare financial planning in 2024. https://www.ccdcare.com/resource-center/forecasting-in-healthcare . Accessed August 2024.
Niaz M, Nwagwu U, et al. Managing Healthcare Product Demand Effectively in The Post-Covid-19 Environment: Navigating Demand Variability and Forecasting Complexities. Am J Econ Manag Bus (AJEMB). 2023;2(8):316–30.
doi: 10.58631/ajemb.v2i8.55
Dautel KA, Agyingi E, Pathmanathan P. Validation framework for epidemiological models with application to COVID-19 models. PLoS Comput Biol. 2023;19(3):e1010968.
doi: 10.1371/journal.pcbi.1010968
pubmed: 36989251
pmcid: 10057797
HealthData gov. COVID-19 Reported Patient Impact and Hospital Capacity by State Timeseries. https://healthdata.gov/Hospital/COVID-19-Reported-Patient-Impact-and-Hospital-Capa/g62h-syeh/data . Accessed Aug 2022.
CDC. Previous COVID-19 Forecasts: Deaths - 2021. https://www.cdc.gov/coronavirus/2019-ncov/science/forecasting/forecasting-us-previous-2021.html . Accessed Sep 2022.
Bleichrodt A, Luo R, Kirpich A, Chowell G. Evaluating the forecasting performance of ensemble subepidemic frameworks and other time series models for the 2022–2023 mpox epidemic. R Soc Open Sci. 2024;11(7). https://doi.org/10.1098/rsos.240248 .
Columbia University. Columbia University COVID-19 Projection. https://github.com/shaman-lab/COVID-19Projection . Accessed Sep 2022.
Covid19Sim. Covid19Sim predictions. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/Covid19Sim-Simulator . Accessed Sep 2022.
Georgia Tech. GT-DeepCOVID predictions. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/GT-DeepCOVID . Accessed Sep 2022.
IHME. IHME COVID-19 estimate downloads. https://www.healthdata.org/node/8787 . Accessed Sep 2021.
JHU. JHU APL Bucky predictions. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/JHUAPL-Bucky . Accessed Sep 2022.
JHU. JHU IDD model predictions. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/JHU_IDD-CovidSP . Accessed Sep 2022.
Karlen. Karlen model predictions. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/Karlen-pypm . Accessed Sep 2022.
UCLA. UCLA model predictions. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/UCLA-SuEIR . Accessed Sep 2022.
USC. USC model predictions. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/USC-SI_kJalpha . Accessed Sep 2022.
The COVID Tracking Project. The COVID Tracking Project. https://covidtracking.com/data/download . Accessed Sep 2021.
CDC. COVID Data Tracker Trends in Number of COVID-19 Cases and Deaths in the US Reported to CDC, by State/Territory. https://covid.cdc.gov/covid-data-tracker/#trends_weeklycases_select_00 . Accessed Sep 2022.
New York government. Daily Hospitalization Summary. https://coronavirus.health.ny.gov/daily-hospitalization-summary . Accessed Sep 2022.
CDC. COVID Data Tracker Prevalent Hospitalizations of Patients with Confirmed COVID-19, United States. https://covid.cdc.gov/covid-data-tracker/#hospitalizations . Accessed Sep 2022.
CDC. COVID Data Tracker New Hospital Admissions of Patients with Confirmed COVID-19, United States. https://covid.cdc.gov/covid-data-tracker/#new-hospital-admissions . Accessed Sep 2022.
Our World in Data. Number of COVID-19 Patients in intensive care unit (ICU). https://ourworldindata.org/grapher/current-covid-patients-icu?time=2020-07-11..latest . Accessed Sep 2022.
AdvaMed. COVID-19 Diagnostic Supply Registry. https://www.advamed.org/wp-content/uploads/2022/10/AdvaMed-COVID-Testing-Supply-Registry-092422.pdf . Accessed Sep 2022.
American Hospital Association. Fast Facts on U.S. Hospitals, 2022. https://www.aha.org/statistics/fast-facts-us-hospitals . Accessed Sep 2022.
US CDC. FluSight 2023-2024. https://github.com/cdcepi/FluSight-forecast-hub . Accessed Sep 2024.
Get US PPE. Get Us PPE. https://getusppe.org/ . Accessed Sep 2022.