Evaluating the influence of data collector training for predictive risk of death models: an observational study.
critical care
healthcare quality improvement
quality measurement
Journal
BMJ quality & safety
ISSN: 2044-5423
Titre abrégé: BMJ Qual Saf
Pays: England
ID NLM: 101546984
Informations de publication
Date de publication:
03 2021
03 2021
Historique:
received:
30
01
2020
revised:
16
03
2020
accepted:
17
03
2020
pubmed:
2
4
2020
medline:
16
10
2021
entrez:
2
4
2020
Statut:
ppublish
Résumé
Severity-of-illness scoring systems are widely used for quality assurance and research. Although validated by trained data collectors, there is little data on the accuracy of real-world data collection practices. To evaluate the influence of formal data collection training on the accuracy of scoring system data in intensive care units (ICUs). Quality assurance audit conducted using survey methodology principles. Between June and December 2018, an electronic document with details of three fictitious ICU patients was emailed to staff from 19 Australian ICUs who voluntarily submitted data on a web-based data entry form. Their entries were used to generate severity-of-illness scores and risks of death (RoDs) for four scoring systems. The primary outcome was the variation of severity-of-illness scores and RoDs from a reference standard. 50/83 staff (60.3%) submitted data. Using Bayesian multilevel analysis, severity-of-illness scores and RoDs were found to be significantly higher for untrained staff. The mean (95% high-density interval) overestimation in RoD due to training effect for patients 1, 2 and 3, respectively, were 0.24 (0.16, 0.31), 0.19 (0.09, 0.29) and 0.24 (0.1, 0.38) respectively (Bayesian factor In a fictitious patient dataset, data collection staff without formal training significantly overestimated the severity-of-illness scores and RoDs compared with trained staff. Both groups exhibited wide variability. Strategies to improve practice may include providing adequate training for all data collection staff, refresher training for previously trained staff and auditing the raw data submitted by individual ICUs. The results of this simulated study need revalidation on real patients.
Sections du résumé
BACKGROUND
Severity-of-illness scoring systems are widely used for quality assurance and research. Although validated by trained data collectors, there is little data on the accuracy of real-world data collection practices.
OBJECTIVE
To evaluate the influence of formal data collection training on the accuracy of scoring system data in intensive care units (ICUs).
STUDY DESIGN AND METHODS
Quality assurance audit conducted using survey methodology principles. Between June and December 2018, an electronic document with details of three fictitious ICU patients was emailed to staff from 19 Australian ICUs who voluntarily submitted data on a web-based data entry form. Their entries were used to generate severity-of-illness scores and risks of death (RoDs) for four scoring systems. The primary outcome was the variation of severity-of-illness scores and RoDs from a reference standard.
RESULTS
50/83 staff (60.3%) submitted data. Using Bayesian multilevel analysis, severity-of-illness scores and RoDs were found to be significantly higher for untrained staff. The mean (95% high-density interval) overestimation in RoD due to training effect for patients 1, 2 and 3, respectively, were 0.24 (0.16, 0.31), 0.19 (0.09, 0.29) and 0.24 (0.1, 0.38) respectively (Bayesian factor
INTERPRETATION
In a fictitious patient dataset, data collection staff without formal training significantly overestimated the severity-of-illness scores and RoDs compared with trained staff. Both groups exhibited wide variability. Strategies to improve practice may include providing adequate training for all data collection staff, refresher training for previously trained staff and auditing the raw data submitted by individual ICUs. The results of this simulated study need revalidation on real patients.
Identifiants
pubmed: 32229628
pii: bmjqs-2020-010965
doi: 10.1136/bmjqs-2020-010965
doi:
Types de publication
Journal Article
Observational Study
Langues
eng
Pagination
202-207Informations de copyright
© Author(s) (or their employer(s)) 2021. No commercial re-use. See rights and permissions. Published by BMJ.
Déclaration de conflit d'intérêts
Competing interests: None declared.