Development of an Endotracheal Intubation Formative Assessment Tool.
Regression analysis
endotracheal intubation
training rubric
video analysis
Journal
The journal of education in perioperative medicine : JEPM
ISSN: 2333-0406
Titre abrégé: J Educ Perioper Med
Pays: United States
ID NLM: 101122652
Informations de publication
Date de publication:
Historique:
entrez:
21
5
2020
pubmed:
21
5
2020
medline:
21
5
2020
Statut:
epublish
Résumé
Valid methods for providing detailed formative feedback on direct laryngoscopy and endotracheal intubation (ETI) performance do not exist. We are developing an observation-based assessment tool for measuring performance and providing feedback during ETI. Based on the literature and interviews of experts, we proposed an initial ETI metric with 22 items. Six anesthesiology experts used it to assess the quality of ETI performance in videotaped intubations. Following metric revisions, 2 expert groups assessed 2 collections of videos (27 total) using the revised metric. Two reference standards for comparison with metric scores were created with a third and fourth group of experts; (1) an average global rating (1-100) of each ETI performance and (2) average rank-ordered performance from best to worst. Rater agreement and correlations between the 2 methods were calculated. Regression analysis determined items that optimally discriminated quality. When calculating a score based on all clinically important terms, multiple weightings were evaluated. Metric items had high average rater agreement (80%) with intraclass correlation coefficients averaging 0.83. Correlations of the reference rank and score were high for both video collections (-0.96, High observer agreement and strong correlations between metric and rank data support the validity of using this metric to assess ETI performance. Different weighting models yielded scores that correlated strongly with the ratings and ranks from global assessment. When using the metric to predict competency, a 3-item regression model is most accurate in predicting a global score.
Sections du résumé
BACKGROUND
BACKGROUND
Valid methods for providing detailed formative feedback on direct laryngoscopy and endotracheal intubation (ETI) performance do not exist. We are developing an observation-based assessment tool for measuring performance and providing feedback during ETI.
METHODS
METHODS
Based on the literature and interviews of experts, we proposed an initial ETI metric with 22 items. Six anesthesiology experts used it to assess the quality of ETI performance in videotaped intubations. Following metric revisions, 2 expert groups assessed 2 collections of videos (27 total) using the revised metric. Two reference standards for comparison with metric scores were created with a third and fourth group of experts; (1) an average global rating (1-100) of each ETI performance and (2) average rank-ordered performance from best to worst. Rater agreement and correlations between the 2 methods were calculated. Regression analysis determined items that optimally discriminated quality. When calculating a score based on all clinically important terms, multiple weightings were evaluated.
RESULTS
RESULTS
Metric items had high average rater agreement (80%) with intraclass correlation coefficients averaging 0.83. Correlations of the reference rank and score were high for both video collections (-0.96,
CONCLUSIONS
CONCLUSIONS
High observer agreement and strong correlations between metric and rank data support the validity of using this metric to assess ETI performance. Different weighting models yielded scores that correlated strongly with the ratings and ranks from global assessment. When using the metric to predict competency, a 3-item regression model is most accurate in predicting a global score.
Types de publication
Journal Article
Langues
eng
Pagination
E635Subventions
Organisme : NIBIB NIH HHS
ID : R01 EB005807
Pays : United States
Organisme : NHLBI NIH HHS
ID : R01 HL119248
Pays : United States
Informations de copyright
© 2020 Society for Education in Anesthesia.
Déclaration de conflit d'intérêts
Funding: National Institutes of Health 5R01HL119248-04
Références
J Surg Educ. 2008 Mar-Apr;65(2):101-8
pubmed: 18439529
J Pediatr. 2017 Sep;188:252-257.e6
pubmed: 28456389
Anesthesiology. 2011 Jan;114(1):42-8
pubmed: 21150574
Scand J Pain. 2019 Jul 26;19(3):523-532
pubmed: 30901318
Respir Care. 2014 Jun;59(6):1006-19; discussion 1019-21
pubmed: 24891204
Air Med J. 2016 May-Jun;35(3):138-42
pubmed: 27255875
Otolaryngol Head Neck Surg. 1994 Oct;111(4):453-9
pubmed: 7936678
Br J Anaesth. 1989 Jan;62(1):6-12
pubmed: 2917111
Anesth Analg. 1992 Mar;74(3):411-4
pubmed: 1539823
J Clin Epidemiol. 2019 Aug;112:20-27
pubmed: 30930247
Br J Anaesth. 2015 Aug;115(2):302-7
pubmed: 26170352
Anesth Analg. 1996 Mar;82(3):456-61
pubmed: 8623942
Anesth Analg. 1996 Mar;82(3):462-8
pubmed: 8623943