Picking the Right Tool for the Job: A Reliability Study of 4 Assessment Tools for Central Venous Catheter Insertion.
Journal
Journal of graduate medical education
ISSN: 1949-8357
Titre abrégé: J Grad Med Educ
Pays: United States
ID NLM: 101521733
Informations de publication
Date de publication:
Aug 2019
Aug 2019
Historique:
received:
11
02
2019
revised:
04
05
2019
accepted:
08
05
2019
entrez:
24
8
2019
pubmed:
24
8
2019
medline:
29
1
2020
Statut:
ppublish
Résumé
Determining procedural competence requires psychometrically sound assessment tools. A variety of instruments are available to determine procedural performance for central venous catheter (CVC) insertion, but it is not clear which ones should be used in the context of competency-based medical education. We compared several commonly used instruments to determine which should be preferentially used to assess competence in CVC insertion. Junior residents completing their first intensive care unit rotation between July 31, 2006, and March 9, 2007, were video-recorded performing CVC insertion on task trainer mannequins. Between June 1, 2016, and September 30, 2016, 3 experienced raters judged procedural competence on the historical video recordings of resident performance using 4 separate tools, including an itemized checklist, Objective Structured Assessment of Technical Skills (OSATS), a critical error assessment tool, and the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE). Generalizability theory (G-theory) was used to compare the performance characteristics among the tools. A decision study predicted the optimal testing environment using the tools. At the time of the original recording, 127 residents rotated through intensive care units at the University of Calgary, Alberta, Canada. Seventy-seven of them (61%) met inclusion criteria, and 55 of those residents (71%) agreed to participate. Results from the generalizability study (G-study) demonstrated that scores from O-SCORE and OSATS were the most dependable. Dependability could be maintained for O-SCORE and OSATS with 2 raters. Our results suggest that global rating scales, such as the OSATS or the O-SCORE tools, should be preferentially utilized for assessment of competence in CVC insertion.
Sections du résumé
BACKGROUND
BACKGROUND
Determining procedural competence requires psychometrically sound assessment tools. A variety of instruments are available to determine procedural performance for central venous catheter (CVC) insertion, but it is not clear which ones should be used in the context of competency-based medical education.
OBJECTIVE
OBJECTIVE
We compared several commonly used instruments to determine which should be preferentially used to assess competence in CVC insertion.
METHODS
METHODS
Junior residents completing their first intensive care unit rotation between July 31, 2006, and March 9, 2007, were video-recorded performing CVC insertion on task trainer mannequins. Between June 1, 2016, and September 30, 2016, 3 experienced raters judged procedural competence on the historical video recordings of resident performance using 4 separate tools, including an itemized checklist, Objective Structured Assessment of Technical Skills (OSATS), a critical error assessment tool, and the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE). Generalizability theory (G-theory) was used to compare the performance characteristics among the tools. A decision study predicted the optimal testing environment using the tools.
RESULTS
RESULTS
At the time of the original recording, 127 residents rotated through intensive care units at the University of Calgary, Alberta, Canada. Seventy-seven of them (61%) met inclusion criteria, and 55 of those residents (71%) agreed to participate. Results from the generalizability study (G-study) demonstrated that scores from O-SCORE and OSATS were the most dependable. Dependability could be maintained for O-SCORE and OSATS with 2 raters.
CONCLUSIONS
CONCLUSIONS
Our results suggest that global rating scales, such as the OSATS or the O-SCORE tools, should be preferentially utilized for assessment of competence in CVC insertion.
Identifiants
pubmed: 31440337
doi: 10.4300/JGME-D-19-00107.1
pii: Customer: JGME-D_19-00107R1
pmc: PMC6699549
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
422-429Déclaration de conflit d'intérêts
Conflict of interest: The authors declare they have no competing interests.
Références
Acad Med. 1999 Oct;74(10):1129-34
pubmed: 10536636
BMJ. 2003 Mar 8;326(7388):546-8
pubmed: 12623920
Med Educ. 2003 May;37(5):464-9
pubmed: 12709190
J Contin Educ Health Prof. 2003 Winter;23(1):4-12
pubmed: 12739254
J Manipulative Physiol Ther. 2006 Jul-Aug;29(6):463-7
pubmed: 16904492
Ann Emerg Med. 2006 Nov;48(5):540-7
pubmed: 17052555
Int J Lab Hematol. 2007 Aug;29(4):261-78
pubmed: 17617077
Med Educ. 2008 Jun;42(6):580-8
pubmed: 18482089
J Man Manip Ther. 2009;17(3):163-70
pubmed: 20046623
Med Educ. 2010 Jan;44(1):109-17
pubmed: 20078762
Med Teach. 2011;33(3):206-14
pubmed: 21345060
Adv Health Sci Educ Theory Pract. 2012 Oct;17(4):457-70
pubmed: 21877217
JAMA. 2011 Sep 7;306(9):978-88
pubmed: 21900138
Anesthesiology. 2012 Mar;116(3):539-73
pubmed: 22307320
Risk Manag Healthc Policy. 2011;4:27-39
pubmed: 22312225
Acad Med. 2012 Oct;87(10):1401-7
pubmed: 22914526
Med Teach. 2012;34(11):960-92
pubmed: 23140303
Adv Health Sci Educ Theory Pract. 1996 Jan;1(3):209-13
pubmed: 24179020
Acta Anaesthesiol Scand. 2014 May;58(5):508-24
pubmed: 24593804
Med Educ. 2015 Feb;49(2):161-73
pubmed: 25626747
Adv Health Sci Educ Theory Pract. 2015 Dec;20(5):1149-75
pubmed: 25702196
Med Educ. 2015 Oct;49(10):1004-15
pubmed: 26383072
Anaesthesia. 2016 Dec;71(12):1498-1499
pubmed: 27870183
Br J Surg. 1997 Feb;84(2):273-8
pubmed: 9052454
Am J Surg. 1997 Mar;173(3):226-30
pubmed: 9124632