Colonoscopy Direct Observation of Procedural Skills Assessment Tool for Evaluating Competency Development During Training.
Journal
The American journal of gastroenterology
ISSN: 1572-0241
Titre abrégé: Am J Gastroenterol
Pays: United States
ID NLM: 0421030
Informations de publication
Date de publication:
02 2020
02 2020
Historique:
pubmed:
19
11
2019
medline:
29
4
2020
entrez:
19
11
2019
Statut:
ppublish
Résumé
Formative colonoscopy direct observation of procedural skills (DOPS) assessments were updated in 2016 and incorporated into UK training but lack validity evidence. We aimed to appraise the validity of DOPS assessments, benchmark performance, and evaluate competency development during training in diagnostic colonoscopy. This prospective national study identified colonoscopy DOPS submitted over an 18-month period to the UK training e-portfolio. Generalizability analyses were conducted to evaluate internal structure validity and reliability. Benchmarking was performed using receiver operator characteristic analyses. Learning curves for DOPS items and domains were studied, and multivariable analyses were performed to identify predictors of DOPS competency. Across 279 training units, 10,749 DOPS submitted for 1,199 trainees were analyzed. The acceptable reliability threshold (G > 0.70) was achieved with 3 assessors performing 2 DOPS each. DOPS competency rates correlated with the unassisted caecal intubation rate (rho 0.404, P < 0.001). Demonstrating competency in 90% of assessed items provided optimal sensitivity (90.2%) and specificity (87.2%) for benchmarking overall DOPS competence. This threshold was attained in the following order: "preprocedure" (50-99 procedures), "endoscopic nontechnical skills" and "postprocedure" (150-199), "management" (200-249), and "procedure" (250-299) domain. At item level, competency in "proactive problem solving" (rho 0.787) and "loop management" (rho 0.780) correlated strongest with the overall DOPS rating (P < 0.001) and was the last to develop. Lifetime procedure count, DOPS count, trainer specialty, easier case difficulty, and higher cecal intubation rate were significant multivariable predictors of DOPS competence. This study establishes milestones for competency acquisition during colonoscopy training and provides novel validity and reliability evidence to support colonoscopy DOPS as a competency assessment tool.
Identifiants
pubmed: 31738285
doi: 10.14309/ajg.0000000000000426
pii: 00000434-202002000-00017
doi:
Types de publication
Journal Article
Observational Study
Langues
eng
Sous-ensembles de citation
IM
Pagination
234-243Commentaires et corrections
Type : CommentIn
Références
Nasca TJ, Philibert I, Brigham T, et al. The next GME accreditation system—Rationale and benefits. N Engl J Med 2012;366(11):1051–6.
Ekkelenkamp VE, Koch AD, de Man RA, et al. Training and competence assessment in GI endoscopy: A systematic review. Gut 2016;65(4):607–15.
Siau K, Hawkes ND, Dunckley P. Training in endoscopy. Curr Treat Options Gastroenterol 2018;16(3):345–61.
Walsh CM. In-training Gastrointestinal Endoscopy Competency Assessment Tools: Types of tools, validation and impact. Best Pract Res Clin Gastroenterol 2016;30(3):357–74.
Siau K, Green JT, Hawkes ND, et al. Impact of the Joint Advisory Group on Gastrointestinal Endoscopy (JAG) on endoscopy services in the UK and beyond. Frontline Gastroenterol 2019;10(2):93–106.
Barton JR, Corbett S, van der Vleuten CP. The validity and reliability of a direct observation of procedural skills assessment tool: Assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 2012;75(3):591–7.
Siau K, Dunckley P, Valori R, et al. Changes in scoring of direct observation of procedural skills (DOPS) forms and the impact on competence assessment. Endoscopy 2018;50(8):770–8.
Joint Advisory Group on Gastrointestinal Endoscopy. Formative DOPS: Colonoscopy and flexible sigmoidoscopy. 2016 (https://www.thejag.org.uk/Downloads/DOPS%20forms%20(international%20and%20reference%20use%20only)/Formative%20DOPS_Colonoscopy%20and%20Flexible%20sigmoidoscopy.pdf). Accessed April 20, 2019.
Mehta T, Dowler K, McKaig BC, et al. Development and roll out of the JETS e-portfolio: A web based electronic portfolio for endoscopists. Frontline Gastroenterol 2010;2(1):35.
Joint Advisory Group on Gastrointestinal Endoscopy. JAG trainee certification process: Colonoscopy (provisional and full). 2017 (https://www.thejag.org.uk/Downloads/JETS%20-%20certification%20for%20trainees/Colonoscopy%20application%20criteria%20and%20process.pdf). Accessed April 20, 2019.
Siau K, Hodson J, Valori RM, et al. Performance indicators in colonoscopy after certification for independent practice: Outcomes and predictors of competence. Gastrointest Endosc 2019;89(3):482–92.e2.
Joint Advisory Group on Gastrointestinal Endoscopy. Summative DOPS: Colonoscopy and flexible sigmoidoscopy. 2016 (https://www.thejag.org.uk/Downloads/DOPS%20forms%20(international%20and%20reference%20use%20only)/Summative%20DOPS_Colonoscopy%20and%20Flexible%20sigmoidoscopy.pdf). Accessed April 20, 2019.
Siau K, Anderson JT, Valori R, et al. Certification of UK gastrointestinal endoscopists and variations between trainee specialties: Results from the JETS e-portfolio. Endosc Int Open 2019;7(4):E551–60.
Crossley J, Davies H, Humphris G, et al. Generalisability: A key to unlock professional assessment. Med Educ 2002;36(10):972–8.
Webb NM, Shavelson RJ, Haertel EH. 4 reliability coefficients and generalizability theory. In: Rao CR, Sinharay S (eds). Handbook of Statistics: Elsevier, 2006, pp 81–124.
Ruopp MD, Perkins NJ, Whitcomb BW, et al. Youden index and optimal cut-point estimated from observations affected by a lower limit of detection. Biom J 2008;50(3):419–30.
Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am Psychol 1995;50(9):741.
Cook DA, Hatala R. Validation of educational assessments: A primer for simulation and beyond. Adv Simul (Lond) 2016;1(1):31.
Leung FW. Assessment of trainees' performance in colonoscopy. Gastrointest Endosc 2018;87(1):270–1.
Fried GM, Marks JM, Mellinger JD, et al. ASGE's assessment of competency in endoscopy evaluation tools for colonoscopy and EGD. Gastrointest Endosc 2014;80(2):366–7.
Sedlack RE. The Mayo Colonoscopy Skills Assessment Tool: Validation of a unique instrument to assess colonoscopy skills in trainees. Gastrointest Endosc 2010;72(6):1125–33.
Sedlack RE, Coyle WJ. Assessment of competency in endoscopy: Establishing and validating generalizable competency benchmarks for colonoscopy. Gastrointest Endosc 2016;83(3):516–23.e1.
Walsh CM, Ling SC, Khanna N, et al. Gastrointestinal endoscopy competency assessment tool: Reliability and validity evidence. Gastrointest Endosc 2015;81(6):1417–24.e2.
Kumar NL, Kugener G, Perencevich ML, et al. The SAFE-T assessment tool: Derivation and validation of a web-based application for point-of-care evaluation of gastroenterology fellow performance in colonoscopy. Gastrointest Endosc 2018;87(1):262–9.
Siau K, Crossley J, Dunckley P, et al. Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: Nationwide evidence of validity and competency development during training. Surg Endosc 2019. [Epub ahead of print March 27, 2019].
Siau K, Hodson J, Ravindran S, et al. Variability in cecal intubation rate by calculation method: A call for standardization of key performance indicators in endoscopy. Gastrointest Endosc 2019;89:1026–1036.e2.
Ward ST, Mohammed MA, Walt R, et al. An analysis of the learning curve to achieve competency at colonoscopy using the JETS database. Gut 2014;63(11):1746.
Sedlack RE. Training to competency in colonoscopy: Assessing and defining competency standards. Gastrointest Endosc 2011;74(2):355–66.e2.
Cohen J, Cohen SA, Vora KC, et al. Multicenter, randomized, controlled trial of virtual-reality simulator training in acquisition of competency in colonoscopy. Gastrointest Endosc 2006;64(3):361–8.
Joint Advisory Group on Gastrointestinal Endoscopy. Formative DOPyS: Colonoscopy and flexible sigmoidoscopy. 2016 (https://www.thejag.org.uk/Downloads/DOPS%20Forms%20For%20International%20and%20reference%20use%20only/Formative%20DOPyS_Colonoscopy%20and%20Flexible%20sigmoidoscopy.pdf). Accessed November 21, 2017.
Lee TJW, Siau K, Esmaily S, et al. Development of a national automated endoscopy database: The United Kingdom National Endoscopy Database (NED). United European Gastroenterol J 2019;7:798–806.