Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.
Competence
DOPS
Esophagogastroduodenoscopy
Formative assessment
Gastroscopy
OGD
Journal
Surgical endoscopy
ISSN: 1432-2218
Titre abrégé: Surg Endosc
Pays: Germany
ID NLM: 8806653
Informations de publication
Date de publication:
01 2020
01 2020
Historique:
received:
23
11
2018
accepted:
06
03
2019
pubmed:
27
3
2019
medline:
21
10
2020
entrez:
27
3
2019
Statut:
ppublish
Résumé
Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.
Sections du résumé
BACKGROUND
Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.
METHODS
This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence.
RESULTS
In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence.
CONCLUSION
This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.
Identifiants
pubmed: 30911922
doi: 10.1007/s00464-019-06737-7
pii: 10.1007/s00464-019-06737-7
pmc: PMC6946748
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
105-114Commentaires et corrections
Type : ErratumIn
Références
Surg Endosc. 2010 Aug;24(8):1834-41
pubmed: 20112113
Int J Nurs Stud. 2014 Mar;51(3):511-21
pubmed: 24183474
Best Pract Res Clin Gastroenterol. 2016 Jun;30(3):357-74
pubmed: 27345645
Endoscopy. 2018 Aug;50(8):770-778
pubmed: 29614526
Gut. 2016 Apr;65(4):607-15
pubmed: 25636697
Lancet. 2015 Jul 11;386(9989):137-44
pubmed: 25956718
Gastrointest Endosc. 2012 Mar;75(3):591-7
pubmed: 22227035
Gastrointest Endosc. 2014 Jan;79(1):1-7
pubmed: 24239255
J Gastrointestin Liver Dis. 2017 Dec;26(4):339-344
pubmed: 29253046
Gastrointest Endosc. 2017 Oct;86(4):751-754
pubmed: 28917356
Curr Treat Options Gastroenterol. 2018 Sep;16(3):345-361
pubmed: 30019105
Frontline Gastroenterol. 2011 Jan;2(1):35-42
pubmed: 28839580
Ann Intern Med. 1993 Jan 1;118(1):40-4
pubmed: 8416157
Gut. 2014 Nov;63(11):1746-54
pubmed: 24470280
Gut. 2017 Jun;66(6):1022-1033
pubmed: 26976733
Gastrointest Endosc. 2018 Jan;87(1):270-271
pubmed: 29241855
Frontline Gastroenterol. 2019 Apr;10(2):93-106
pubmed: 31210174
Clin Gastroenterol Hepatol. 2015 Mar;13(3):480-487.e2
pubmed: 25117772
Endoscopy. 2017 Jun;49(6):524-528
pubmed: 28399610
Adv Simul (Lond). 2016 Dec 7;1:31
pubmed: 29450000
Med Educ. 2011 Jun;45(6):560-9
pubmed: 21501218
Endosc Int Open. 2019 Apr;7(4):E551-E560
pubmed: 30957005