Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery.
Assessment
Simulation
Surgery
Surgical education and training
Journal
Surgical endoscopy
ISSN: 1432-2218
Titre abrégé: Surg Endosc
Pays: Germany
ID NLM: 8806653
Informations de publication
Date de publication:
17 Jul 2024
17 Jul 2024
Historique:
received:
25
03
2024
accepted:
30
06
2024
medline:
18
7
2024
pubmed:
18
7
2024
entrez:
17
7
2024
Statut:
aheadofprint
Résumé
Simulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation. Eight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of trainers. Trainee performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure-Based Assessment (PBA) tool. Validity evidence was analyzed in accordance with Messick's validity framework. Both 'junior' (ST2-ST4) and 'senior' trainees (ST 5-ST8) were included to allow for comparative analysis. Thirteen trainees were assessed by ten assessors across eight stations. Inter-station reliability was high (α = 0.81), and inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004), while overall mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall's tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all 'senior' trainees passing and 4/6 of junior trainees failing the assessment. This study reports validity evidence for a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery.
Sections du résumé
BACKGROUND
BACKGROUND
Simulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation.
METHODS
METHODS
Eight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of trainers. Trainee performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure-Based Assessment (PBA) tool. Validity evidence was analyzed in accordance with Messick's validity framework. Both 'junior' (ST2-ST4) and 'senior' trainees (ST 5-ST8) were included to allow for comparative analysis.
RESULTS
RESULTS
Thirteen trainees were assessed by ten assessors across eight stations. Inter-station reliability was high (α = 0.81), and inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004), while overall mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall's tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all 'senior' trainees passing and 4/6 of junior trainees failing the assessment.
CONCLUSION
CONCLUSIONS
This study reports validity evidence for a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery.
Identifiants
pubmed: 39020120
doi: 10.1007/s00464-024-11024-1
pii: 10.1007/s00464-024-11024-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2024. The Author(s).
Références
Hurreiz H (2019) The evolution of surgical training in the UK. Adv Med Educ Pract 10:163–168
doi: 10.2147/AMEP.S189298
pubmed: 30992688
pmcid: 6445234
Brown E, Choi J, Sairi T (2020) Resident involvement in plastic surgery: divergence of patient expectations and experiences with surgeon’s attitudes and actions. J Surg Educ 77:291–299
doi: 10.1016/j.jsurg.2019.10.008
pubmed: 31732224
Aggarwal R, Darzi A (2006) Training in the operating theatre: is it safe? Thorax 61:278–279
doi: 10.1136/thx.2005.051789
pubmed: 16565265
pmcid: 2104598
Elfenbein DM (2016) Confidence crisis among general surgery residents: a systematic review and qualitative discourse analysis. JAMA Surg 151:1166–1175
doi: 10.1001/jamasurg.2016.2792
pubmed: 27603429
pmcid: 6941584
Poudel S, Hirano S, Kurashima Y, Stefanidis D, Akiyama H, Eguchi S, Fukui T, Hagiwara M, Hashimoto D, Hida K, Izaki T, Iwase H, Kawamoto S, Otomo Y, Nagai E, Saito M, Takami H, Takeda Y, Toi M, Yamaue H, Yoshida M, Yoshida S, Kodera Y et al (2020) Are graduating residents sufficiently competent? Results of a national gap analysis survey of program directors and graduating residents in Japan. Surg Today 50:995–1001
doi: 10.1007/s00595-020-01981-0
pubmed: 32125504
Vu JV, George BC, Clark M, Rivard SJ, Regenbogen SE, Kwakye G (2021) Readiness of graduating general surgery residents to perform colorectal procedures. J Surg Educ. https://doi.org/10.1016/j.jsurg.2020.12.015
doi: 10.1016/j.jsurg.2020.12.015
pubmed: 34266789
pmcid: 8217079
Harris KA, Nousiainen MT, Reznick R (2020) Competency-based resident education: the Canadian perspective. Surgery 167:681–684
doi: 10.1016/j.surg.2019.06.033
pubmed: 31431292
(2014) The general surgery milestone project. J Grad Med Educ 6:320–328
James HK, Gregory RJH (2021) The dawn of a new competency-based training era. Bone Joint Open 2:181–190
doi: 10.1302/2633-1462.23.BJO-2020-0155.R1
pubmed: 33739137
pmcid: 8009901
Schijven MP, Bemelman WA (2011) Problems and pitfalls in modern competency-based laparoscopic training. Surg Endosc 25:2159–2163
doi: 10.1007/s00464-010-1515-y
pubmed: 21298547
pmcid: 3116119
Cook T, Lund J (2021) General surgery curriculum—the Intercollegiate Surgical Curriculum Programme. https://www.iscp.ac.uk/media/1103/general-surgery-curriculum-aug-2021-approved-oct-20v3.pdf
Marriott J, Purdie H, Crossley J, Beard JD (2011) Evaluation of procedure-based assessment for assessing trainees’ skills in the operating theatre. Br J Surg 98:450–457
doi: 10.1002/bjs.7342
pubmed: 21254025
Toale C, Morris M, Kavanagh DO (2022) Assessing operative skill in the competency-based education era: lessons from the UK and Ireland. Ann Surg 275(4):e615–e625. https://doi.org/10.1097/SLA.0000000000005242
doi: 10.1097/SLA.0000000000005242
pubmed: 35129531
Agha RA, Fowler AJ (2015) The role and validity of surgical simulation. Int Surg 100:350–357
doi: 10.9738/INTSURG-D-14-00004.1
pubmed: 25692441
pmcid: 4337453
Milburn JA, Khera G, Hornby ST, Malone PSC, Fitzgerald JEF (2012) Introduction, availability and role of simulation in surgical education and training: review of current evidence and recommendations from the association of surgeons in training. Int J Surg 10:393–398
doi: 10.1016/j.ijsu.2012.05.005
pubmed: 22609475
de Montbrun SL, Roberts PL, Lowry AC, Ault GT, Burnstein MJ, Cataldo PA, Dozois EJ, Dunn GD, Fleshman J, Isenberg GA, Mahmoud NN, Reznick RK, Satterthwaite L, Schoetz D Jr, Trudel JL, Weiss EG, Wexner SD, MacRae H (2013) A novel approach to assessing technical competence of colorectal surgery residents: the development and evaluation of the Colorectal Objective Structured Assessment of Technical Skill (COSATS). Ann Surg 258:1001–1006
doi: 10.1097/SLA.0b013e31829b32b8
pubmed: 23817507
Halwani Y, Sachdeva AK, Satterthwaite L, de Montbrun S (2019) Development and evaluation of the General Surgery Objective Structured Assessment of Technical Skill (GOSATS). Br J Surg 106:1617–1622
doi: 10.1002/bjs.11359
pubmed: 31588561
de Montbrun S, Roberts PL, Satterthwaite L, MacRae H (2016) Implementing and evaluating a national certification technical skills examination: the colorectal objective structured assessment of technical skill. Ann Surg 264:1–6
doi: 10.1097/SLA.0000000000001620
pubmed: 26764869
Association AER (1999) American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing
Cook DA, Hatala R (2016) Validation of educational assessments: a primer for simulation and beyond. Adv Simul 1:31
doi: 10.1186/s41077-016-0033-y
Toale C, Morris M, Kavanagh D (2021) Perceptions and experiences of simulation-based assessment of technical skill in surgery: a scoping review. Am J Surg 222:723–730
doi: 10.1016/j.amjsurg.2021.03.035
pubmed: 33849711
Toale C, Morris M, Kavanagh DO (2022) Perspectives on simulation-based assessment of operative skill in surgical training. Med Teach. https://doi.org/10.1080/0142159X.2022.2134001
doi: 10.1080/0142159X.2022.2134001
pubmed: 36288727
Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278
pubmed: 9052454
Nayahangan LJ, Clementsen PF, Paltved C, Lindorff-Larsen KG, Nielsen BU, Konge L (2016) Identifying technical procedures in pulmonary medicine that should be integrated in a simulation-based curriculum: a national general needs assessment. Respiration 91:517–522
doi: 10.1159/000446926
pubmed: 27287472
Toale C, Morris M, Konge L, Nayahangan LJ, Roche A, Heskin L, Kavanagh DO (2024) Generating a prioritized list of operative procedures for simulation-based assessment of general surgery trainees through consensus. Ann Surg 279:900–905
doi: 10.1097/SLA.0000000000006118
pubmed: 37811854
Mayne A, Wilson L, Kennedy N (2020) The usefulness of procedure-based assessments in postgraduate surgical training within the intercollegiate surgical curriculum programme; a scoping review. J Surg Educ 77:1227–1235
doi: 10.1016/j.jsurg.2020.03.005
pubmed: 32451310
Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ 327:1032–1037
doi: 10.1136/bmj.327.7422.1032
pubmed: 14593041
pmcid: 261663
Vaidya A, Aydin A, Ridgley J, Raison N, Dasgupta P, Ahmed K (2020) Current status of technical skills assessment tools in surgery: a systematic review. J Surg Res 246:342–378
doi: 10.1016/j.jss.2019.09.006
pubmed: 31690531
de Blacam C, O’Keeffe DA, Nugent E, Doherty E, Traynor O (2012) Are residents accurate in their assessments of their own surgical skills? The American Journal of Surgery 204:724–731
doi: 10.1016/j.amjsurg.2012.03.003
pubmed: 22608671
Schijven MP, Reznick RK, ten Cate OTJ, Grantcharov TP, Regehr G, Satterthwaite L, Thijssen AS, MacRae HM (2010) Transatlantic comparison of the competence of surgeons at the start of their professional career. Br J Surg 97:443–449
doi: 10.1002/bjs.6858
pubmed: 20095020
Dwyer T, Wright S, Kulasegaram KM, Theodoropoulos J, Chahal J, Wasserstein D, Ringsted C, Hodges B, Ogilvie-Harris D (2016) How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE). BMC Med Educ 16:1
doi: 10.1186/s12909-015-0506-z
pubmed: 26727954
pmcid: 4700734
Toale C, Morris M, Kavanagh DO (2022) Training and assessment using the LapSim laparoscopic simulator: a scoping review of validity evidence. Surg Endosc. https://doi.org/10.1007/s00464-022-09593-0
doi: 10.1007/s00464-022-09593-0
pubmed: 36123545
Andersen SAW, Nayahangan LJ, Park YS, Konge L (2021) Use of generalizability theory for exploring reliability of and sources of variance in assessment of technical skills: a systematic review and meta-analysis. Acad Med 96:1609–1619
doi: 10.1097/ACM.0000000000004150
pubmed: 33951677
Yudkowsky R, Park YS, Downing SM (2019) Assessment in health professions education. Routledge
doi: 10.4324/9781315166902
Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L, van der Vleuten C (2003) Comparison of a rational and an empirical standard setting procedure for an OSCE: objective structured clinical examinations. Med Educ 37:132–139
doi: 10.1046/j.1365-2923.2003.01429.x
pubmed: 12558884
Yudkowsky R, Tumuluru S, Casey P, Herlich N, Ledonne C (2014) A patient safety approach to setting pass/fail standards for basic procedural skills checklists. Simul Healthc 9:277–282
doi: 10.1097/SIH.0000000000000044
pubmed: 25188484
Barsuk JH, Cohen ER, Wayne DB, McGaghie WC, Yudkowsky R (2018) A comparison of approaches for mastery learning standard setting. Acad Med 93:1079–1084
doi: 10.1097/ACM.0000000000002182
pubmed: 29465449
Sousa J, Mansilha A (2019) European panomara on vascular surgery: results from 5 years of FEBVS examinations. Angiologia e Cirurgia Vascular 15:171–175
Stefanidis D, Arora S, Parrack DM, Hamad GG, Capella J, Grantcharov T, Urbach DR, Scott DJ, Jones DB (2012) Research priorities in surgical simulation for the 21st century. Am J Surg 203:49–53
doi: 10.1016/j.amjsurg.2011.05.008
pubmed: 22172482
pmcid: 3322506
Seaman SJ, Jorgensen EM, Tramontano AC, Jones DB, Mendiola ML, Ricciotti HA, Hur HC (2021) Use of fundamentals of laparoscopic surgery testing to assess gynecologic surgeons: a retrospective cohort study of 10-years experience. J Minim Invasive Gynecol 28:794–800
doi: 10.1016/j.jmig.2020.07.005
pubmed: 32681993
Bilgic E, Kaneva P, Okrainec A, Ritter EM, Schwaitzberg SD, Vassiliou MC (2018) Trends in the Fundamentals of Laparoscopic Surgery® (FLS) certification exam over the past 9 years. Surg Endosc 32:2101–2105
doi: 10.1007/s00464-017-5907-0
pubmed: 29067579
Rooney DM, Brissman IC, Gauger PG (2015) Ongoing evaluation of video-based assessment of proctors’ scoring of the fundamentals of laparoscopic surgery manual skills examination. J Surg Educ 72:471–476
doi: 10.1016/j.jsurg.2014.10.017
pubmed: 25863704
Ritter EM, Brissman IC (2016) Systematic development of a proctor certification examination for the Fundamentals of Laparoscopic Surgery testing program. Am J Surg 211:458–463
doi: 10.1016/j.amjsurg.2015.10.001
pubmed: 26651968
Xeroulis G, Dubrowski A, Leslie K (2009) Simulation in laparoscopic surgery: a concurrent validity study for FLS. Surg Endosc 23:161–165
doi: 10.1007/s00464-008-0120-9
pubmed: 18814001
General Medical Council UK (2024) Performance assessments as part of an investigation or hearing. In: General Medical Council UK (ed.) Performance assessments as part of an investigation or hearing. GMC-UK. www.gmc-uk.org
Sousa J, Mansilha A (2021) European panorama on vascular surgery: results from five years of FEBVS examinations. Eur J Vasc Endovasc Surg 61:S41–S42
doi: 10.1016/j.ejvs.2021.01.034
Louridas M, Szasz P, de Montbrun S, Harris KA, Grantcharov TP (2016) International assessment practices along the continuum of surgical training. Am J Surg 212:354–360
doi: 10.1016/j.amjsurg.2015.12.017
pubmed: 27018078