Correlation of Performance on ENTRUST and Traditional Oral Objective Structured Clinical Examination for High-Stakes Assessment in the College of Surgeons of East, Central, and Southern Africa.
Journal
Journal of the American College of Surgeons
ISSN: 1879-1190
Titre abrégé: J Am Coll Surg
Pays: United States
ID NLM: 9431305
Informations de publication
Date de publication:
01 07 2023
01 07 2023
Historique:
medline:
16
6
2023
pubmed:
5
5
2023
entrez:
5
5
2023
Statut:
ppublish
Résumé
To address the global need for accessible evidence-based tools for competency-based education, we developed ENTRUST, an innovative online virtual patient simulation platform to author and securely deploy case scenarios to assess surgical decision-making competence. In partnership with the College of Surgeons of East, Central, and Southern Africa, ENTRUST was piloted during the Membership of the College of Surgeons (MCS) 2021 examination. Examinees (n = 110) completed the traditional 11-station oral objective structured clinical examinations (OSCEs), followed by 3 ENTRUST cases, authored to query similar clinical content of 3 corresponding OSCE cases. ENTRUST scores were analyzed for associations with MCS Examination outcome using independent sample t tests. Correlation of ENTRUST scores to MCS Examination Percentage and OSCE station scores was calculated with Pearson correlations. Bivariate and multivariate analyses were performed to evaluate predictors of performance. ENTRUST performance was significantly higher in examinees who passed the MCS examination compared with those who failed (p < 0.001). The ENTRUST score was positively correlated with MCS Examination Percentage (p < 0.001) and combined OSCE station scores (p < 0.001). On multivariate analysis, there was a strong association between MCS Examination Percentage and ENTRUST Grand Total Score (p < 0.001), Simulation Total Score (p = 0.018), and Question Total Score (p < 0.001). Age was a negative predictor for ENTRUST Grand Total and Simulation Total Score, but not for Question Total Score. Sex, native language status, and intended specialty were not associated with performance on ENTRUST. This study demonstrates feasibility and initial validity evidence for the use of ENTRUST in a high-stakes examination context for assessment of surgical decision-making. ENTRUST holds potential as an accessible learning and assessment platform for surgical trainees worldwide.
Sections du résumé
BACKGROUND
To address the global need for accessible evidence-based tools for competency-based education, we developed ENTRUST, an innovative online virtual patient simulation platform to author and securely deploy case scenarios to assess surgical decision-making competence.
STUDY DESIGN
In partnership with the College of Surgeons of East, Central, and Southern Africa, ENTRUST was piloted during the Membership of the College of Surgeons (MCS) 2021 examination. Examinees (n = 110) completed the traditional 11-station oral objective structured clinical examinations (OSCEs), followed by 3 ENTRUST cases, authored to query similar clinical content of 3 corresponding OSCE cases. ENTRUST scores were analyzed for associations with MCS Examination outcome using independent sample t tests. Correlation of ENTRUST scores to MCS Examination Percentage and OSCE station scores was calculated with Pearson correlations. Bivariate and multivariate analyses were performed to evaluate predictors of performance.
RESULTS
ENTRUST performance was significantly higher in examinees who passed the MCS examination compared with those who failed (p < 0.001). The ENTRUST score was positively correlated with MCS Examination Percentage (p < 0.001) and combined OSCE station scores (p < 0.001). On multivariate analysis, there was a strong association between MCS Examination Percentage and ENTRUST Grand Total Score (p < 0.001), Simulation Total Score (p = 0.018), and Question Total Score (p < 0.001). Age was a negative predictor for ENTRUST Grand Total and Simulation Total Score, but not for Question Total Score. Sex, native language status, and intended specialty were not associated with performance on ENTRUST.
CONCLUSIONS
This study demonstrates feasibility and initial validity evidence for the use of ENTRUST in a high-stakes examination context for assessment of surgical decision-making. ENTRUST holds potential as an accessible learning and assessment platform for surgical trainees worldwide.
Identifiants
pubmed: 37144790
doi: 10.1097/XCS.0000000000000740
pii: 00019464-202307000-00014
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
117-127Informations de copyright
Copyright © 2023 by the American College of Surgeons. Published by Wolters Kluwer Health, Inc. All rights reserved.
Références
Liebert CA, Melcer EF, Keehl O, et al. Validity evidence for ENTRUST as an assessment of surgical decision-making for the inguinal hernia Entrustable Professional Activity (EPA). J Surg Educ 2022;79:e202–e212.
Lin DT, Melcer EF, Keehl O, et al. ENTRUST: a serious game-based virtual patient platform to assess Entrustable Professional Activities (EPAs) in graduate medical education. J Grad Med Educ 2023;15:228–236.
ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med 2007;82:542–547.
Ten Cate O, Chen HC, Hoff RG, et al. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Med Teach 2015;37:983–1002.
Brasel KJ, Klingensmith ME, Englander R, et al. Entrustable Professional Activities in general surgery: development and implementation. J Surg Educ 2019;76:1174–1186.
ten Cate O. When I say … entrustability. Med Educ 2020;54:103–104.
Messick S. Standards of validity and the validity of standards in performance assessment. Educ Meas Issues Pract 1995;14:5–8.
Cook DA, Zendejas B, Hamstra SJ, et al. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract 2014;19:233–250.
Cate O. A primer on entrustable professional activities. Korean J Med Educ 2018;30:1–10.
ABS announces transition to entrustable professional activities for general surgery resident evaluation. American Board of Surgery. February 14, 2022. Available at: https://www.absurgery.org/default.jsp?news_epas0222 . Accessed October 16, 2022.
Eaton M, Scully R, Schuller M, et al. Value and barriers to use of the SIMPL tool for resident feedback. J Surg Educ 2019;76:620–627.
Sandhu G, Nikolian VC, Magas CP, et al. OpTrust: validity of a tool assessing intraoperative entrustment behaviors. Ann Surg 2018;267:670–676.