Comparing 2 Approaches for the File Review of Residency Applications.
Journal
Journal of graduate medical education
ISSN: 1949-8357
Titre abrégé: J Grad Med Educ
Pays: United States
ID NLM: 101521733
Informations de publication
Date de publication:
Apr 2021
Apr 2021
Historique:
received:
11
06
2020
revised:
13
10
2020
revised:
11
12
2020
accepted:
15
12
2020
entrez:
26
4
2021
pubmed:
27
4
2021
medline:
29
7
2021
Statut:
ppublish
Résumé
The residency selection process relies on subjective information in applications, as well as subjective assessment of applications by reviewers. This inherent subjectivity makes residency selection prone to poor reliability between those reviewing files. We compared the interrater reliability of 2 assessment tools during file review: one rating applicant traits (ie, leadership, communication) and the other using a global rating of application elements (ie, curriculum vitae, reference letters). Ten file reviewers were randomized into 2 groups, and each scored 7 general surgery applications from the 2019-2020 cycle. The first group used an element-based (EB) scoring tool, while the second group used a trait-based (TB) scoring tool. Feedback was collected, discrimination capacities were measured using variation in scores, and interrater reliability (IRR) was calculated using intraclass correlation (ICC) in a 2-way random effects model. Both tools identified the same top-ranked and bottom-ranked applicants; however, discrepancies were noted for middle-ranked applicants. The score range for the 5 middle-ranked applicants was greater with the TB tool (6.43 vs 3.80), which also demonstrated fewer tie scores. The IRR for TB scoring was superior to EB scoring (ICC [2, 5] = 0.82 vs 0.55). The TB tool required only 2 raters to achieve an ICC ≥ 0.70. Using a TB file review strategy can facilitate file review with improved reliability compared to EB, and a greater spread of candidate scores. TB file review potentially offers programs a feasible way to optimize and reflect their institution's core values in the process.
Sections du résumé
BACKGROUND
BACKGROUND
The residency selection process relies on subjective information in applications, as well as subjective assessment of applications by reviewers. This inherent subjectivity makes residency selection prone to poor reliability between those reviewing files.
OBJECTIVES
OBJECTIVE
We compared the interrater reliability of 2 assessment tools during file review: one rating applicant traits (ie, leadership, communication) and the other using a global rating of application elements (ie, curriculum vitae, reference letters).
METHODS
METHODS
Ten file reviewers were randomized into 2 groups, and each scored 7 general surgery applications from the 2019-2020 cycle. The first group used an element-based (EB) scoring tool, while the second group used a trait-based (TB) scoring tool. Feedback was collected, discrimination capacities were measured using variation in scores, and interrater reliability (IRR) was calculated using intraclass correlation (ICC) in a 2-way random effects model.
RESULTS
RESULTS
Both tools identified the same top-ranked and bottom-ranked applicants; however, discrepancies were noted for middle-ranked applicants. The score range for the 5 middle-ranked applicants was greater with the TB tool (6.43 vs 3.80), which also demonstrated fewer tie scores. The IRR for TB scoring was superior to EB scoring (ICC [2, 5] = 0.82 vs 0.55). The TB tool required only 2 raters to achieve an ICC ≥ 0.70.
CONCLUSIONS
CONCLUSIONS
Using a TB file review strategy can facilitate file review with improved reliability compared to EB, and a greater spread of candidate scores. TB file review potentially offers programs a feasible way to optimize and reflect their institution's core values in the process.
Identifiants
pubmed: 33897958
doi: 10.4300/JGME-D-20-00619.1
pmc: PMC8054590
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
240-245Déclaration de conflit d'intérêts
Conflict of interest: The authors declare they have no competing interests.
Références
Acad Med. 2010 Oct;85(10 Suppl):S25-8
pubmed: 20881697
Arch Surg. 2008 Jul;143(7):647-51; discussion 651-2
pubmed: 18645106
J Chiropr Med. 2016 Jun;15(2):155-63
pubmed: 27330520
Arch Pediatr Adolesc Med. 2000 Mar;154(3):256-60
pubmed: 10710023
Acad Med. 2009 Oct;84(10 Suppl):S21-4
pubmed: 19907379
Adv Health Sci Educ Theory Pract. 2013 Aug;18(3):325-41
pubmed: 22581567
Med Educ. 2015 Feb;49(2):161-73
pubmed: 25626747
West J Emerg Med. 2019 Jul;20(4):641-646
pubmed: 31316704
J Grad Med Educ. 2010 Sep;2(3):322-6
pubmed: 21976076
J Surg Educ. 2012 May-Jun;69(3):340-3
pubmed: 22483135
Ann Surg. 2010 Sep;252(3):537-1; discussion 541-3
pubmed: 20739855
Clin Orthop Relat Res. 2002 Jun;(399):260-4
pubmed: 12011719