Impact of panelists' experience on script concordance test scores of medical students.
Medical student
Panelist
Script concordance test
Journal
BMC medical education
ISSN: 1472-6920
Titre abrégé: BMC Med Educ
Pays: England
ID NLM: 101088679
Informations de publication
Date de publication:
17 Sep 2020
17 Sep 2020
Historique:
received:
29
03
2020
accepted:
10
09
2020
entrez:
18
9
2020
pubmed:
19
9
2020
medline:
15
5
2021
Statut:
epublish
Résumé
The evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel. In 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots. A panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students' SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students' SCT scores, whatever the experience level of the expert panel. Even though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.
Sections du résumé
BACKGROUND
BACKGROUND
The evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel.
METHODS
METHODS
In 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots.
RESULTS
RESULTS
A panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students' SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students' SCT scores, whatever the experience level of the expert panel.
CONCLUSIONS
CONCLUSIONS
Even though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.
Identifiants
pubmed: 32943030
doi: 10.1186/s12909-020-02243-w
pii: 10.1186/s12909-020-02243-w
pmc: PMC7499961
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
313Références
Artif Intell Med. 2019 May;96:59-67
pubmed: 31164211
Med Educ. 2001 May;35(5):430-6
pubmed: 11328512
BMC Med Educ. 2016 Feb 03;16:47
pubmed: 26846526
Teach Learn Med. 2006 Winter;18(1):22-7
pubmed: 16354136
Med Educ. 2018 Mar;52(3):336-346
pubmed: 29318646
Acad Emerg Med. 2016 Dec;23(12):1362-1367
pubmed: 27442908
Can Med Educ J. 2016 Oct 18;7(2):e104-e113
pubmed: 28344697
Med Educ. 2012 Jun;46(6):552-63
pubmed: 22626047
Acad Med. 2000 Feb;75(2):182-90
pubmed: 10693854
Teach Learn Med. 2000 Fall;12(4):189-95
pubmed: 11273368
Med Educ. 2011 Apr;45(4):329-38
pubmed: 21401680
Med Educ. 2005 Mar;39(3):284-91
pubmed: 15733164
Med Teach. 2007 Feb;29(1):49-53
pubmed: 17538834
Rev Med Interne. 2016 Mar;37(3):166-72
pubmed: 26827272
Med Teach. 2013;35(3):184-93
pubmed: 23360487
BMC Med Inform Decis Mak. 2008 May 06;8:18
pubmed: 18460199
Acad Med. 2019 Mar;94(3):333-337
pubmed: 30334840