Identifying the at-risk General Practice trainee: a retrospective cohort meta-analysis of General Practice registrar flagging.
Flagging
General Practice training
Meta-analysis
Postgraduate medical education
Remediation
Journal
Advances in health sciences education : theory and practice
ISSN: 1573-1677
Titre abrégé: Adv Health Sci Educ Theory Pract
Pays: Netherlands
ID NLM: 9612021
Informations de publication
Date de publication:
08 2021
08 2021
Historique:
received:
19
01
2020
accepted:
23
01
2021
pubmed:
16
2
2021
medline:
26
10
2021
entrez:
15
2
2021
Statut:
ppublish
Résumé
A central principle of programmatic assessment is that the final decision is not a surprise to the learner. To achieve this, assessments must demonstrate predictive and consequential validity, however, to date, research has only focussed on the former. The present study attempts to address this gap by examining the predictive and consequential validity of flagging systems used by Australian General Practice regional training organisations (RTOs) in relation to Fellowship examinations. Informed by unstructured interviews with Senior Medical Educators to understand the flagging system of each RTO, meta-analyses of routinely-collected flagging data were used to examine the predictive validity of flagging at various points in training and exam performance. Additionally, flagging system features identified from the interviews were used to inform exploratory subgroup analyses and meta-regressions to further assess the predictive and consequential validity of these systems. Registrars flagged near the end of their training were two to four times more likely to fail Fellowship exams than their non-flagged counterparts. Regarding flagging system features, having graded (i.e. ordinal) flagging systems was associated with higher accuracy, whilst involving the assigned medical educator in remediation and initiating a formal diagnostic procedure following a flag improved registrars' chances of passing exams. These results demonstrate both predictive and consequential validity of flagging systems. We argue that flagging is most effective when initiated early in training in conjunction with mechanisms to maximise diagnostic accuracy and the quality of remediation programs.
Identifiants
pubmed: 33587217
doi: 10.1007/s10459-021-10031-4
pii: 10.1007/s10459-021-10031-4
doi:
Types de publication
Journal Article
Meta-Analysis
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
1001-1025Informations de copyright
© 2021. The Author(s), under exclusive licence to Springer Nature B.V. part of Springer Nature.
Références
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: Wiley.
doi: 10.1002/9780470743386
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
doi: 10.1191/1478088706qp063oa
Butler, G., & Caulley, S. (2019, October). Unpacking the FRACGP written assessment. Paper presented at GP19, Adelaide.
Cilliers, F. J., Schuwirth, L. W., Adendorff, H. J., Herman, N., & van der Vleuten, C. P. (2010). The mechanism of impact of summative assessment on medical students’ learning. Advances in Health Sciences Education, 15(5), 695–715.
doi: 10.1007/s10459-010-9232-9
Cilliers, F. J., Schuwirth, L. W., Herman, N., Adendorff, H. J., & van der Vleuten, C. P. (2012a). A model of the pre-assessment learning effects of summative assessment in medical education. Advances in Health Sciences Education, 17(1), 39–53.
doi: 10.1007/s10459-011-9292-5
Cilliers, F. J., Schuwirth, L. W., & van der Vleuten, C. P. (2012b). A model of the pre-assessment learning effects of assessment is operational in an undergraduate clinical context. BMC Medical Education, 12(1), 9.
doi: 10.1186/1472-6920-12-9
de Virgilio, C., Yaghoubian, A., Kaji, A., Collins, J. C., Deveney, K., et al. (2010). Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Archives of Surgery, 145(9), 852–856. https://doi.org/10.1001/archsurg.2010.177 .
doi: 10.1001/archsurg.2010.177
Dudek, N., Marks, M., & Regehr, G. (2005). Failure to fail: The perspectives of clinical supervisors. Academic Medicine, 80(10 suppl), S84–S87.
doi: 10.1097/00001888-200510001-00023
Gielen, S., Dochy, F., & Dierick, S. (2003). Evaluating the consequential validity of new modes of assessment: The influence of assessment on learning, including pre-, post-, and true assessment effects. In M. Segers, F. Dochy, & E. Cascallar (Eds.), Optimising new modes of assessment: In search of qualities and standards (pp. 37–54). Berlin: Springer.
doi: 10.1007/0-306-48125-1_3
Glaser, B. G., & Strauss, A. L. (1967). The discovery of gorunded theory: Strategies for qualitative research. New Brunswick, NJ: Aldine.
Harrison, C. J., Könings, K. D., Dannefer, E. F., Schuwirth, L. W., Wass, V., et al. (2016). Factors influencing students’ receptivity to formative feedback emerging from different assessment cultures. Perspectives on Medical Education, 5(5), 276–284.
doi: 10.1007/s40037-016-0297-x
Harrison, C. J., Könings, K. D., Schuwirth, L., Wass, V., & van der Vleuten, C. (2015). Barriers to the uptake and use of feedback in the context of summative assessment. Advances in Health Sciences Education, 20(1), 229–245.
doi: 10.1007/s10459-014-9524-6
Harrison, C. J., Könings, K. D., Schuwirth, L. W., Wass, V., & van der Vleuten, C. P. (2017). Changing the culture of assessment: the dominance of the summative assessment paradigm. BMC Medical Education, 17(1), 73.
doi: 10.1186/s12909-017-0912-5
Heeneman, S., Schut, S., Donkers, J., Van der Vleuten, C., & Muijtjens, A. M. M. (2017). Embedding of the progress test in an assessment program designed according to the principles of programmatic assessment. Medical Teacher, 39(1), 44–52. https://doi.org/10.1080/0142159X.2016.1230183 .
doi: 10.1080/0142159X.2016.1230183
Higgins, J., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327, 557–560.
doi: 10.1136/bmj.327.7414.557
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I
doi: 10.1037/1082-989X.11.2.193
Magin, P., Stewart, R., Turnock, A., Tapley, A., Holliday, E., et al. (2017). Early predictors of need for remediation in the Australian General Practice Training program: A retrospective cohort study. Advances in Health Sciences Education, 22(4), 915–929.
doi: 10.1007/s10459-016-9722-5
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23.
doi: 10.3102/0013189X023002013
Naidoo, S., Lopes, S., Patterson, F., Mead, H. M., & MacLeod, S. (2017). Can colleagues’, patients’ and supervisors’ assessments predict successful completion of postgraduate medical training? Medical Education, 51(4), 423–431. https://doi.org/10.1111/medu.13128 .
doi: 10.1111/medu.13128
Patterson, F., Rowett, E., Hale, R., Grant, M., Roberts, C., et al. (2016). The predictive validity of a situational judgement test and multiple-mini interview for entry into postgraduate training in Australia. BMC Medical Education, 16, 87.
doi: 10.1186/s12909-016-0606-4
Prentice, S., Benson, J., Kirkpatrick, E. & Schuwirth, L. (2020). Workplace-based assessments in postgraduate medical education – a hermeneutic review. Medical Education, 54(11), 981–992. https://doi.org/10.1111/medu.14221
Schuwirth, L. W., & van der Vleuten, C. P. (2011). Programmatic assessment: From assessment of learning to assessment for learning. Medical Teacher, 33(6), 478–485.
doi: 10.3109/0142159X.2011.565828
Stewart, R., Cooling, N., Emblen, G., Turnock, A., Tapley, A., et al. (2018). Early predictors of summative assessment performance in general practice post-graduate training: A retrospective cohort study. Medical Teacher, 40(11), 1166–1174.
doi: 10.1080/0142159X.2018.1470609
The Royal Australian College of General Practitioners. (2018a). RACGP Education: Exam report 2018.1 AKT. East Melbourne, VIC: RACGP.
The Royal Australian College of General Practitioners. (2018b). RACGP Education: Exam report 2018.1 KFP. East Melbourne, VIC: RACGP.
The Royal Australian College of General Practitioners. (2018c). RACGP Education: Exam report 2018.1 OSCE. East Melbourne, VIC: RACGP.
The Royal Australian College of General Practitioners. (2018d). RACGP Education: Exam report 2018.2 AKT. East Melbourne, VIC: RACGP.
The Royal Australian College of General Practitioners. (2018e). RACGP Education: Exam report 2018.2 KFP. East Melbourne, VIC: RACGP.
The Royal Australian College of General Practitioners. (2018f). RACGP Education: Exam report 2018.2 OSCE. East Melbourne, VIC: RACGP.
van der Vleuten, C. P., & Schuwirth, L. (2005). Assessing professional competence: from methods to programmes. Medical Education, 39(3), 309–317.
doi: 10.1111/j.1365-2929.2005.02094.x
van der Vleuten, C. P., Schuwirth, L., Driessen, E., Dijkstra, J., Tigelaar, D., et al. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34(3), 205–214.
doi: 10.3109/0142159X.2012.652239
von Elmn, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., et al. (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for reporting observational studies. Bulletin of the World Health Organisation, 85(11), 867–872.
doi: 10.2471/BLT.07.045120
Watling, C., Driessen, E., van der Vleuten, C. P., Vanstone, M., & Lingard, L. (2013). Beyond individualism: professional culture and its influence on feedback. Medical Education, 47(6), 585–594.
doi: 10.1111/medu.12150