Longitudinal assessment of undergraduate dental students: Building evidence for validity.
continuous assessment
longitudinal assessment
trajectories
trajectory
undergraduate clinical assessment
validity
Journal
European journal of dental education : official journal of the Association for Dental Education in Europe
ISSN: 1600-0579
Titre abrégé: Eur J Dent Educ
Pays: England
ID NLM: 9712132
Informations de publication
Date de publication:
Nov 2023
Nov 2023
Historique:
revised:
27
10
2022
received:
21
12
2021
accepted:
31
03
2023
medline:
23
10
2023
pubmed:
4
5
2023
entrez:
4
5
2023
Statut:
ppublish
Résumé
To investigate the content and criterion validity, and reliability of longitudinal clinical assessment of undergraduate dental student clinical competence by determining patterns of clinical performance and comparing them with validated standalone undergraduate examinations. Group-based trajectory models tracking students' clinical performance over time were produced from LIFTUPP© data for three dental student cohorts (2017-19; n = 235) using threshold models based on the Bayesian information criterion. Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Criterion validity was investigated using performance indicator 5 to create distinct trajectories of performance before linking and cross-tabulating trajectory group memberships with a 'top 20%' performance in the final Bachelor of Dental Surgery (BDS) examinations. Reliability was calculated using Cronbach's alpha. Threshold 4 models showed all students followed a single upward trajectory in all three cohorts, showing clear progression in competence over three clinical BDS years. A threshold 5 model produced two distinct trajectories, and in each cohort a 'better performing' trajectory was identified. Students allocated to the 'better performing' trajectories scored higher on average in the final examinations for cohort 2 (29% vs 18% (BDS4); 33% vs. 15% (BDS5)) and cohort 3 (19% vs. 16% (BDS4); 21% vs. 16% (BDS5)). Reliability for the undergraduate examinations was high for all three cohorts (≥0.8815) and did not change appreciably when longitudinal assessment was included. There is some evidence to support that longitudinal data have a degree of content and criterion validity for assessing the development of clinical competence in undergraduate dental students, which should increase confidence in decisions based on these data. The findings also provide a good foundation for subsequent research.
Types de publication
Journal Article
Langues
eng
Pagination
1136-1150Subventions
Organisme : Dorothy Geddes Research Fellowship
Organisme : NHS Education for Scotland
Informations de copyright
© 2023 The Authors. European Journal of Dental Education published by John Wiley & Sons Ltd.
Références
General Chiropractic Council. Education Standards. General Chiropractic Council; 2017.
General Dental Council. Preparing for Practice (2015 Revised Edition). General Dental Council; 2015.
General Medical Council. Outcomes for Graduates. General Medical Council; 2018.
General Optical Council. Core Competencies for Optometrists. General Optical Council; 2016.
General Osteopathic Council. Guidance for Osteopathic Pre-Registration Education. General Osteopathic Council; 2015.
UK Health and Safety Executive. Who regulates health and social care? [Online]. UK Health and Safety Executive. Accessed July 2020.
Roudsari RV. Assessment of competence in dentistry: the expectations, perceptions, and predictions. Doctor of Philosophy in Clinical Dentistry. University of Manchester; 2017.
Williams J, Baillie S, Rhind S, Warman S, Sandy J, Ireland AA. Guide to Assessment in Dental Education. University of Bristol; 2015. Accessed: September 2020.
Field JC, Walmsley AD, Paganelli C, et al. The graduating European dentist: contemporaneous methods of teaching, learning and assessment in dental undergraduate education. Eur J Dent Educ. 2017;21:28-35.
Prescott-Clements L, Van der Vleuten CP, Schuwirth LW, Hurst Y, Rennie JS. Evidence for validity within workplace assessment: the longitudinal evaluation of performance (LEP). Med Educ. 2008;42:488-495.
Albino JEN, Young SK, Neumann LM, et al. Assessing dental Students' competence: best practice recommendations in the performance assessment literature and investigation of current practices in predoctoral dental education. J Dent Educ. 2008;72:1405-1435.
Dawson LJ, Mason BG, Bissell V, Youngson C. Calling for a re-evaluation of the data required to credibly demonstrate a dental student is safe and ready to practice. Eur J Dent Educ. 2017;21:130-135.
Patel US, Tonni I, Gadbury-Amyot C, Van der Vleuten CPM, Escudier M. Assessment in a global context: an international perspective on dental education. Eur J Dent Educ. 2018;22(Suppl 1):21-27.
Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013;50:1-73.
Jones BL, Nagin DS. A Stata Plugin for Estimating Group-Based Trajectory Models. Carnegie Mellon University; 2012. https://ssrc.indiana.edu/doc/wimdocs/2013-03-29_nagin_trajectory_stata-plugin-info.pdf. Accessed: September 2020.
Jones BL, Nagin DS. A note on a Stata plugin for estimating group-based trajectory models. Sociol Methods Res. 2013;42(4):608-613.
Orcutt HK, Erickson DJ, Wolfe J. The course of PTSD symptoms among gulf war veterans: a growth mixture modeling approach. J Trauma Stress. 2004;17:195-202.
Dekker MC, Ferdinand RF, Van Lang NDJ, Bongers IL, Van der Ende JD, Verhulst FC. Developmental trajectories of depressive symptoms from early childhood to late adolescence: gender differences and adult outcome. J Child Psychol Psychiatry. 2007;48:657-666.
Jester JM, Nigg JT, Buu A, et al. Trajectories of childhood aggression and inattention/hyperactivity: differential effects on substance abuse in adolescence. J Am Acad Child Adolesc Psychiatry. 2008;47:1158-1165.
Mora PA, Bennett IM, Elo IT, Mathew L, Coyne JC, Culhane JF. Distinct trajectories of perinatal depressive symptomatology: evidence from growth mixture modeling. Am J Epidemiol. 2009;169:24-32.
Girard LC, Tremblay RE, Nagin D, Côté SM. Development of aggression subtypes from childhood to adolescence: a group-based multi-trajectory modelling perspective. J Abnorm Child Psychol. 2019;47(5):825-838.
Weisburd D, Bushway S, Lum C, Yang SM. Trajectories of crime at places: a longitudinal study of street segments in the city of Seattle. Criminology. 2004;42:283-320.
Nagin DS, Odgers CL. Group-based trajectory modeling in clinical research. Annu Rev Clin Psychol. 2010;6:109-138.
Nagin D, Piquero A. Using the group-based trajectory model to study crime over the life course. J Crim Just Educ. 2010;21(2):105-116.
Melhuish E, Sylva K, Sammons P, Siraj-Blatchford I, Taggart B. Effective Pre-School, Primary & Secondary Education Project: an Investigation of Children's Learning Trajectories from 3 to 11 Years of Age in Literacy and Numeracy. Institute of Education, University of London; 2011.
Sutcliffe AG, Gardiner J, Melhuish E. Educational Progress of looked-after children in England: a study using group trajectory analysis. Pediatrics. 2017;140(3):e20170503.
Shearer DM, Thomson WM, Broadbent JM, McLean R, Poulton R, Mann J. High-risk glycated hemoglobin trajectories established by mid-20s: findings from a birth cohort study. Br Med J Open Diabet Res Care. 2016;4:e000243.
Broadbent JM, Thomson WM, Poulton R. Trajectory patterns of dental caries experience in the permanent dentition to the fourth decade of life. J Dent Res. 2008;87(1):69-72.
Thomson WM, Shearer DM, Broadbent JM, Foster Page LA, Poulton R. The natural history of periodontal attachment loss during the third and fourth decades of life. J Clin Periodontol. 2013;40(7):672-680.
Kline P. A Pyschometrics Primer. Free Association Books; 2000.
Schuwirth LWT, van der Vleuten CPM. How to design a useful test: the principles of assessment. In: Swanwick T, ed. Understanding Medical Education: Evidence, Theory and Practice. 2nd ed. John Wiley & Sons, Ltd.; 2014.
Jolly B, Dalton MJ. Written assessment. In: Swanwick T, Forrest K, O'Brien BC, eds. Understanding Medical Education: Evidence, Theory, and Practice. 3rd ed. Wiley-Blackwell; 2019.
Boursicot KAM, Roberts TE, Burdick WP. Structured assessments of clinical competence. In: Swanwick T, Forrest K, O'Brien BC, eds. Understanding Medical Education: Evidence, Theory, and Practice. Wiley-Blackwell; 2019.
Nagin D. Group-Based Modeling of Development. Harvard University Press; 2005.
Webb GI. Posterior probability. In: Sammut C, Webb GI, eds. Encyclopedia of Machine Learning. Springer; 2011.
Raftery AE. Bayesian model selection in social research. Sociol Methodol. 1995;25:111-163.
Hoffman JIE. Hypothesis testing: the null hypothesis, significance and type I error. Basic Biostatistics for Medical and Biomedical Practitioners. 2nd ed. Elsevier Science; 2019:159-171.
Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297-334.
Holmboe HS, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents' clinical competence: a randomized trial. Ann Intern Med. 2004;140(11):874-881.
Brown G, Manogue M, Martin M. The validity and reliability of an OSCE in dentistry. Eur J Dental Educ. 1999;3(3):117-125.
Hodges B. Validity and the OSCE. Med Teach. 2003;25(3):250-254.
Park RS, Chibnall JT, Blaskiewicz RJ, Furman GE, Powell JK, Mohr CJ. Construct validity of an objective structured clinical examination (OSCE) in psychiatry: associations with the clinical skills examination and other indicators. Acad Psychiatry. 2004;28(2):122-128.
Varkey P, Natt N, Lesnick T, Downing S, Yudkowsky R. Validity evidence for an OSCE to assess competency in systems-based practice and practice-based learning and improvement: a preliminary investigation. Acad Med. 2008;83(8):775-780.
Taghva A, Panaghi L, Rasoulian M, Bolhari J, Zarghami M, Esfahani MN. Evaluation of reliability and validity of the psychiatry OSCE in Iran. Acad Psychiatry. 2010;34(2):154-157.
Barry M, Bradshaw C, Noonan M. Improving the content and face validity of OSCE assessment marking criteria on an undergraduate midwifery programme: a quality initiative. Nurse Educ Pract. 2013;13(5):477-480.
Nickbakht M, Amiri M, Latifi SM. Study of the reliability and validity of objective structured clinical examination (OSCE) in the assessment of clinical skills of audiology students. Global J Health Sci. 2013;5(3):64-68.
Sam AH, Westacott R, Gurnell M, Wilson R, Meeran K, Brown C. Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: cross-sectional study. BMJ Open. 2019;9(9):e032550.
Yepes-Rios M, Dudek N, Duboyce R, Curtis J, Allard RJ, Varpio L. The failure to fail underperforming trainees in health professions education: a BEME systematic review: BEME guide No. 42. Med Teach. 2016;38:1092-1099.
Dudek NL, Marks MB, Regehr G. Failure to fail: the perspectives of clinical supervisors. Acad Med. 2005;80:S84-S87.
Cleland JA, Knight LV, Rees CE, Tracey S, Bond CM. Is it me or is it them? Factors that influence the passing of underperforming students. Med Educ. 2008;42:800-809.
Chambers DW. Toward a competency-based curriculum. J Dent Educ. 1993;57:790-793.
Chambers DW. Competency-based dental education in context. Eur J Dent Educ. 1998;2:8-13.
Licari FW, Chambers DW. Some paradoxes in competency-based dental education. J Dent Educ. 2008;72:8-18.
Bush HM, Schreiber RS, Oliver SJ. Failing to fail: clinicians' experience of assessing underperforming dental students. Eur J Dent Educ. 2013;17:198-207.
Wilby KJ, Dolmans DHJM, Austin Z, Govaerts MJB. Assessors' interpretations of narrative data on communication skills in a summative OSCE. Med Educ. 2019;53:1003-1012.
Berendonk C, Stalmeijer RE, Schuwirth LWT. Expertise in performance assessment: assessors' perspectives. Adv Health Sci Educ Theory Pract. 2013;18:559-571.
Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 2011;86:S1-S7.
Cleland J, Arnold R, Chesser A. Failing finals is often a surprise for the student but not the teacher: identifying difficulties and supporting students with academic difficulties. Med Teach. 2005;27:504-508.
Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ. 2009;43:767-775.
McManus IC, Thompson M, Mollon J. Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Med Educ. 2006;6:42.
Hodges B. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 2013;35:564-568.
Lockyer J, Carraccio C, Chan M, et al. Core principles of assessment in competency-based medical education. Med Teach. 2017;39:609-616.
Hoang NS, Lau JN. A call for mixed methods in competency-based medical education: how we can prevent the overfitting of curriculum and assessment. Acad Med. 2018;93:996-1001.
Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37:830-837.
Harris P, Bhanji F, Topps M, et al. Evolving concepts of assessment in a competency-based world. Med Teach. 2017;39:603-608.
Royal KD. Four tenets of modern validity theory for medical education assessment and evaluation. Adv Med Educ Pract. 2017;8:567-570.
Roeder K, Lynch KG, Nagin DS. Modeling uncertainty in latent class membership: a case study in criminology. J Am Stat Assoc. 1999;94:766-776.
Nagin DS, Tremblay RE. Developmental trajectory groups: fact or a useful statistical fiction? Criminology (Beverly Hills). 2005;43:873-904.
NHS Education for Scotland. How to Assess your Vocational Dental Practitioner (VDP) Using the Longitudinal Evaluation of Performance (LEP). NHS Education for Scotland; 2021.