Modeling Diagnostic Expertise in Cases of Irreducible Uncertainty: The Decision-Aligned Response Model.
Journal
Academic medicine : journal of the Association of American Medical Colleges
ISSN: 1938-808X
Titre abrégé: Acad Med
Pays: United States
ID NLM: 8904605
Informations de publication
Date de publication:
01 01 2023
01 01 2023
Historique:
entrez:
28
12
2022
pubmed:
29
12
2022
medline:
31
12
2022
Statut:
ppublish
Résumé
Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician's decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example. The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from "definitely STEMI" to "definitely pericarditis," including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale ("definitely STEMI" to "definitely pericarditis"). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review. Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories. The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.
Identifiants
pubmed: 36576770
doi: 10.1097/ACM.0000000000004918
pii: 00001888-202301000-00028
pmc: PMC9780042
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
88-97Informations de copyright
Copyright © 2022 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Association of American Medical Colleges.
Références
Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288:3019–3026.
Ilgen JS, Eva KW, de Bruin A, Cook DA, Regehr G. Comfort with uncertainty: Reframing our conceptions of how clinicians navigate complex clinical situations. Adv Health Sci Educ Theory Pract. 2019;24:797–809.
Zhang S, Petersen JH. Quantifying rater variation for ordinal data using a rating scale model. Stat Med. 2018;37:2223–2237.
Schwarz RD. Trace lines for classification decisions. Appl Meas Edu. 1998;4:311–330.
Baldwin P, Bernstein J, Wainer H. Hip psychometrics. Stat Med. 2009;28:2277–2292.
Downing S. Item response theory: Applications of modern test theory in medical education. Med Educ. 2003;37:739–745.
Billings MS, Deruchie K, Hussie K, et al. NBME Item-Writing Guide. Constructing Written Test Questions for the Health Sciences. Philadelphia, PA: NBME; 2020. https://www.nbme.org/sites/default/files/2020-11/NBME_Item%20Writing%20Guide_2020.pdf . Accessed July 12, 2022.
Hatala R, Gutman J, Lineberry M, Triola M, Pusic M. How well is each learner learning? Validity investigation of a learning curve-based assessment approach for ECG interpretation. Adv Health Sci Educ Theory Pract. 2019;24:45–63.
Leblanc VR, Norman GR, Brooks LR. Effect of a diagnostic suggestion on diagnostic accuracy and identification of clinical features. Acad Med. 2001;76:S18–S20.
Crocker L, Algina J. Item Analysis. In: Introduction to Classical and Modern Test Theory. New York, NY: Holt, Rinehart, and Winston; 1986:311–338.
Samejima F. Estimation of latent ability using a response pattern of graded scores. Psychometrika. 1969;34:1–97.
Raykov T, Pusic M. Evaluation of polytomous item locations in multicomponent measuring instruments: A note on a latent variable modeling procedure [published online ahead of print March 2022]. Educ Psychol Meas. doi:10.1177%2F00131644211072829.
Durning SJ, Lubarsky S, Torre D, Dory V, Holmboe E. Considering “nonlinearity” across the continuum in medical education assessment: Supporting theory, practice, and future research directions. J Contin Educ Health Prof. 2015;35:232–243.
Lubarsky S, Dory V, Duggan P, Gagnon R, Charlin B. Script concordance testing: From theory to practice: AMEE guide no. 75. Med Teach. 2013;35:184–193.
Lineberry M, Kreiter CD, Bordage G. Threats to validity in the use and interpretation of script concordance test scores. Med Educ. 2013;47:1175–1183.
Cook DA, Oh SY, Pusic MV. Assessments of physicians’ electrocardiogram interpretation skill: A systematic review. Acad Med. 2022;97:603–615.
Brush JE Jr, Sherbino J, Norman GR. Diagnostic reasoning in cardiovascular medicine. BMJ. 2022;376:e064389.
Yoon JS, Boutis K, Pecaric MR, Fefferman NR, Ericsson KA, Pusic MV. A think-aloud study to inform the design of radiograph interpretation practice. Adv Health Sci Educ Theory Pract. 2020;25:877–903.
Ericsson KA. Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. Ericsson KA, Charness N, Feltovich P, Hoffman RR, eds. In: Cambridge Handbook of Expertise and Expert Performance. New York, NY: Cambridge University Press; 2006:223–242.
Crandall B, Klein G, Hoffman RR. Working Minds: A Practitioner’s Guide to Cognitive Task Analysis. Cambridge, MA: MIT Press; 2006.
Andrich D, Marais I. Invariance of comparisons—Separation of person and item parameters. In: A Course in Rasch Measurement Theory: Measuring in the Educational, Social and Health Sciences. Singapore: Springer; 2019:89–96.
Wright BD, Masters GN. Rating Scale Analysis. Rasch Measurement. Chicago, IL: Mesa Press; 1982. https://research.acer.edu.au/cgi/viewcontent.cgi?article=1001&context=measurement . Accessed July 12, 2022.
Rasch G. On general laws and the meaning of measurement in psychology. Berkeley Symp Math Stat Probab. 1961;4.4;321–333.
DeVellis RF, Thorpe CT. Scale Development: Theory and Applications. 5th ed. New York, NY: Sage Publications; 2022.