Psychometric validation of PROM instruments.
Rasch analyses
classical test theory
confirmatory factor analyses
construct validity
differential item functioning
modern test theory
patient-reported outcome measures
psychometric validation
Journal
Scandinavian journal of medicine & science in sports
ISSN: 1600-0838
Titre abrégé: Scand J Med Sci Sports
Pays: Denmark
ID NLM: 9111504
Informations de publication
Date de publication:
Jun 2021
Jun 2021
Historique:
revised:
19
11
2020
received:
25
05
2020
accepted:
14
12
2020
pubmed:
21
12
2020
medline:
8
6
2021
entrez:
20
12
2020
Statut:
ppublish
Résumé
The aim was to provide an overview of the different statistical methods for validation of patient-reported outcome measures, ranging from simple statistical methods available in all software packages to advanced statistical models that require specialized software. A non-technical summary of classical test theory (CTT) and modern test theory (MTT) is provided. Specifically, confirmatory factor analysis, item response theory, and Rasch analysis is outlined. One CTT and three MTT methods were used to validate the two subscales (Symptoms and Quality of Life) from the Knee Injury and Osteoarthritis Outcome Score (KOOS). For each methodology, two analyses were considered: (i) a unidimensional analysis ignoring the pre-specified dimensionality, and (ii) a two-dimensional analysis using the pre-specified dimensionality. While CTT did not adequately address central issues regarding the validity of the KOOS subscales, the three MTT methods yielded very similar results. In conclusion, MTT methods offer analysis of all relevant properties related to the validity of patient-reported outcome measures, while this is not the case for CTT. Claims about sufficient validity based on CTT methods are inadequate and should not be trusted.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
1225-1238Informations de copyright
© 2020 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd.
Références
Mokkink LB, de Vet HCW, Prinsen CAC, et al. COSMIN Risk of bias checklist for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27:1171-1179.
Rosenbaum PR. Criterion-related construct validity. Psychometrika. 1989;54:625-633.
Novick MR. The axioms and principal results of classical test theory. J Math Psychol. 1966;3:1-18.
Spearman C. General intelligence, objectively determined and measured. Am J Psychol. 1904;15:201.
DeVellis RF. Classical test theory. Med Care. 2006;44:S50-S59.
Traub RE. Classical test theory in historical perspective. Educ Measure Issues Pract. 2005;16:8-14.
Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297-334.
Crutzen R, Peters G-JY. Scale quality: alpha is an inadequate estimate and factor-analytic evidence is needed first of all. Health Psychol Rev. 2017;11:242-247.
Sijtsma K. On the use, the misuse, and the very limited usefulness of Cronbach’s Alpha. Psychometrika. 2009;74:107-120.
Birnbaum AL. Latent trait models and their use in inferring an examinee’s ability. In: Lord FM, Novick MR, eds. Statistical Theories of Mental Test. Reading, MA: Addison-Wesley; 1968.
Holland PW, Wainer H. Differential Item Functioning. Hillsdale, NJ: Erlbaum; 1993.
Lord FM. Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Erlbaum Associates; 1980.
Groenvold M, Bjorner JB, Klee MC, Kreiner S. Test for item bias in a quality of life questionnaire. J Clin Epidemiol. 1995;48:805-816.
Teresi JA, Ramirez M, Lai J-S, Silver S. Occurrences and sources of Differential Item Functioning (DIF) in patient-reported outcome measures: Description of DIF methods, and review of measures of depression, quality of life and general health. Psychol Sci Q. 2008;50:538.
Lazarsfeld PF. Latent structure analysis. In: Koch S, ed. Psychology: A Study of a Science, vol. 3. New York: McGraw-Hill Book Company; 1959:476-535.
Lazarsfeld PF. Latent structure analysis and test theory. In: Gulliksen H, Messick S, ed. Psychological Scaling: Theory and Applications. New York: McGraw-Hill Book Company; 1960:78-88.
Lord FM, Novick MR. Statistical theories of mental test scores. Reading, MA: Addison-Wesley Pub. Co; 1968.
Mokken RJ. A Theory and Procedure of Scale Analysis with Applications in Political Research. The Hague, The Netherlands: Mouton; 1971.
Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen: Danish National Institute for Educational Research; 1960.
Rasch G. On general laws and the meaning of measurement in psychology. In: Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, Volume 4: Contributions to biology and problems of medicine. Berkeley: University of California Press; 1961:321-333.
Jöreskog KG. A general approach to confirmatory maximum likelihood factor analysis. Psychometrika. 1969;34:183-202.
Kline RB. Principles and Practice of Structural Equation Modelling. 3rd edn. New York: Guilford Publications; 2010.
Muthén B. Latent variable structural equation modeling with categorical data. J Econometrics. 1983;22:43-65.
Muthén B. A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika. 1984;49:115-132.
Muthén BO, Satorra A. Technical aspects of Muthén’s liscomp approach to estimation of latent variable relations with a comprehensive measurement model. Psychometrika. 1995;60:489-503.
Jöreskog KG, Olsson UH, Wallentin FY. Multivariate Analysis with LISREL, 1st edn. Cham, Switzerland: Springer Publishing Company, Incorporated; 2016.
Byrne BM. Structural Equation Modeling with EQS and EQS/Windows: Basic Concepts, Applications, and Programming, 1st edn. Thousand Oaks, CA: Sage Publications, Inc.; 1994.
Muthén LK, Muthén BO. Mplus User’s Guide, 8th edn. Los Angeles, CA: Muthén & Muthén; 1998-2017.
Arbuckle JL. Amos (Version 23.0) [Computer Program]. Chicago: IBM SPSS; 2014.
Hartmann WM. The CALIS Procedure: Extended User’s Guide. Cary, NC: SAS Institute; 1992.
Rosseel Y. lavaan: An R Package for structural equation modeling. J Stat Software. 2012;48:1-36.
Holst KK, Budtz-Jørgensen E. Linear latent variable models: the lava-package. Comput Statistics. 2013;28:1385-1452.
Van der Linden WJ, Hambleton RK. Handbook of Modern Item Response Theory. New York, NY: Springer, New York; 1997.
Orlando M, Thissen D, Corp R, Carolina N. Likelihood-based item-fit indices for dichotomous item response theory models. Appl Psychol Meas. 2000;24:50-64.
Glas CAW, Suárez-Falcón JCS. A comparison of item-fit statistics for the three-parameter logistic model. Appl Psychol Meas. 2003;27:87-106.
Maydeu-Olivares A, Joe H. Limited- and full-information estimation and goodness-of-fit testing in 2 n contingency tables. J Am Stat Assoc. 2005;100:1009-1020.
Rizopoulos D. ltm: An R package for latent variable modeling and item response theory analyses. J Stat Software. 2006;17:1-25.
Robitzsch A, Kiefer T, Wu MTAM. Test analysis modules. 2018. https://CRAN.R-project.org/package=TAM
Chalmers RP. mirt: A multidimensional item response theory package for the R environment. J Stat Software. 2012;48(6):1-29.
An X, Yung Y-F. Item Response Theory: What it is and How You Can Use the IRT Procedure to Apply It. Cary, NC: SAS Institute; 2014.
Hardouin J-B, Mesbah M. The SAS Macro-Program %AnaQol to estimate the parameters of item responses theory models. Commun Stat Simul Comput. 2007;36:437-453.
Olsbjerg M, Christensen KB. SAS macros for longitudinal IRT models. PeerJ Preprints. 2018:6:e26740v1.
Cai L, Thissen D, du Toit SHC. IRTPRO for Windows [Computer software]. Lincolnwood, IL: Scientific Software International; 2011.
Ali US, Chang H-H, Anderson CJ. Location indices for ordinal polytomous items based on item response theory. ETS Res Rep Series. 2015;2015(2):1-13.
Fischer GH, Molenaar IW, eds. Rasch Models. New York, NY: Springer; 1995.
Christensen KB, Kreiner S, Mesbah M, eds. Rasch Models in Health. Hoboken, NJ: John Wiley & Sons, Inc.; 2013.
Kreiner S. Validity and objectivity: reflections on the role and nature of Rasch models. Nord Psychol. 2007;59:268-298.
Andrich D. Controversy and the Rasch model: a characteristic of incompatible paradigms? Med Care. 2004;42:I7-I16.
Glas CAW, Verhelst ND. Testing the Rasch model. In: Fischer GH, Molenaar IW, eds. Rasch Models - Foundations, Recent Developments, and Applications. Berlin, New York: Springer-Verlag; 1995:69-95.
Glas CAW, Verhelst ND. Tests of fit for polytomous Rasch models. In: Fischer GH, Molenaar IW, eds. Rasch Models - Foundations, Recent Developments, and Applications. Berlin, New York: Springer-Verlag; 1995:325-352.
Andersen EB. Sufficient statistics and latent trait models. Psychometrika. 1977;42:69-81.
Andrich D. A Rating formulation for ordered response categories. Psychometrika. 1978;43:561-573.
Masters GN. A Rasch model for partial credit scoring. Psychometrika. 1982;47:149-174.
Christensen KB. Fitting polytomous Rasch models in SAS. J Appl Meas. 2006;7:407-417.
Christensen KB. Conditional maximum likelihood estimation in polytomous Rasch models using SAS. ISRN Comput Math. 2013;2013:1-8.
Christensen KB, Olsbjerg M. Marginal maximum likelihood estimation in polytomous Rasch models using SAS. Pub Inst Stat Univ Paris. 2013;57:69-84.
Hardouin JB. Rasch analysis: estimation and tests with the Raschtest module. Stata J. 2007;7:22-44.
Mair P, Hatzinger R. Extended Rasch modeling: the eRm package for the application of IRT Models in R. J Stat Software. 2007;20:9.
Mair P, Hatzinger R. CML based estimation of extended Rasch models with the eRm package in R. Psychol Sci. 2007;49:26-43.
Andrich D, Sheridan B, Luo G. RUMM2030 [Computer Software and Manual]. Perth, Australia: RUMM Laboratory; 2010.
Linacre JM. WINSTEPS: Rasch Measurement Computer Program. Beaverton, Oregon: winsteps.com; 2011.
Kreiner S. Introduction to DIGRAM. Copenhagen: Department of statistics, University of Copenhagen; 2003.
Kreiner S, Nielsen T. Item Analysis in DIGRAM 3.04: Part I: Guided Tours. Copenhagen: Department of statistics, University of Copenhagen; 2013.
Kreiner S, Christensen KB. Graphical Rasch models. In: Mesbah M, Cole B, Lee MLT, eds. Statistical Methods for Quality of Life Studies. Dordrecht: Springer; 2002:187-203.
Roos EM, Roos HP, Lohmander LS, Ekdahl C, Beynnon BD. Knee injury and osteoarthritis outcome score (KOOS)-Development of a self-administered outcome measure. J Orthop Sports Phys Ther. 1998;28:88-96.
Comins J, Brodersen J, Krogsgaard M, Beyer N. Rasch analysis of the Knee injury and Osteoarthritis Outcome Score (KOOS): a statistical re-evaluation. Scand J Med Sci Sports. 2008;18:336-345.
Streiner DL, Norman GR, Cairney J. Health Measurement Scales, vol. 1. Oxford: Oxford University Press; 2015.
Hu L, Bentler P. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modelling. 2017;6:1-55.
Chen W-H, Thissen D. Local dependence indexes for item pairs using item response theory. J Educ Behav Stat. 1997;22:265-289.
Thissen D, Wainer H. Test Scoring. Mahwah, NJ: Lawrence Erlbaum Associates; 2001.
Tennant A, Penta M, Tesio L, et al. Assessing and adjusting for cross-cultural validity of impairment and activity limitation scales through differential item functioning within the framework of the Rasch model: the PRO-ESOR project. Med Care. 2004;42:I37-48.
Christensen KB, Kreiner S. Item fit statistics. In: Rasch Models in Health. Hoboken, NJ: John Wiley and Sons, Inc.; 2013:83-104.
Andrich D. Rasch Models for Measurement. London: Sage Publications; 1988.
Andrich D. An expanded derivation of the threshold structure of the polytomous Rasch model that dispels any “threshold disorder controversy”. Educ Psychol Meas. 2013;73:78-124.
Piscitelli D, Pellicciari L. Responsiveness: is it time to move beyond ordinal scores and approach interval measurements? Clin Rehabil. 2018;32:1426-1427.
Grimby G, Tennant A, Tesio L. The use of raw scores from ordinal scales: time to end malpractice? J Rehabil Med. 2012;44:97-98.
Ørnbjerg L, Christensen KB, Tennant A, Hetland M. Validation and assessment of minimally clinically important difference of the unadjusted Health Assessment Questionnaire in a Danish cohort: uncovering ordinal bias. Scand J Rheumatol. 2020;49:1-7.
Forero CG, Maydeu-Olivares A, Gallardo-Pujol D. Factor analysis with ordinal indicators: a Monte Carlo study comparing DWLS and ULS estimation. Structural Equ Modeling. 2009;16:625-641.
Reise SP, Yu J. Parameter recovery in the graded response model using MULTILOG. J Educ Meas. 1990;27:133-144.
Chen WH, Lenderking W, Jin Y, Wyrwich KW, Gelhorn H, Revicki DA. Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Qual Life Res. 2014;23:485-493.