Meta-analyzing individual participant data from studies with complex survey designs: A tutorial on using the two-stage approach for data from educational large-scale assessments.
Programme for International Student Assessment
complex survey designs
educational large-scale assessments
individual participant data
meta-analysis
Journal
Research synthesis methods
ISSN: 1759-2887
Titre abrégé: Res Synth Methods
Pays: England
ID NLM: 101543738
Informations de publication
Date de publication:
Jan 2023
Jan 2023
Historique:
revised:
20
04
2022
received:
10
12
2021
accepted:
08
06
2022
pubmed:
8
7
2022
medline:
18
1
2023
entrez:
7
7
2022
Statut:
ppublish
Résumé
Descriptive analyses of socially important or theoretically interesting phenomena and trends are a vital component of research in the behavioral, social, economic, and health sciences. Such analyses yield reliable results when using representative individual participant data (IPD) from studies with complex survey designs, including educational large-scale assessments (ELSAs) or social, health, and economic survey and panel studies. The meta-analytic integration of these results offers unique and novel research opportunities to provide strong empirical evidence of the consistency and generalizability of important phenomena and trends. Using ELSAs as an example, this tutorial offers methodological guidance on how to use the two-stage approach to IPD meta-analysis to account for the statistical challenges of complex survey designs (e.g., sampling weights, clustered and missing IPD), first, to conduct descriptive analyses (Stage 1), and second, to integrate results with three-level meta-analytic and meta-regression models to take into account dependencies among effect sizes (Stage 2). The two-stage approach is illustrated with IPD on reading achievement from the Programme for International Student Assessment (PISA). We demonstrate how to analyze and integrate standardized mean differences (e.g., gender differences), correlations (e.g., with students' socioeconomic status [SES]), and interactions between individual characteristics at the participant level (e.g., the interaction between gender and SES) across several PISA cycles. All the datafiles and R scripts we used are available online. Because complex social, health, or economic survey and panel studies share many methodological features with ELSAs, the guidance offered in this tutorial is also helpful for synthesizing research evidence from these studies.
Types de publication
Meta-Analysis
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
5-35Subventions
Organisme : Deutsche Forschungsgemeinschaft
ID : 442358899
Informations de copyright
© 2022 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd.
Références
Loeb S, Dynarski S, McFarland D, Morris P, Reardon S, Reber S. Descriptive Analysis in Education: A Guide for Researchers. (NCEE 2017-4023). Institute of Education Sciences; 2017.
Naemi B, Gonzales E, Bertling J, et al. Large-scale group score assessments: past, present, and future. In: Saklofske DH, Reynolds CR, Schwean VL, eds. The Oxford Handbook of Child Psychological Assessment. Oxford Library of Psychology, Oxford University Press; 2013. doi:10.1093/oxfordhb/9780199796304.013.0006
Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ. 2010;340:c221. doi:10.1136/bmj.c221
Rao SR, Graubard BI, Schmid CH, et al. Meta-analysis of survey data: application to health services research. Health Serv Outcomes Res Methodol. 2008;8(2):98-114. doi:10.1007/s10742-008-0032-0
Else-Quest NM, Hyde JS, Linn MC. Cross-national patterns of gender differences in mathematics: a meta-analysis. Psychol Bull. 2010;136:103-127.
Nowell A, Hedges LV. Trends in gender differences in academic achievement from 1960 to 1994: an analysis of differences in mean, variance, and extreme scores. Sex Roles. 1998;39:21-43.
Reilly D, Neumann DL, Andrews G. Gender differences in reading and writing achievement: evidence from the National Assessment of Educational Progress (NAEP). Am Psychol. 2019;74(4):445-458. doi:10.1037/amp0000356
Sirin SR. Socioeconomic status and academic achievement: a meta-analytic review of research. Rev Educ Res. 2005;75:417-453.
Keller LK, Preckel F, Eccles JS, Brunner M. Top-performing math students in 82 countries: an integrative data analysis of gender differences in achievement, achievement profiles, and achievement motivation. J Educ Psychol. 2022;114(5):966-991. doi:10.1037/edu0000685
Keller LK, Preckel F, Brunner M. Nonlinear relations between achievement and academic self-concepts in elementary and secondary school: an integrative data analysis across 13 countries. J Educ Psychol. 2021;113(3):585-604. doi:10.1037/edu0000533
Blömeke S, Nilsen T, Scherer R. School innovativeness is associated with enhanced teacher collaboration, innovative classroom practices, and job satisfaction. J Educ Psychol. 2021;113(8):1645-1667. doi:10.1037/edu0000668
Guo J, Hu X, Marsh HW, Pekrun R. Relations of epistemic beliefs with motivation, achievement, and aspirations in science: generalizability across 72 societies. J Educ Psychol. 2022;114(4):734-751. doi:10.1037/edu0000660
Marsh HW. Cross-cultural generalizability of year in school effects: negative effects of acceleration and positive effects of retention on academic self-concept. J Educ Psychol. 2016;108(2):256-273. doi:10.1037/edu0000059
Else-Quest NM, Hyde JS. Intersectionality in quantitative psychological research: II. Methods and techniques. Psychol Women Q. 2016;40(3):319-336. doi:10.1177/0361684316647953
OECD. PISA 2015. Technical Report. OECD; 2017.
Rutkowski L, von Davier M, Rutkowski D, eds. Handbook of International Large-Scale Assessment Background, Technical Issues, and Methods of Data Analysis. CRC Press; 2014.
Findley MG, Kikuta K, Denly M. External validity. Annu Rev Polit Sci. 2021;24(1):365-393. doi:10.1146/annurev-polisci-041719-102556
Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company; 2002.
Kelley K, Preacher KJ. On effect size. Psychol Methods. 2012;17(2):137-152. doi:10.1037/a0028086
National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. Vol 25303. National Academies Press; 2019. doi:10.17226/25303
Simmonds M, Stewart G, Stewart L. A decade of individual participant data meta-analyses: a review of current practice. Contemp Clin Trials. 2015;45:76-83. doi:10.1016/j.cct.2015.06.012
Ahn S, Ames AJ, Myers ND. A review of meta-analyses in education: methodological strengths and weaknesses. Rev Educ Res. 2012;82(4):436-476. doi:10.3102/0034654312458162
Pigott TD, Polanin JR. Methodological guidance paper: high-quality meta-analysis in a systematic review. Rev Educ Res. 2020;90(1):24-46. doi:10.3102/0034654319877153
Tierney JF, Stewart LA, Clarke M. Chapter 26: Individual participant data. In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane; 2021 www.training.cochrane.org/handbook
Debray TPA, Moons KGM, van Valkenhoef G, et al. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Methods. 2015;6(4):293-309. doi:10.1002/jrsm.1160
Gray H, Lyth A, McKenna C, Stothard S, Tymms P, Copping L. Sex differences in variability across nations in reading, mathematics and science: a meta-analytic extension of Baye and Monseur (2016). Large-Scale Assess Educ. 2019;7(1):2. doi:10.1186/s40536-019-0070-9
Heeringa S, West BT, Berglund PA. Applied Survey Data Analysis. Taylor & Francis; 2010.
Lumley T. Complex Surveys: A Guide to Analysis Using R. John Wiley; 2010.
Valliant R, Dever JA, Kreuter F. Practical Tools for Designing and Weighting Survey Samples. 2nd ed. Springer; 2018.
Riley RD, Tierney JF, Stewart LA, eds. Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research. Wiley; 2021.
Burke DL, Ensor J, Riley RD. Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ. Stat Med. 2017;36(5):855-875. doi:10.1002/sim.7141
Morris TP, Fisher DJ, Kenward MG, Carpenter JR. Meta-analysis of Gaussian individual patient data: two-stage or not two-stage? Stat Med. 2018;37(9):1419-1438. doi:10.1002/sim.7589
Cooper HM, Hedges LV, Valentine JC. The Handbook of Research Synthesis and Meta-Analysis. 3rd ed. Russel Sage Foundation; 2019.
Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol. 2019;70(1):747-770. doi:10.1146/annurev-psych-010418-102803
Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis. Res Synth Methods. 2011;2(1):61-76. doi:10.1002/jrsm.35
Konstantopoulos S, Hedges LV. Statistically analyzing effect sizes: fixed- and random-effects models. In: Cooper HM, Hedges LV, Valentine JC, eds. Handbook of Research Synthesis and Meta-Analysis. 3rd ed. Russell Sage Foundation; 2019:245-280.
Riley RD, Burke DL, Morris T. One-stage versus two-stage approach to IPD meta-analysis: differences and recommendations. In: Riley RD, Tierney JF, Stewart LA, eds. Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research. Wiley Series in Statistics in Practice. Wiley; 2021:199-217.
Belias M, Rovers MM, Reitsma JB, Debray TPA, IntHout J. Statistical approaches to identify subgroups in meta-analysis of individual participant data: a simulation study. BMC Med Res Methodol. 2019;19(1):183. doi:10.1186/s12874-019-0817-6
Nevitt SJ, Marson AG, Davie B, Reynolds S, Williams L, Smith CT. Exploring changes over time and characteristics associated with data retrieval across individual participant data meta-analyses: systematic review. BMJ. 2017;357:j1390. doi:10.1136/bmj.j1390
Hussong AM, Curran PJ, Bauer DJ. Integrative data analysis in clinical psychology research. Annu Rev Clin Psychol. 2013;9(1):61-89. doi:10.1146/annurev-clinpsy-050212-185522
Siddique J, de Chavez PJ, Howe G, Cruden G, Brown CH. Limitations in using multiple imputation to harmonize individual participant data for meta-analysis. Prev Sci. 2018;19(1):95-108. doi:10.1007/s11121-017-0760-x
Cheung MWL, Jak S. Analyzing big data in psychology: a split/analyze/meta-analyze approach. Front Psychol. 2016;7. doi:10.3389/fpsyg.2016.00738
Hedges LV. What are effect sizes and why do we need them. Child Dev Perspect. 2008;2(3):167-171. doi:10.1111/j.1750-8606.2008.00060.x
Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1(2):97-111. doi:10.1002/jrsm.12
Tipton E, Pustejovsky JE, Ahmadi H. A history of meta-regression: technical, conceptual, and practical developments between 1974 and 2018. Res Synth Methods. 2019;10(2):161-179. doi:10.1002/jrsm.1338
Robinson WS. Ecological correlations and the behavior of individuals. Am Sociol Rev. 1950;15:351-357.
Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med. 2002;21(11):1559-1573. doi:10.1002/sim.1187
Riley RD, Debray TPA, Fisher D, et al. Individual participant data meta-analysis to examine interactions between treatment effect and participant-level covariates: statistical recommendations for conduct and planning. Stat Med. 2020;39(15):2115-2137. doi:10.1002/sim.8516
Fisher DJ, Carpenter JR, Morris TP, Freeman SC, Tierney JF. Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? BMJ. 2017;356:j573. doi:10.1136/bmj.j573
Enders CK, Tofighi D. Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. Psychol Methods. 2007;12:121-138.
Cohen J, Cohen P, Aiken LS, West SG. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. 3rd ed. Lawrence Erlbaum Associates; 2003.
Dalal DK, Zickar MJ. Some common myths about centering predictor variables in moderated multiple regression and polynomial regression. Organ Res Methods. 2012;15(3):339-362. doi:10.1177/1094428111430540
Fisher DJ, Copas AJ, Tierney JF, Parmar MKB. A critical review of methods for the assessment of patient-level interactions in individual participant data meta-analysis of randomized trials, and guidance for practitioners. J Clin Epidemiol. 2011;64(9):949-967. doi:10.1016/j.jclinepi.2010.11.016
Simmonds MC, Higgins JPT. Covariate heterogeneity in meta-analysis: criteria for deciding between meta-regression and individual patient data. Stat Med. 2007;26(15):2982-2999. doi:10.1002/sim.2768
Gasparrini A, Armstrong B, Kenward MG. Multivariate meta-analysis for non-linear and other multi-parameter associations. Stat Med. 2012;31(29):3821-3839. doi:10.1002/sim.5471
Stapleton LM. Incorporating sampling weights into single- and multilevel analyses. In: Rutkowski L, von Davier M, Rutkowski D, eds. Handbook of International Large-Scale Assessment Background, Technical Issues, and Methods of Data Analysis. CRC Press; 2014:363-388.
Snijders TAB, Bosker RJ. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. 2nd ed. Sage; 2012.
Raudenbush SW, Bryk AS. Hierarchical Linear Models. 2nd ed. Sage; 2002.
Tanner-Smith EE, Tipton E. Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Res Synth Methods. 2014;5(1):13-30. doi:10.1002/jrsm.1091
Audigier V, White IR, Jolani S, et al. Multiple imputation for multilevel data with continuous and binary variables. Stat Sci. 2018;33(2):160-183. doi:10.1214/18-STS646
OECD. PISA 2006. Technical Report. OECD; 2009.
Graham JW. Missing data analysis: making it work in the real world. Annu Rev Psychol. 2009;60:549-576.
Peugh JL, Enders CK. Missing data in educational research: a review of reporting practices and suggestions for improvement. Rev Educ Res. 2004;74(4):525-556. doi:10.3102/00346543074004525
Van Buuren S. Flexible Imputation of Missing Data. 2nd ed. CRC Press; 2018 https://stefvanbuuren.name/fimd/
Skinner C, Wakefield J. Introduction to the design and analysis of complex survey data. Stat Sci. 2017;32(2):165-175. doi:10.1214/17-STS614
Organisation for Economic Co-operation and Development. PISA Data Analysis Manual. SPSS. 2nd ed. OECD; 2009.
Rutkowski L, Gonzalez E, Joncas M, von Davier M. International large-scale assessment data: issues in secondary analysis and reporting. Educ Res. 2010;39(2):142-151. doi:10.3102/0013189X10363170
Stapleton LM. An assessment of practical solutions for structural equation modeling with complex sample data. Struct Equ Model Multidiscip J. 2006;13(1):28-58. doi:10.1207/s15328007sem1301_2
Brunner M, Keller U, Wenger M, Fischbach A, Lüdtke O. Between-school variation in students' achievement, motivation, affect, and learning strategies: results from 81 countries for planning group-randomized trials in education. J Res Educ Eff. 2018;11(3):452-478. doi:10.1080/19345747.2017.1375584
Rust K, Rao J. Variance estimation for complex surveys using replication techniques. Stat Methods Med Res. 1996;5(3):283-310. doi:10.1177/096228029600500305
Rust K. Sampling, weighting, and variance estimation in international large-scale assessments. In: Rutkowski L, von Davier M, Rutkowski D, eds. Handbook of International Large-Scale Assessment Background, Technical Issues, and Methods of Data Analysis. CRC Press; 2014:117-153.
Borenstein M, Hedges LV. Effect sizes for meta-analysis. In: Cooper HM, Hedges LV, Valentine JC, eds. Handbook of Research Synthesis and Meta-Analysis. 3rd ed. Russell Sage Foundation; 2019:207-243.
Pek J, Flora DB. Reporting effect sizes in original psychological research: a discussion and tutorial. Psychol Methods. 2018;23(2):208-225. doi:10.1037/met0000126
Cumming G, Calin-Jageman R. Introduction to the New Statistics: Estimation, Open Science, and Beyond. Routledge; 2016.
Hill CJ, Bloom HS, Black AR, Lipsey MW. Empirical benchmarks for interpreting effect sizes in research. Child Dev Perspect. 2008;2:172-177.
von Davier M, Sinharay S. Analytics in international large-scale assessments: item response theory and population models. In: Rutkowski L, von Davier M, Rutkowski D, eds. Handbook of International Large-Scale Assessment Background, Technical Issues, and Methods of Data Analysis. CRC Press; 2014:155-201.
Debray TPA, Snell KIE, Quartagno M, Jolani S, Moons KGM, Riley RD. Dealing with missing data in an IPD meta-analysis. In: Riley RD, Tierney JF, Stewart LA, eds. Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research. Wiley Series in Statistics in Practice. Wiley; 2021:499-524.
Resche-Rigon M, White IR. Multiple imputation by chained equations for systematically and sporadically missing multilevel data. Stat Methods Med Res. 2018;27(6):1634-1649. doi:10.1177/0962280216666564
Lüdtke O, Robitzsch A, West SG. Regression models involving nonlinear effects with missing data: a sequential modeling approach using Bayesian estimation. Psychol Methods. 2020;25(2):157-181. doi:10.1037/met0000233
Resche-Rigon M, White IR, Bartlett WJ, SAE P, Thompson SG. Multiple imputation for handling systematically missing confounders in meta-analysis of individual participant data. Stat Med. 2013;32(28):4890-4905. doi:10.1002/sim.5894
Jolani S, Debray TPA, Koffijberg H, Van Buuren S, Moons KGM. Imputation of systematically missing predictors in an individual participant data meta-analysis: a generalized approach using MICE. Stat Med. 2015;34(11):1841-1863. doi:10.1002/sim.6451
Kunkel D, Kaizar EE. A comparison of existing methods for multiple imputation in individual participant data meta-analysis. Stat Med. 2017;36(22):3507-3532. doi:10.1002/sim.7388
Enders CK, Du H, Keller BT. A model-based imputation procedure for multilevel regression models with random coefficients, interaction effects, and nonlinear terms. Psychol Methods. 2020;25(1):88-112. doi:10.1037/met0000228
Sterne JAC, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393. doi:10.1136/bmj.b2393
Carpenter JR, Kenward MG. Multiple Imputation and Its Application. John Wiley & Sons, Ltd; 2013. doi:10.1002/9781119942283
White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med. 2011;30(4):377-399. doi:10.1002/sim.4067
Collins LM, Schafer JL, Kam CM. A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychol Methods. 2001;6(4):330-351.
Burgess S, White IR, Resche-Rigon M, Wood AM. Combining multiple imputation and meta-analysis with individual participant data. Stat Med. 2013;32(26):4499-4514. doi:10.1002/sim.5844
Rubin DB. Nested multiple imputation of NMES via partially incompatible MCMC. Stat Neerlandica. 2003;57(1):3-18. doi:10.1111/1467-9574.00217
Robitzsch A, Lüdtke O. Mdmb: Model Based Treatment of Missing Data. R package version 1.5-8; 2021. https://cran.r-project.org/web/packages/mdmb/index.html
Grund S, Lüdtke O, Robitzsch A. Multiple imputation of missing data in multilevel models with the R package mdmb: a flexible sequential modeling approach. Behav Res Methods. 2021;53(6):2631-2649. doi:10.3758/s13428-020-01530-0
van Ginkel JR. Standardized regression coefficients and newly proposed estimators for R2 in multiply imputed data. Psychometrika. 2020;85(1):185-205. doi:10.1007/s11336-020-09696-4
West BT, Sakshaug JW, Aurelien GAS. Accounting for complex sampling in survey estimation: a review of current software tools. J Off Stat. 2018;34(3):721-752. doi:10.2478/jos-2018-0034
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2021 https://www.R-project.org/
Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates. Res Synth Methods. 2010;1(1):39-65. doi:10.1002/jrsm.5
Scammacca N, Roberts G, Stuebing KK. Meta-analysis with complex research designs: dealing with dependence from multiple measures and multiple group comparisons. Rev Educ Res. 2014;84(3):328-364. doi:10.3102/0034654313500826
Cheung MWL. Meta-Analysis: A Structural Equation Modeling Approach. Wiley; 2015.
Pustejovsky JE, Tipton E. Meta-analysis with robust variance estimation: expanding the range of working models. Prev Sci Off J Soc Prev Res. 2022;23(3):425-438. doi:10.1007/s11121-021-01246-3
Bender R, Friede T, Koch A, et al. Methods for evidence synthesis in the case of very few studies. Res Synth Methods. 2018;9(3):382-392. doi:10.1002/jrsm.1297
IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol. 2014;14(1):25. doi:10.1186/1471-2288-14-25
Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Stat Med. 2003;22(17):2693-2710. doi:10.1002/sim.1482
Sidik K, Jonkman JN. On constructing confidence intervals for a standardized mean difference in meta-analysis. Commun Stat - Simul Comput. 2003;32(4):1191-1203. doi:10.1081/SAC-120023885
Hedges LV, Olkin I. Statistical Methods for Meta-Analysis. Academic Press; 1985.
Tipton E. Small sample adjustments for robust variance estimation with meta-regression. Psychol Methods. 2015;20(3):375-393. doi:10.1037/met0000011
Veroniki AA, Jackson D, Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods. 2016;7(1):55-79. doi:10.1002/jrsm.1164
Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3). doi:10.18637/jss.v036.i03
Borenstein M, Higgins JPT, Hedges LV, Rothstein HR. Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Res Synth Methods. 2017;8(1):5-18. doi:10.1002/jrsm.1230
Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane; 2021 www.training.cochrane.org/handbook
Polanin JR, Hennessy EA, Tanner-Smith EE. A review of meta-analysis packages in R. J Educ Behav Stat. 2017;42(2):206-242. doi:10.3102/1076998616674315
Pustejovsky JE. ClubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections. R package version 0.5.3. 2021. https://CRAN.R-project.org/package=clubSandwich
Hyde JS. The gender similarity hypothesis. Am Psychol. 2005;60:581-592.
OECD. Literacy Skills for the World of Tomorrow: Further Results from PISA 2000. OECD; 2003.
Wang S, Jiao H, Young MJ, Brooks T, Olson J. Comparability of computer-based and paper-and-pencil testing in K-12 reading assessments: a meta-analysis of testing mode effects. Educ Psychol Meas. 2008;68(1):5-24. doi:10.1177/0013164407305592
Tipton E, Pustejovsky JE, Ahmadi H. Current practices in meta-regression in psychology, education, and medicine. Res Synth Methods. 2019;10(2):180-194. doi:10.1002/jrsm.1339
Belsley DA, Kuh E, Welsch RE. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley & Sons, Inc; 1980. doi:10.1002/0471725153
Viechtbauer W, Cheung MWL. Outlier and influence diagnostics for meta-analysis. Res Synth Methods. 2010;1(2):112-125. doi:10.1002/jrsm.11
Glass GV. Meta-analysis at middle age: a personal history. Res Synth Methods. 2015;6(3):221-231. doi:10.1002/jrsm.1133
Pellegrino JW, Baxter GP, Glaser R. Addressing the “two disciplines” problem: linking theories of cognition and learning with assessment and instructional practice. Rev Res Educ. 1999;24:307-354.
Slagt M, Dubas JS, Deković M, van Aken MAG. Differences in sensitivity to parenting depending on child temperament: a meta-analysis. Psychol Bull. 2016;142(10):1068-1110. doi:10.1037/bul0000061
Oberski D. lavaan.survey: an R package for complex survey analysis of structural equation models. J Stat Softw. 2014;57(1):1-27. doi:10.18637/jss.v057.i01
Singer JD, Braun HI. Testing international education assessments. Science. 2018;360(6384):38-40. doi:10.1126/science.aar4952
Pigott TD, Williams R, Polanin J. Combining individual participant and aggregated data in a meta-analysis with correlational studies. Res Synth Methods. 2012;3(4):257-268. doi:10.1002/jrsm.1051
Johnson BT. Toward a more transparent, rigorous, and generative psychology. Psychol Bull. 2021;147(1):1-15. doi:10.1037/bul0000317
Neimann Rasmussen L, Montgomery P. The prevalence of and factors associated with inclusion of non-English language studies in Campbell systematic reviews: a survey and meta-epidemiological study. Syst Rev. 2018;7(1):129. doi:10.1186/s13643-018-0786-6
Giustini D. Retrieving grey literature, information, and data in the digital age. In: Cooper HM, Hedges LV, Valentine JC, eds. Handbook of Research Synthesis and Meta-Analysis. 3rd ed. Russell Sage Foundation; 2019:101-126.
Barroso C, Ganley CM, McGraw AL, Geer EA, Hart SA, Daucourt MC. A meta-analysis of the relation between math anxiety and math achievement. Psychol Bull. 2021;147(2):134-168. doi:10.1037/bul0000307
Stewart LA, Riley RD, Tierney JF. Planning and initiating an IPD meta-analysis project. In: Riley RD, Tierney JF, Stewart LA, eds. Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research. Wiley Series in Statistics in Practice. Wiley; 2021:21-43.
Ensor J, Burke DL, Snell KIE, Hemming K, Riley RD. Simulation-based power calculations for planning a two-stage individual participant data meta-analysis. BMC Med Res Methodol. 2018;18(1):41. doi:10.1186/s12874-018-0492-z
Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ. 2012;344:d7762. doi:10.1136/bmj.d7762
Stewart LA, Clarke M, Rovers M, et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. Jama. 2015;313(16):1657-1665. doi:10.1001/jama.2015.3656