Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study.
Meta-analysis
Multilevel
Multivariate
Outcome-effects
Robust variance estimation method
Journal
Behavior research methods
ISSN: 1554-3528
Titre abrégé: Behav Res Methods
Pays: United States
ID NLM: 101244316
Informations de publication
Date de publication:
04 2021
04 2021
Historique:
pubmed:
19
8
2020
medline:
8
6
2021
entrez:
19
8
2020
Statut:
ppublish
Résumé
In meta-analysis, primary studies often include multiple, dependent effect sizes. Several methods address this dependency, such as the multivariate approach, three-level models, and the robust variance estimation (RVE) method. As for today, most simulation studies that explore the performance of these methods have focused on the estimation of the overall effect size. However, researchers are sometimes interested in obtaining separate effect size estimates for different types of outcomes. A recent simulation study (Park & Beretvas, 2019) has compared the performance of the three-level approach and the RVE method in estimating outcome-specific effects when several effect sizes are reported for different types of outcomes within studies. The goal of this paper is to extend that study by incorporating additional simulation conditions and by exploring the performance of additional models, such as the multivariate model, a three-level model that specifies different study-effects for each type of outcome, a three-level model that specifies a common study-effect for all outcomes, and separate three-level models for each type of outcome. Additionally, we also tested whether the a posteriori application of the RV correction improves the standard error estimates and the 95% confidence intervals. Results show that the application of separate three-level models for each type of outcome is the only approach that consistently gives adequate standard error estimates. Also, the a posteriori application of the RV correction results in correct 95% confidence intervals in all models, even if they are misspecified, meaning that Type I error rate is adequate when the RV correction is implemented.
Identifiants
pubmed: 32808180
doi: 10.3758/s13428-020-01459-4
pii: 10.3758/s13428-020-01459-4
doi:
Types de publication
Journal Article
Meta-Analysis
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
702-717Références
Becker, B. J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & E. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 499–525). Orlando, FL: Academic Press.
doi: 10.1016/B978-012691360-6/50018-5
Bell, R. M., & McCaffrey, D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28(2), 169-181.
Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19, 211–229.
doi: 10.1037/a0032968
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd). Hillsdale, NJ: Erlbaum.
Fernández-Castilla, B., Jamshidi, L., Declercq, L., Beretvas, S. N., Onghena, P., & Van Den Noortgate, W. (2020). The application of meta-analytic models with multiple random effects: A systematic review. Manuscript accepted for publication in Behavior Research Methods.
Fisher, Z., Tipton, E., & Zhipeng, H. (2017). Robumeta: Robust Variance Meta-Regression. R package version 2.0. https://CRAN.R-project.org/package=robumeta
Glass, G. V. (1976). Primary, Secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
doi: 10.3102/0013189X005010003
Gleser, L. J., Olkin, I. (1994). Stochastically dependent effect sizes. In H. Cooper & L. V. Hedges (Eds.). The Handbook of Research Synthesis (pp. 339–355). New York, NY: Russel Sage Foundation.
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.
doi: 10.1002/jrsm.5
Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling. An overview and a meta-analysis. Sociological Methods & Research, 26, 329–367.
doi: 10.1177/0049124198026003003
Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1, 227–235.
doi: 10.1037/1082-989X.1.3.227
Lebuda, I., Zabelina, D. L., & Karwowski, M. (2016). Mind full of ideas: A meta-analysis of the mindfulness–creativity link. Personality and Individual Differences, 93, 22–26.
doi: 10.1016/j.paid.2015.09.040
Lee, S. (2014). Within study dependence in meta-analysis: Comparison of GLS method and multilevel approaches (Doctoral dissertation). Available from ProQuest Dissertations and Theses database (UMI No. 3344745).
Moeyaert, M., Ugille, M., Beretvas, S. N., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017). Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology, 20, 559–572.
doi: 10.1080/13645579.2016.1252189
Owen, K. B., Parker, P. D., Van Zanden, B., MacMillan, F., Astell-Burt, T., & Lonsdale, C. (2016). Physical activity and school engagement in youth: A systematic review and meta-analysis. Educational Psychologist, 51, 129–145.
doi: 10.1080/00461520.2016.1151793
Park, S., & Beretvas, S. N. (2019). Synthesizing effects for multiple outcomes per study using robust variance estimation versus the three-level model. Behavior Research Methods, 51, 152–171.
doi: 10.3758/s13428-018-1156-y
Pustejovsky, J. (2018). clubSandwich: Cluster-Robust (Sandwich) Variance Estimators with Small-Sample Corrections. R package version 0.3.2. https://CRAN.R-project.org/package=clubSandwich
Pustejovsky, J., Tipton, B., & Aloe, A. (2018, July). Combining robust variance estimation with models for dependent effect sizes. Paper presented at the 13
Pustejovsky, J. E. & Tipton, E. (2017). Small sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business and Economic Statistics, 36(4), 672–683.
doi: 10.1080/07350015.2016.1247004
Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111–120.
doi: 10.1037/0033-2909.103.1.111
Raudenbush, S. W., & Bryk, A. S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75–98.
doi: 10.3102/10769986010002075
Spruit, A., Assink, M., van Vugt, E., van der Put, C., & Stams, G. J. (2016). The effects of physical activity interventions on psychosocial outcomes in adolescents: A meta-analytic review. Clinical Psychology Review, 45, 56–71.
doi: 10.1016/j.cpr.2016.03.006
Tipton, E. (2013). Robust variance estimation in meta-regression with binary dependent effects. Research Synthesis Methods, 4, 169–187.
doi: 10.1002/jrsm.1070
Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20, 375–393.
doi: 10.1037/met0000011
Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10, 161–179.
doi: 10.1002/jrsm.1338
Team, R. C. (2012). R: A language and environment for statistical computing. 2012. Vienna, Austria: R Foundation for Statistical Computing, 10.
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45, 576–594.
doi: 10.3758/s13428-012-0261-6
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47, 1274–1294.
doi: 10.3758/s13428-014-0527-2
Viechtbauer, W. (2010). Conducting Meta-Analyses in R with the metafor Package. Journal of Statistical Software, 36, 1–48.
doi: 10.18637/jss.v036.i03