Capturing the underlying distribution in meta-analysis: Credibility and tolerance intervals.
Monte Carlo simulation
credibility interval
prediction interval
standardized mean difference
tolerance interval
Journal
Research synthesis methods
ISSN: 1759-2887
Titre abrégé: Res Synth Methods
Pays: England
ID NLM: 101543738
Informations de publication
Date de publication:
May 2021
May 2021
Historique:
revised:
12
11
2020
received:
17
04
2020
accepted:
28
01
2021
pubmed:
6
2
2021
medline:
29
10
2021
entrez:
5
2
2021
Statut:
ppublish
Résumé
Tolerance intervals provide a bracket intended to contain a percentage (e.g., 80%) of a population distribution given sample estimates of the mean and variance. In random-effects meta-analysis, tolerance intervals should contain researcher-specified proportions of underlying population effect sizes. Using Monte Carlo simulation, we investigated the coverage for five relevant tolerance interval estimators: the Schmidt-Hunter credibility intervals, a prediction interval, two content tolerance intervals adapted to meta-analysis, and a bootstrap tolerance interval. None of the intervals contained the desired percentage of coverage at the nominal rates in all conditions. However, the prediction worked well unless the number of primary studies was small (<30), and one of the content tolerance intervals approached nominal levels with small numbers (<20) of primary studies. The bootstrap tolerance interval achieved near nominal coverage if there were sufficient numbers of primary studies (30+) and large enough sample sizes (N ≅ 70) in the included primary studies, although it slightly exceeded nominal coverage with large numbers of large-sample primary studies. Next, we showed the results of applying the intervals to real data using a set of previously published analyses and provided suggestions for practice. Tolerance intervals incorporate error of estimation into the construction of proper brackets for fractions of population true effects. In many contexts, such intervals approach the desired nominal levels of coverage.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
264-290Informations de copyright
© 2021 John Wiley & Sons, Ltd.
Références
Schmidt FL, Hunter JE. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. 3rd ed. Thousand Oaks, CA: Sage; 2015.
Hedges LV, Vevea JL. Fixed- and random-effects models in meta-analysis. Psychol Methods. 1998;3:486-504.
Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to Meta-Analysis. West Sussex, UK: Wiley; 2009.
Aytug ZG, Rothstein HR, Zhou W, Kern MC. Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organ Res Methods. 2012;15(1):103-133.
Carlson KD, Ji FX. Citing and building on meta-analytic findings: a review and recommendations. Organ Res Methods. 2011;14(4):696-717.
Paterson TA, Harms PD, Steel P, Credé M. An assessment of the magnitude of effect sizes: evidence from 30 years of meta-analysis in management. J Leadership Organ Stud. 2016;23(1):66-81.
Bosco FA, Aguinis H, Singh K, Field JG, Pierce CA. Correlational effect size benchmarks. J Appl Psychol. 2015;100:431-449.
IntHout J, Ioannidis JPA, Rovers MM, Goeman JJ. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open. 2016;6(7):e010247.
Vardeman SB. What about the other intervals? Am Stat. 1992;46(3):193-197.
Hahn GJ, Meeker WQ. Statistical Intervals: A Guide for Practitioners. New York, NY: John Wiley & Sons; 1991.
Millsap RE. Tolerance intervals: alternatives to credibility intervals in validity generalization research. Appl Psychol Measur. 1988;12(1):27-32.
Mee RW. β-expectation and β-content tolerance limits for balanced one-way ANOVA random model. Dent Tech. 1984;26(3):251-254.
Lin T-Y, Liao C-T. A β-expectation tolerance interval for general balanced mixed linear models. Comput Stat Data Anal. 2006;50(4):911-925.
Hoffman D, Kringle R. Two-sided tolerance intervals for balanced and unbalanced random effects models. J Biopharm Stat. 2005;15:283-293.
Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc: Ser A (Stat Soc). 2009;172:137-159.
Chiolero A, Santschi V, Burnand B, Platt RW, Paradis G. Meta-analyses: with confidence or prediction intervals? Eur J Epidemiol. 2012;27:823-825.
Proschan F. Confidence and tolerance intervals for the normal distribution. J Am Stat Assoc. 1953;48(263):550-564.
Whitener EM. Confusion of confidence intervals and credibility intervals in meta-analysis. J Appl Psychol. 1990;75:315-321.
Young DS. Tolerance: an R package for estimating tolerance intervals. J Stat Software. 2010;36(5):1-39.
Howe WG. Two-sided tolerance limits for normal populations, some improvements. J Am Stat Assoc. 1969;64(326):610-629.
Beckman RJ, Tietjen GL. Two-sided tolerance limits for balanced random-effects ANOVA models. Technometrics. 1989;31(2):185-197.
Burdick RK, Quiroz J, Iyer HK. The present status of confidence interval estimation for one-factor random models. J Stat Plann Inference. 2006;136:4307-4325.
Mee RW. Tolerance limits and bounds for proportions based on data subject to measurement error. J Qual Technol. 1984;16(2):74-80.
Jaech JL. Removing the effects of measurement errors in constructing statistical tolerance intervals. J Qual Technol. 1984;16(2):69-73.
Bhaumik DK, Kulkarni PM. A simple and exact method of constructing tolerance intervals for one-way ANOVA with random effects. Am Stat. 1996;50(4):319-323.
Mee RW, Owen DB. Improved factors for one-sided tolerance limits for balanced one-way ANOVA random model. J Am Stat Assoc. 1983;78(384):901-905.
R: A language and environment for statistical computing [computer program]. Vienna, Austria: R Foundation for Statistical Computing; 2016.
Bennett AA, Bakker AB, Field JG. Recovery from work-related effort: a meta-analysis. J Organ Behav. 2018;39:262-275.
Brannick MT, Yang L, Cafri G. Comparison of weights for meta-analysis of r and d under realistic conditions. Organ Res Methods. 2011;14(4):587-607.
Veroniki AA, Jackson D, Viecehtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods. 2016;7:55-79.
Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3):1-48.
Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Dordrecht, Germany: Springer Science+Business Media; 1993.
Li JC, Cui Y, Chan W. Bootstrap confidence intervals for the mean correlation corrected for case IV range restriction: a more adequate procedure for meta-analysis. J Appl Psychol. 2013;98(1):183-193.
Sanchez-Meca J, Marin-Martinez F. Confidence intervals for the overall effect size in random-effects meta-analysis. Psychol Methods. 2008;13(1):31-48.
Hill CJ, Bloom HJ, Black AR, Lipsey MW. Empirical benchmarks for interpreting effect sizes in research. Child Dev Perspect. 2008;2(3):172-177.
Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates; 1988.
Schulze R. Current methods for meta-analysis: approaches, issues and developments. J Psychol. 2007;215:90-103.
Baltes BB, Brigs TE, Huff JW, Wright JA, Neuman GA. Flexible and compressed workweek schedules: a meta-analysis of their effects on work-related criteria. J Appl Psychol. 1999;84(4):496-513.
Dwight SA, Donovan JJ. Do warnings not to fake reduce faking? Hum Perform. 2003;16(1):1-23.
Stewart WH, Roth PL. Risk propensity differences between entrepeneurs and managers: a meta-analytic review. J Appl Psychol. 2001;86(1):145-153.
Kierein NM, Gold MA. Pygmalion in work organizations: a meta-analysis. J Organ Behav. 2000;21:913-928.
Hughes AM, Gregory ME, Joseph DL, et al. Saving lives: a meta-analysis of team training in healthcare. J Appl Psychol. 2016;101(9):1266-1304.
Berry CM, Cullen MJ, Meyer JM. Racial/ethnic subgroup differences in cognitive ability test range restriction: implications for differential validity. J Appl Psychol. 2014;99(1):21-37.
Ben-Shakhar G, Elaad E. The validity of psychophysiological detection of information with the guilty knowledge test: a meta-analytic review. J Appl Psychol. 2003;88(1):131-151.
Hosoda M, Stone-Romero EF, Coats G. The effects of physical attractiveness on job-related outcomes: a meta-analysis of experimental studies. Pers Psychol. 2003;56(2):431-462.
Richman WL, Kiesler S, Weisband S, Drasgow F. A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires and interviews. J Appl Psychol. 1999;84(5):754-775.
Nguyen HD, Ryan AM. Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence. J Appl Psychol. 2008;93(6):1314-1334.
Koch AJ, D'Mello SD, Sackett PR. A meta-analysis of gender stereotypes and bias in experimental simulation of employment decision making. J Appl Psychol. 2015;100(1):128-161.
Badura KL, Grijalva E, Newman DA, Yan TT, Jeon G. Gender and leadership emergence: a meta-analysis and explanatory model. Pers Psychol. 2018;71:335-367.
Courtright SH, McCormick BW, Postlethwaite BE, Reeves CJ, Mount MK. A meta-analysis of sex differences in physical ability: revised estimates and strategies for reducing differences n selection contexts. J Appl Psychol. 2013;98(4):623-641.
Riggle RJ, Edmondson DR, Hansen JD. A meta-analysis of the relationship between perceived organizational support and job outcomes: 20 years of research. J Bus Res. 2009;62:1027-1030.
Cawley BD, Keeping LM, Levy PE. Participation in the performance appriasal process and employee reactions: a meta-analytic review of field investigations. J Appl Psychol. 1998;83(4):615-633.
Quinones MA, Ford JK, Teachout MS. The relationship between work experience and job performance: a conceptual and meta-analytic review. Pers Psychol. 1995;48(4):887-910.
Taris TW. Is there a relationship between burnout and objective performance? A critical review of 16 studies. Work Stress. 2006;20(4):316-334.
Tosi HL, Werner S, Katz JP, Gomez-Mejia LR. How much does performance matter? A meta-analysis of CEO pay studies. J Manage. 2000;26(2):301-339.
Wagner JA III, Stimpert JL, Fubara EI. Board composition and organizational performance: two studies of insider/outsider effects. J Manage Stud. 1998;35(5):655-677.
Webber SS, Donahue LM. Impact of highly and less job-related diversity on work group cohesion and performance: a meta-analysis. J Manage. 2001;27(2):141-162.
Morris SB, Daisley RL, Wheeler M, Boyer P. A meta-analysis of the relationship between individual assessments and job performance. J Appl Psychol. 2015;100(1):5-20.
Longmire NH, Harrison DA. Seeing their side versus feeling their pain: differential consequences of perspective-taking and empathy at work. J Appl Psychol. 2018;103(8):894-915.
Kooij DTAM, Kanfer R, Betts M, Rudolph CW. Future time perspective: a systematic review and meta-analysis. J Appl Psychol. 2018;103(8):867-893.
Wright TA, Bonett DG. The moderating effects of employee tenure on the relation between organizational commitment and job performance: a meta-analysis. J Appl Psychol. 2002;87(6):1183-1190.
Young HR, Glerum DR, Wang W, Joseph DL. Who are the most engaged at work? A meta-analysis of personality and employee engagement. J Organ Behav. 2018;39:1330-1346.
Kossek EE, Ozeki C. Work-family conflict, policies, and the job-life satisfaction relationship: a review and directions for organizational behavior-human resources research. J Appl Psychol. 1998;83(2):139-149.
Wang G, Van Iddekinge CH, Bishoff J, Zhang L. Meta-analytic and primary investigations of the role of followers in ratings of leadership behavior in organizations. J Appl Psychol. 2019;104(1):70-106.
Shockley KM, Shen W, DeNunzio MM, Arvan ML, Knudsen EA. Disentangling the relationship betwen gender and work-family conflict: an integration of theoretical perspectives using meta-analytic methods. J Appl Psychol. 2017;102(12):1601-1635.
Krishnamoorthy K, Mathew T. One-sided tolerance limits in balanced and unbalanced one-way random models based on generalized confidence intervals. Technometrics. 2004;46(1):44-52.
Brannick MT, Potter SM, Teng Y. Quantifying uncertainty in the meta-analytic lower bound estimate. Psychol Methods. 2019;24(6):754-773.
Borenstein M. Common Mistakes in Meta-Analysis and how to Avoid them. New York, NY: Biostat; 2019.
Schmidt FL, Hunter JE. Development of a general solution to the problem of validity generalization. J Appl Psychol. 1977;62(5):529-540.
Hoijtink H, Chow S. Bayesian hypothesis testing: editorial to the special issue on Bayesian data analysis. Psychol Methods. 2017;22(2):211-216.
Rhodes KM, Turner RM, Higgins JPT. Predictive distributions were developed for the extent of heterogeneity in meta-analyses of continuous outcome data. J Clin Epidemiol. 2015;68:52-60.
Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JPT. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane database of systematic reviews. Int J Epidemiol. 2012;41:818-827.
Turner RM, Jackson D, Wei Y, Thompson SG, Higgins JPT. Predictive distributions for between-study heterogeneity and simple methods for their application in Bayesian meta-analysis. Stat Med. 2015;34:984-998.
Rover C, Bender R, Dias S, et al. On weakly informative prior distributions for the heterogeneity parameter in Bayesian random-effects meta-analysis. Res Synth Methods 2021 Jan 24. https://doi.org/10.1002/jrsm.1475. Online ahead of print.