Detecting publication selection bias through excess statistical significance.
excess statistical significance
meta-analysis
publication selection bias
statistical power
Journal
Research synthesis methods
ISSN: 1759-2887
Titre abrégé: Res Synth Methods
Pays: England
ID NLM: 101543738
Informations de publication
Date de publication:
Nov 2021
Nov 2021
Historique:
revised:
08
05
2021
received:
07
01
2021
accepted:
09
06
2021
pubmed:
2
7
2021
medline:
6
11
2021
entrez:
1
7
2021
Statut:
ppublish
Résumé
We introduce and evaluate three tests for publication selection bias based on excess statistical significance (ESS). The proposed tests incorporate heterogeneity explicitly in the formulas for expected and ESS. We calculate the expected proportion of statistically significant findings in the absence of selective reporting or publication bias based on each study's SE and meta-analysis estimates of the mean and variance of the true-effect distribution. A simple proportion of statistical significance test (PSST) compares the expected to the observed proportion of statistically significant findings. Alternatively, we propose a direct test of excess statistical significance (TESS). We also combine these two tests of excess statistical significance (TESSPSST). Simulations show that these ESS tests often outperform the conventional Egger test for publication selection bias and the three-parameter selection model (3PSM).
Types de publication
Journal Article
Meta-Analysis
Langues
eng
Sous-ensembles de citation
IM
Pagination
776-795Informations de copyright
© 2021 John Wiley & Sons Ltd.
Références
Sterne JA, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.
Doucouliagos H, Paldam M, Stanley TD. Skating on thin evidence: implications for public policy. Eur J Pol Econ. 2018;54:16-25.
Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349(6251):aac4716.
Stanley TD. Limitations of PET-PEESE and other meta-analysis methods. Social Psychol Person Sci. 2017;8:581-591.
Klein RA, Vianello M, Hasselman F, et al. Many Labs 2: investigating variation in replicability across sample and setting. Adv Methods Pract Psychol Sci. 2018;1(4):443-490.
Kvarven A, Strømland E, Johannesson M. Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nat Hum Behav. 2019;4:423-434. https://doi.org/10.1038/s41562-019-0787-z
Stanley TD. Making Meta-Analysis Credible: Supplemental materials. 2019. https://osf.io/eh974/. Accessed June 16, 2021.
Stanley TD. Meta-regression methods for detecting and estimating empirical effect in the presence of publication bias. Oxford Bull Econ Stat. 2008;70:103-127.
Hedges LV. Estimation of effect size under nonrandom sampling: the effects of censoring studies yielding statistically insignificant mean differences. J Educ Behav Stat. 1984;9(1):61-85.
Hedges LV, Vevea JL. Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model. J Educ Behav Stat. 1996;21(4):299-332.
Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. Br Med J. 1997;315:629-634.
Stanley TD. Beyond publication bias. J Econ Surv. 2005;19:309-347.
Ioannidis JPA, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials. 2007;4:245-253.
Iyengar S, Greenhouse JB. Selection models and the file drawer problem. Stat Sci. 1988;3:109-117.
McShane BB, Böckenholt U, Hansen KT. Adjusting for publication bias in meta-analysis an evaluation of selection methods and some cautionary notes. Perspect Psychol Sci. 2016;11(5):730-749.
Carter EC, Schönbrodt FD, Gervais WM, Hilgard J. Correcting for bias in psychology: a comparison of meta-analytic methods. Adv Methods Pract Psychol Sci. 2019;2(2):115-144.
Pustejovsky JE, Rodgers MA. Testing for funnel plot asymmetry of standardized mean differences. Res Synth Methods. 2019;10:57-71.
Moreno SG, Sutton AJ, Ades AE, et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol. 2009;9:2.
Ioannidis JPA, Stanley TD, Doucouliagos C. The power of bias in economics research. Econ J. 2017;127:F236-F265.
Stanley TD, Carter EC, Doucouliagos H. What meta-analyses reveal about the replicability of psychological research. Psychol Bull. 2018;144:1325-1346.
Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. New York: Academic Press; 1988.
Fraley RC, Vazire S. The n-pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One. 2014;9:e109019.
Cohen J. The statistical power of abnormal-social psychological research: a review. J Abnorm Soc Psychol. 1962;65:145-153.
Sedlmeier P, Gigerenzer G. Do studies of statistical power have an effect on the power of studies? Psychol Bull. 1989;105:309-316.
Rossi J. Statistical power of psychological research: what have we gained in 20 years? J Consult Clin Psychol. 1990;58:646-656.
Maxwell SE. The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychol Methods. 2004;9(147):147-163.
Vankov I, Bowers J, Munafò MR. On the persistence of low power in psychological science. Q J Exp Psychol (Hove). 2014;67:1037-1040.
Tressoldi PE, Giofré D. The pervasive avoidance of prospective statistical power: major consequences and practical solutions. Front Psychol. 2015;6:726. https://doi.org/10.3389/fpsyg.2015.00726
Bakker M, Veldkamp CLS, van den Akker OR, et al. Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. PLoS ONE. 2020;15(7):e0236079.
Stanley TD, Doucouliagos H. Neither fixed nor random: weighted least squares meta-analysis. Stat Med. 2015;34:2116-2127.
Stanley TD, Doucouliagos H, JPA I. Finding the power to reduce publication bias. Stat Med. 2017;36:1580-1598.
Ioannidis JPA. Clarifications on the application and interpretation of the test for excess significance and its extensions. J Math Psychol. 2013;57(5):184-187.
Stanley TD, Doucouliagos H. Neither fixed nor random: weighted least squares meta-regression analysis. Res Synth Methods. 2017;8:19-42.
Carter EC, Kofler LE, Forster DF, McCullough ME. A series of meta-analytic tests of the depletion effect: self-control does not seem to rely on a limited resource. J Exp Psychol Gen. 2015;144:796-815.
DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7:177-188.
Augusteijn HEM, van Aert RCM, van Assen MALM. The effect of publication bias on the Q test and assessment of heterogeneity. Psychol Methods. 2019;24(1):116-134.
Cohen J. Things I learned (so far). Am Psychol. 1990;45:1304-1312.
Serlin RA, Lapsley DK. Rational appraisal of psychological research and the good-enough principle. In: Keren G, Lewis C, eds. A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues. Hillsdale, NJ: Erlbaum; 1993.
Schneck A. Examining publication bias: a simulation-based evaluation of statistical tests on publication bias. PeerJ. 2017;5:e4115.
Wooldridge JM. Econometric Analysis of Cross Section and Panel Data. Cambridge: MIT Press; 2002.
Stanley TD, Doucouliagos H. Meta-regression approximations to reduce publication selection bias. Res Synth Methods. 2014;5:60-78.
Johnson N, Kotz S. Distributions in Statistics: Continuous Univariate Distribution. New York: Wiley; 1970.
Greene WE. Econometric Analysis. New York: Macmillan; 1990.
Stanley TD, Doucouliagos H. Meta-Regression Analysis in Economics and Business. Oxford, England: Routledge; 2012.
Coburn KM, Vevea JL. Publication bias as a function of study characteristics. Psychol Methods. 2015;20(3):310-330.
Engber D. Daryl Bem proved ESP is real: which means science is broken. Slate, Cover Story. May 17, 2017. https://slate.com/health-and-science/2017/12/daryl-bem-proved-esp-is-real-showed-science-is-broken.html. Accessed June 16, 2021.
Bem DJ. Precognitive habituation: replicable evidence for a process of anomalous cognition. Paper presented at the Parapsychology Association 46th Annual Convention; August 2-4, 2003; Vancouver, Canada.
Bem DJ. Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. J Pers Soc Psychol. 2011;100:407-425.
Bem DJ, Tressoldi P, Rabeyron T, Duggan M. Feeling the future: a meta-analysis of 90 experiments on the anomalous anticipation of random future events. F1000 Res. 2016;4:1188.
Galak J, Leboeuf RA, Nelson LD, Simmons JP. Correcting the past: failures to replicate. J Pers Soc Psychol. 2012;103(6):933-948.
Hagger MS, Chatzisarantis NLD, Alberts H, et al. A multi-lab preregistered replication of the ego-depletion effect. Perspect Psychol Sci. 2016;11:546-573.
Witte EH, Zenker F. Extending a multilab preregistered replication of the ego-depletion effect to a research program. Basic Appl Soc Psychol. 2017;39:74-80.
Camerer CF, Dreber A, Holzmeister F, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav. 2018;2(9):637-644.
Klein RA, Ratliff KA, Vianello M, et al. Investigating variation in replicability: a “Many Labs” replication project. Soc Psychol. 2014;45:142-152.
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358:252-260.
Rhodes KM, Turner RM, Higgins JPT. Predictive distributions were developed for the extent of heterogeneity in meta-analyses of continuous outcome data. J Clin Epidemiol. 2015;68:52-60.
Jeffreys H. Theory of Probability. 1st ed. Oxford, England: Oxford University Press; 1939.
Amad A, Jardri R, Rousseau C, Larochelle Y, Ioannidis JPA, Naudet F. Excess significance bias in repetitive transcranial magnetic stimulation literature for neuropsychiatric disorders. Psychother Psychosom. 2019;88(6):363-370.
Ioannidis JPA. Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry. 2011;68(8):773-780.
Trinquart L, Ioannidis JP, Chatellier G, Ravaud P. A test for reporting bias in trial networks: simulation and case studies. BMC Med Res Methodol. 2014;14:112.
Sterne JA, Egger M, Smith GD. Systematic reviews in healthcare: investigating and dealing with publication and other biases in meta-analysis. BMJ. 2001;23:101-105.
Borenstein M, Hedges LV, Higgins J, Rothstein H. Introduction to Meta-Analysis. Hoboken, NJ: John Wiley & Sons Inc; 2009.