A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism.
Bayesian analysis
aggregate data
bivariate mixed effect model
replication
reproducibility
vibration of effects
Journal
Research synthesis methods
ISSN: 1759-2887
Titre abrégé: Res Synth Methods
Pays: England
ID NLM: 101543738
Informations de publication
Date de publication:
17 Oct 2023
17 Oct 2023
Historique:
revised:
13
09
2023
received:
14
03
2023
accepted:
23
09
2023
pubmed:
17
10
2023
medline:
17
10
2023
entrez:
17
10
2023
Statut:
aheadofprint
Résumé
The number of meta-analyses of aggregate data has dramatically increased due to the facility of obtaining data from publications and the development of free, easy-to-use, and specialised statistical software. Even when meta-analyses include the same studies, their results may vary owing to different methodological choices. Assessment of the replication of meta-analysis provides an example of the variation of effect 'naturally' observed between multiple research projects. Reproducibility of results has mostly been reported using graphical descriptive representations. A quantitative analysis of such results would enable (i) breakdown of the total observed variability with quantification of the variability generated by the replication process and (ii) identification of which variables account for this variability, such as methodological quality or the statistical analysis procedures used. These variables might explain systematic mean differences between results and dispersion of the results. To quantitatively characterise the reproducibility of meta-analysis results, a bivariate linear mixed-effects model was developed to simulate both mean results and their corresponding uncertainty. Results were assigned to several replication groups, those assessing the same studies, outcomes, treatment indication and comparisons classified in the same replication group. A nested random effect structure was used to break down the total variability within each replication group and between these groups to enable calculation of an intragroup correlation coefficient and quantification of reproducibility. Determinants of variability were investigated by modelling both mean and variance parameters using covariates. The proposed model was applied to the example of meta-analyses evaluating direct oral anticoagulants in the acute treatment of venous thromboembolism.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2023 John Wiley & Sons Ltd.
Références
Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485-514. doi:10.1111/1468-0009.12210
Siontis KC, Ioannidis JPA. Replication, duplication, and waste in a quarter million systematic reviews and meta-analyses. Circ Cardiovasc Qual Outcomes. 2018;11(12):e005212. doi:10.1161/CIRCOUTCOMES.118.005212
Riaz IB, Khan MS, Riaz H, Goldberg RJ. Disorganized systematic reviews and meta-analyses: time to systematize the conduct and publication of these study overviews? Am J Med. 2016;129(3):339.e11-339.e18. doi:10.1016/j.amjmed.2015.10.009
Bolland MJ, Grey A. A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture. PLoS ONE. 2014;9(12):e115934. doi:10.1371/journal.pone.0115934
Ker K, Roberts I. Exploring redundant research into the effect of tranexamic acid on surgical bleeding: further analysis of a systematic review of randomised controlled trials. BMJ Open. 2015;5(8):e009460. doi:10.1136/bmjopen-2015-009460
Sigurdson MK, Khoury MJ, Ioannidis JPA. Redundant meta-analyses are common in genetic epidemiology. J Clin Epidemiol. 2020;127:40-48. doi:10.1016/j.jclinepi.2020.05.035
Zhang H, Tang H, He Q, et al. Surgical versus conservative intervention for acute achilles tendon rupture: a PRISMA-compliant systematic review of overlapping meta-analyses. Medicine (Baltimore). 2015;94(45):e1951. doi:10.1097/MD.0000000000001951
McDonald S, Turner S, Page MJ, Turner T. Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency. J Clin Epidemiol. 2022;146:22-31. doi:10.1016/j.jclinepi.2022.02.006
Bolland MJ, Grey A, Reid IR. Differences in overlapping meta-analyses of vitamin D supplements and falls. J Clin Endocrinol Metab. 2014;99(11):4265-4272. doi:10.1210/jc.2014-2562
Campbell KA, Erickson BJ, Saltzman BM, et al. Is local viscosupplementation injection clinically superior to other therapies in the treatment of osteoarthritis of the knee: a systematic review of overlapping meta-analyses. Arthrosc J Arthrosc Relat Surg. 2015;31(10):2036-2045.e14. doi:10.1016/j.arthro.2015.03.030
Capodanno D. Overlapping meta-analyses of bioresorbable vascular scaffolds versus everolimus-eluting stents: bringing clarity or confusion? J Thorac Dis. 2016;8(7):1366-1370. doi:10.21037/jtd.2016.05.14
Naudet F, Schuit E, Ioannidis JPA. Overlapping network meta-analyses on the same topic: survey of published studies. Int J Epidemiol. 2017;46(6):1999-2008. doi:10.1093/ije/dyx138
Doundoulakis I, Antza C, Apostolidou-Kiouti F, et al. Overview of systematic reviews of non-vitamin K oral anticoagulants in atrial fibrillation. Circ Cardiovasc Qual Outcomes. 2018;11(12):e004769. doi:10.1161/CIRCOUTCOMES.118.004769
Ioannidis JPA. Meta-analyses in environmental and occupational health. Occup Environ Med. 2018;75(6):443-445. doi:10.1136/oemed-2016-104128
Guelimi R, Afach S, Régnaux JP, et al. Overlapping network meta-analyses on psoriasis systemic treatments: an overview, quantity does not make quality. Br J Dermatol. 2021;187(1):29-41. doi:10.1111/bjd.20908
Mendoza JFW, Latorraca C d OC, Oliveira R d Á, et al. Methodological quality and redundancy of systematic reviews that compare endarterectomy versus stenting for carotid stenosis. BMJ Evid-Based Med. 2021;26(1):14-18. doi:10.1136/bmjebm-2018-111151
Ioannidis JPA. Why most discovered true associations are inflated. Epidemiology. 2008;19(5):640-648. doi:10.1097/EDE.0b013e31818131e7
El Bahri M, Wang X, Biaggi T, Falissard B, Naudet F, Barry C. A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects. J Clin Epidemiol. 2022;152:140-150. doi:10.1016/j.jclinepi.2022.09.001
Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis. BMC Med. 2019;17(1):174. doi:10.1186/s12916-019-1409-3
Patel CJ, Burford B, Ioannidis JPA. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J Clin Epidemiol. 2015;68(9):1046-1058. doi:10.1016/j.jclinepi.2015.05.029
Klau S, Hoffmann S, Patel CJ, Ioannidis JP, Boulesteix AL. Examining the robustness of observational associations to model, measurement and sampling uncertainty with the vibration of effects framework. Int J Epidemiol. 2021;50(1):266-278. doi:10.1093/ije/dyaa164
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012
Stan Development Team. RStan: the R interface to Stan. 2022 https://mc-stan.org/