Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses - the METSA systematic review.
Meta-analysis
Methodology
Research-on-research
Systematic review
Trial sequential analysis
Journal
BMC medical research methodology
ISSN: 1471-2288
Titre abrégé: BMC Med Res Methodol
Pays: England
ID NLM: 100968545
Informations de publication
Date de publication:
09 Sep 2024
09 Sep 2024
Historique:
received:
20
09
2023
accepted:
21
08
2024
medline:
10
9
2024
pubmed:
10
9
2024
entrez:
9
9
2024
Statut:
epublish
Résumé
Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis. We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision. We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%). The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity. PROSPERO CRD42021273811.
Sections du résumé
BACKGROUND
BACKGROUND
Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis.
METHODS
METHODS
We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision.
RESULTS
RESULTS
We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%).
CONCLUSIONS
CONCLUSIONS
The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity.
STUDY REGISTRATION
BACKGROUND
PROSPERO CRD42021273811.
Identifiants
pubmed: 39251912
doi: 10.1186/s12874-024-02318-y
pii: 10.1186/s12874-024-02318-y
doi:
Types de publication
Journal Article
Systematic Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
196Informations de copyright
© 2024. The Author(s).
Références
Higgins J, Thomas J, Chandler J et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane. 2022. www.training.cochrane.org/handbook
Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94:485–514.
doi: 10.1111/1468-0009.12210
pubmed: 27620683
pmcid: 5020151
Gluud C. Testosterone and alcoholic cirrhosis. Epidemiologic, pathophysiologic and therapeutic studies in men. Dan Med Bull. 1988;35:564–75.
pubmed: 3064977
Fontelo P, Liu F. A review of recent publication trends from top publishing countries. Syst Rev. 2018;7:1–9.
doi: 10.1186/s13643-018-0819-1
Zhao X, Jiang H, Yin J, et al. Changing trends in clinical research literature on PubMed database from 1991 to 2020. Eur J Med Res. 2022;27:1–11.
doi: 10.1186/s40001-022-00717-9
Ioannidis JPA, Stuart ME, Brownlee S, et al. How to survive the medical misinformation mess. Eur J Clin Invest. 2017;47:795–802.
doi: 10.1111/eci.12834
pubmed: 28881000
Garattini S, Jakobsen JC, Wetterslev J, et al. Evidence-based clinical practice: overview of threats to the validity of evidence and how to minimise them. Eur J Intern Med. 2016;32:13–21.
doi: 10.1016/j.ejim.2016.03.020
pubmed: 27160381
Brok J, Thorlund K, Wetterslev J, et al. Apparently conclusive meta-analyses may be inconclusive–trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses. Int J Epidemiol. 2009;38:287–98.
doi: 10.1093/ije/dyn188
pubmed: 18824466
Imberger G, Gluud C, Boylan J, et al. Systematic reviews of anesthesiologic interventions reported as statistically significant: problems with power, precision, and type 1 error protection. Anesth Analg. 2015;121:1611–22.
doi: 10.1213/ANE.0000000000000892
pubmed: 26579662
Turner RM, Bird SM, Higgins JPTT. The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS ONE. 2013;8:e59202.
doi: 10.1371/journal.pone.0059202
pubmed: 23544056
pmcid: 3609745
Thorlund K, Engstrøm J, Wetterslev J et al. User manual for Trial Sequential Analysis (TSA). Copenhagen Trial Unit, Centre for Clinical Intervention Research. www.ctu.dk/tsa/files/tsa_manual.pdf
Thorlund K, Imberger G, Walsh M, et al. The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis–a simulation study. PLoS ONE. 2011;6. https://doi.org/10.1371/JOURNAL.PONE.0025491 .
Garcia-Alamino JM, Bankhead C, Heneghan C, et al. Impact of heterogeneity and effect size on the estimation of the optimal information size: analysis of recently published meta-analyses. BMJ Open. 2017;7. https://doi.org/10.1136/BMJOPEN-2017-015888 .
Imberger G, Thorlund K, Gluud C, et al. False-positive findings in Cochrane meta-analyses with and without application of trial sequential analysis: an empirical review. BMJ Open. 2016;6:e011890.
doi: 10.1136/bmjopen-2016-011890
pubmed: 27519923
pmcid: 4985805
Gluud C, Wetterslev J, Higgins J et al. Trial Sequential Analysis or sequential meta-analysis. Cochrane Sci Comm. 2017. https://methods.cochrane.org/sites/default/files/public/uploads/2017_1_18_may_scientific_committee_agenda_docs.pdf (accessed 15 May 2021).
Wetterslev J, Thorlund K, Brok J, et al. Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2008;61:64–75.
doi: 10.1016/j.jclinepi.2007.03.013
pubmed: 18083463
Brok J, Thorlund K, Gluud C, et al. Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. J Clin Epidemiol. 2008;61:763–9.
doi: 10.1016/j.jclinepi.2007.10.007
pubmed: 18411040
Simmonds M, Salanti G, McKenzie J, et al. Living systematic reviews: 3. Statistical methods for updating meta-analyses. J Clin Epidemiol. 2017;91:38–46.
doi: 10.1016/j.jclinepi.2017.08.008
pubmed: 28912004
Wetterslev J, Thorlund K, Brok J, et al. Estimating required information size by quantifying diversity in random-effects model meta-analyses. BMC Med Res Methodol. 2009;9. https://doi.org/10.1186/1471-2288-9-86 .
Payne T, Moran B, Loadsman J, et al. Importance of sequential methods in meta-analysis: implications for postoperative mortality, delirium, and stroke management. Br J Anaesth. 2023;130:395–401.
doi: 10.1016/j.bja.2023.01.011
pubmed: 36931783
Ioannidis JPA, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383:166–75.
doi: 10.1016/S0140-6736(13)62227-8
pubmed: 24411645
pmcid: 4697939
Kulinskaya E, Huggins R, Dogo SH. Sequential biases in accumulating evidence. Res Synth Methods. 2016;7:294–305.
doi: 10.1002/jrsm.1185
pubmed: 26626562
Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
doi: 10.1136/bmj.c332
pubmed: 20332509
pmcid: 2844940
Thomas J, Askie L, Berlin J et al. Chapter 22: Prospective approaches to accumulating evidence. In: Higgins J, Thomas J, Chandle J, eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). 2022.
Wetterslev J, Jakobsen JC, Gluud C. Trial Sequential Analysis in systematic reviews with meta-analysis. BMC Med Res Methodol. 2017;17:1–18.
doi: 10.1186/s12874-017-0315-7
Riberholt CG, Olsen MH, Milan JB, et al. Major mistakes and errors in the use of Trial Sequential Analysis in systematic reviews or meta-analyses – protocol for a systematic review. Syst Rev. 2022;11:114.
doi: 10.1186/s13643-022-01987-4
pubmed: 35659769
pmcid: 9166392
Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372. https://doi.org/10.1136/BMJ.N71 .
Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA Statement for reporting literature searches in systematic reviews. Syst Rev. 2021;10. https://doi.org/10.1186/S13643-020-01542-Z .
Covidence©. Covidence systematic review software [Computer program]. www.covidence.org.
Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inf. 2009;42:377–81.
doi: 10.1016/j.jbi.2008.08.010
Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inf. 2019;95:103208.
doi: 10.1016/j.jbi.2019.103208
Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Online). 2017;358:4008.
Schünemann HJ, Brozek J, Guyatt G et al. GRADE handbook for grading quality of evidence and strength of recommendations. 2013. https://gdt.gradepro.org/app/handbook/handbook.html#h.ygojbnr1bi5y (accessed 27 April 2022).
Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:4008.
doi: 10.1136/bmj.j4008
Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383:156–65.
doi: 10.1016/S0140-6736(13)62229-1
pubmed: 24411644
Al-Shahi Salman R, Beller E, Kagan J, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383:176–85.
doi: 10.1016/S0140-6736(13)62297-7
pubmed: 24411646
Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383:257–66.
doi: 10.1016/S0140-6736(13)62296-5
pubmed: 24411650
pmcid: 4533904
Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383:267–76.
doi: 10.1016/S0140-6736(13)62228-X
pubmed: 24411647
Hopewell S, Boutron I, Altman DG, et al. Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013;3. https://doi.org/10.1136/BMJOPEN-2013-003342 .
Useem J, Brennan A, LaValley M, et al. Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis. PLoS ONE. 2015;10. https://doi.org/10.1371/JOURNAL.PONE.0144980 .
Collier A, Heilig L, Schilling L, et al. Cochrane Skin Group systematic reviews are more methodologically rigorous than other systematic reviews in dermatology. Br J Dermatol. 2006;155:1230–5.
doi: 10.1111/j.1365-2133.2006.07496.x
pubmed: 17107394
Windsor B, Popovich I, Jordan V, et al. Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologies. Hum Reprod. 2012;27:3460–6.
doi: 10.1093/humrep/des342
pubmed: 23034152
Moore A, Fisher E, Eccleston C. Flawed, futile, and fabricated-features that limit confidence in clinical research in pain and anaesthesia: a narrative review. Br J Anaesth. 2023;130. https://doi.org/10.1016/J.BJA.2022.09.030 .
Moher D, Tetzlaff J, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4:447–55.
doi: 10.1371/journal.pmed.0040078
Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13. https://doi.org/10.1371/JOURNAL.PMED.1002028 .
Moher D, Shamseer L, Clarke M et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. 2015. https://doi.org/10.1186/2046-4053-4-1
Nguyen PY, Kanukula R, McKenzie JE, et al. Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: cross sectional meta-research study. BMJ. 2022;379. https://doi.org/10.1136/BMJ-2022-072428 .
Charles P, Giraudeau B, Dechartres A, et al. Reporting of sample size calculation in randomised controlled trials: review. BMJ. 2009;338:1256.
doi: 10.1136/bmj.b1732
Jakobsen JC, Wetterslev J, Winkel P, et al. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14. https://doi.org/10.1186/1471-2288-14-120 .
Imberger G, Damgaard Vejlby A, Hansen SB, et al. Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLoS ONE. 2011;6. https://doi.org/10.1371/JOURNAL.PONE.0028422 .
Claire R, Gluud C, Berlin I, et al. Using Trial Sequential Analysis for estimating the sample sizes of further trials: example using smoking cessation intervention. BMC Med Res Methodol. 2020;20:1–10.
doi: 10.1186/s12874-020-01169-7
Wetterslev J, Jakobsen JC, Gluud C. Trial Sequential Analysis in systematic reviews with meta-analysis. BMC Med Res Methodol. 2017;17. https://doi.org/10.1186/S12874-017-0315-7 .
Pereira TV, Horwitz RI, Ioannidis JPA. Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012;308:1676–84.
doi: 10.1001/jama.2012.13444
pubmed: 23093165
Korang SK, von Rohden E, Veroniki AA, et al. Vaccines to prevent COVID-19: a living systematic review with Trial Sequential Analysis and network meta-analysis of randomized clinical trials. PLoS ONE. 2022;17. https://doi.org/10.1371/JOURNAL.PONE.0260733 .
Sidebotham D, Popovich I, Lumley T. A Bayesian analysis of mortality outcomes in multicentre clinical trials in critical care. Br J Anaesth. 2021;127:487–94.
doi: 10.1016/j.bja.2021.06.026
pubmed: 34275603
Carrasco-Labra A, Devji T, Qasim A, et al. Serious reporting deficiencies exist in minimal important difference studies: current state and suggestions for improvement. J Clin Epidemiol. 2022;150:25–32.
doi: 10.1016/j.jclinepi.2022.06.010
pubmed: 35760237
Carrasco-Labra A, Devji T, Qasim A, et al. Minimal important difference estimates for patient-reported outcomes: a systematic survey. J Clin Epidemiol. 2021;133:61–71.
doi: 10.1016/j.jclinepi.2020.11.024
pubmed: 33321175
Zeng L, Brignardello-Petersen R, Hultcrantz M, et al. GRADE Guidance 34: update on rating imprecision using a minimally contextualized approach. J Clin Epidemiol. 2022;0. https://doi.org/10.1016/j.jclinepi.2022.07.014 .
Schünemann HJ, Neumann I, Hultcrantz M, et al. GRADE guidance 35: update on rating imprecision for assessing contextualized certainty of evidence and making decisions. J Clin Epidemiol. 2022;150:225–42.
doi: 10.1016/j.jclinepi.2022.07.015
pubmed: 35934266
Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence --imprecision. J Clin Epidemiol. 2011;64:1283–93.
doi: 10.1016/j.jclinepi.2011.01.012
pubmed: 21839614
Nguyen VT, Jung K, Gupta V. Examining data visualization pitfalls in scientific publications. Vis Comput Ind Biomed Art. 2021;4. https://doi.org/10.1186/S42492-021-00092-Y .
CRAN - Package RTSA. https://cran.r-project.org/web/packages/RTSA/index.html (accessed 18 June 2023).
Sørensen AL, Olsen MH, Lange T et al. Prospective and retrospective sequential meta-analysis. https://cran.r-project.org/web/packages/RTSA/vignettes/prospective_retrospective.html (accessed 27 December 2023).
Goh ET, Stokes CS, Sidhu SS et al. L-ornithine L-aspartate for prevention and treatment of hepatic encephalopathy in people with cirrhosis. Cochrane Database of Systematic Reviews. 2018;2018. https://doi.org/10.1002/14651858.CD012410.PUB2/MEDIA/CDSR/CD012410/IMAGE_N/NCD012410-CMP-004-04.PNG
Stokes CS, Goh ET, Vilstrup H et al. L-ornithine L-aspartate for people with cirrhosis and hepatic encephalopathy. Cochrane Database of Systematic Reviews. 2016;2016. https://doi.org/10.1002/14651858.CD012410/INFORMATION/EN