Inconsistencies in the Methodological Framework Throughout Published Studies in High-Impact Orthopaedic Journals: A Systematic Review.
Journal
The Journal of bone and joint surgery. American volume
ISSN: 1535-1386
Titre abrégé: J Bone Joint Surg Am
Pays: United States
ID NLM: 0014030
Informations de publication
Date de publication:
19 01 2022
19 01 2022
Historique:
pubmed:
15
10
2021
medline:
12
2
2022
entrez:
14
10
2021
Statut:
ppublish
Résumé
Both the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) and Consolidated Standards of Reporting Trials (CONSORT) guidelines recommend that clinical trials follow a study framework that aligns with their objective to test the relative efficacy or safety (equality) or effectiveness (superiority, noninferiority, or equivalence) between interventions. We conducted a systematic review to assess the proportion of studies that demonstrated inconsistency between the framing of their research question, sample size calculation, and conclusion and those that should have framed their research question differently based on the compared interventions. We included studies from 5 high-impact-factor orthopaedic journals published in 2017 and 2019 that compared at least 2 interventions using patient-reported outcome measures. We included 228 studies. The sample size calculation was reported in 60.5% (n = 138) of studies. Of these, 52.2% (n = 72) were inconsistent between the framing of their research question, sample size calculation, and conclusion. The majority (n = 137) of sample size calculations were for equality, but 43.8% of these studies concluded superiority, noninferiority, or equivalence. Studies that framed their research question as equality (n = 186) should have been framed as superiority (n = 129), equivalence (n = 52), or noninferiority (n = 3). Only 2 studies correctly framed their research question as equality. Studies published in high-impact journals were inconsistent between the framing of their research question, sample size calculation, and conclusion. Authors may be misinterpreting research findings and making clinical recommendations solely based on p values. Researchers are encouraged to state and justify their methodological framework and choice of margin(s) in a publicly published protocol as they have implications for sample size and the applicability of conclusions. The results of clinical research must be interpreted using confidence intervals, with careful consideration as to how the confidence intervals relate to clinically meaningful differences in outcomes between treatments. The more typical practice of relying on p values leaves the clinician at high risk of erroneous interpretation, recommendation, and/or action.
Sections du résumé
BACKGROUND
Both the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) and Consolidated Standards of Reporting Trials (CONSORT) guidelines recommend that clinical trials follow a study framework that aligns with their objective to test the relative efficacy or safety (equality) or effectiveness (superiority, noninferiority, or equivalence) between interventions. We conducted a systematic review to assess the proportion of studies that demonstrated inconsistency between the framing of their research question, sample size calculation, and conclusion and those that should have framed their research question differently based on the compared interventions.
METHODS
We included studies from 5 high-impact-factor orthopaedic journals published in 2017 and 2019 that compared at least 2 interventions using patient-reported outcome measures.
RESULTS
We included 228 studies. The sample size calculation was reported in 60.5% (n = 138) of studies. Of these, 52.2% (n = 72) were inconsistent between the framing of their research question, sample size calculation, and conclusion. The majority (n = 137) of sample size calculations were for equality, but 43.8% of these studies concluded superiority, noninferiority, or equivalence. Studies that framed their research question as equality (n = 186) should have been framed as superiority (n = 129), equivalence (n = 52), or noninferiority (n = 3). Only 2 studies correctly framed their research question as equality.
CONCLUSIONS
Studies published in high-impact journals were inconsistent between the framing of their research question, sample size calculation, and conclusion. Authors may be misinterpreting research findings and making clinical recommendations solely based on p values. Researchers are encouraged to state and justify their methodological framework and choice of margin(s) in a publicly published protocol as they have implications for sample size and the applicability of conclusions.
CLINICAL RELEVANCE
The results of clinical research must be interpreted using confidence intervals, with careful consideration as to how the confidence intervals relate to clinically meaningful differences in outcomes between treatments. The more typical practice of relying on p values leaves the clinician at high risk of erroneous interpretation, recommendation, and/or action.
Identifiants
pubmed: 34648473
doi: 10.2106/JBJS.21.00116
pii: 00004623-202201190-00011
doi:
Types de publication
Journal Article
Systematic Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
181-188Informations de copyright
Copyright © 2021 by The Journal of Bone and Joint Surgery, Incorporated.
Déclaration de conflit d'intérêts
Disclosure: The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article (http://links.lww.com/JBJS/G744).
Références
Chan AW, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, Berlin JA, Dickersin K, Hróbjartsson A, Schulz KF, Parulekar WR, Krleza-Jeric K, Laupacis A, Moher D. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013 Jan 8;346:e7586.
Chow SC, Shao J, Wang H. Sample Size Calculations in Clinical Research. 1st ed. Marcel Dekker; 2003. [Ebook].
Dunn DT, Copas AJ, Brocklehurst P. Superiority and non-inferiority: two sides of the same coin? Trials. 2018 Sep 17;19(1):499.
Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJW; CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006 Mar 8;295(10):1152-60.
Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016 Apr;31(4):337-50.
Kamper SJ. Confidence intervals: linking evidence to practice. J Orthop Sports Phys Ther. 2019 Oct;49(10):763-4.
Shafiq N, Malhotra S. Superiority trials: raising the bar of null hypothesis statistical testing. Evid Based Med. 2015 Oct;20(5):154-5.
Bigirumurame T, Kasim AS. Can testing clinical significance reduce false positive rates in randomized controlled trials? A snap review. BMC Res Notes. 2017 Dec 28;10(1):775.
Althunian TA, de Boer A, Groenwold RHH, Klungel OH. Defining the noninferiority margin and analysing noninferiority: an overview. Br J Clin Pharmacol. 2017 Aug;83(8):1636-42.
Rehal S, Morris TP, Fielding K, Carpenter JR, Phillips PPJ. Non-inferiority trials: are they inferior? A systematic review of reporting in major medical journals. BMJ Open. 2016 Oct 7;6(10):e012594.
Paesmans M, Grigoriu B, Ocak S, Roelandts M, Lafitte JJ, Holbrechts S, Berghmans T, Meert AP, Moretti L, Danyi S, Pasleau F, Ameye L, Van Meerhaeghe A, Sculier JP. Systematic qualitative review of randomised trials conducted in nonsmall cell lung cancer with a noninferiority or equivalence design. Eur Respir J. 2015 Feb;45(2):511-24.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration; 2011. Accessed 2018 Oct 30. www.handbook.cochrane.org
Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg. 2010;8(5):336-41.
Clarivate. InCites Journal Citation Report - Web of Science Group. 2017. https://jcr.clarivate.com/JCRJournalHomeAction.action?
McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-82.
Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015 May 8;350:h2147.
Yazici Y, McAlindon TE, Fleischmann R, Gibofsky A, Lane NE, Kivitz AJ, Skrepnik N, Armas E, Swearingen CJ, DiFrancesco A, Tambiah JRS, Hood J, Hochberg MC. A novel Wnt pathway inhibitor, SM04690, for the treatment of moderate to severe osteoarthritis of the knee: results of a 24-week, randomized, controlled, phase 1 study. Osteoarthritis Cartilage. 2017 Oct;25(10):1598-606.
Jo CH, Chai JW, Jeong EC, Oh S, Shin JS, Shim H, Yoon KS. Intra-articular injection of mesenchymal stem cells for the treatment of osteoarthritis of the knee: a 2-year follow-up study. Am J Sports Med. 2017 Oct;45(12):2774-83. Epub 2017 Jul 26.
Brooks D, Stratford P. Pilot studies and their suitability for publication in physiotherapy Canada. Physiother Can. 2009 Spring;61(2):66-7.
Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009 Jun 3;10:37.
Wangge G, Roes KCB, de Boer A, Hoes AW, Knol MJ. The challenges of determining noninferiority margins: a case study of noninferiority randomized controlled trials of novel oral anticoagulants. CMAJ. 2013 Feb 19;185(3):222-7.
Greene WL, Concato J, Feinstein AR. Claims of equivalence in medical research: are they supported by the evidence? Ann Intern Med. 2000 May 2;132(9):715-22.
Bland JM, Altman DG. Transformations, means, and confidence intervals. BMJ. 1996 Apr 27;312(7038):1079-1079.
Reito A, Raittio L, Helminen O. Revisiting the sample size and statistical power of randomized controlled trials in orthopaedics after 2 decades. JBJS Rev. 2020 Feb;8(2):e0079.
Walker E, Nowacki AS. Understanding equivalence and noninferiority testing. J Gen Intern Med. 2011 Feb;26(2):192-6.
Cook JA, Julious SA, Sones W, Hampson LV, Hewitt C, Berlin JA, Ashby D, Emsley R, Fergusson DA, Walters SJ, Wilson EC, MacLennan G, Stallard N, Rothwell JC, Bland M, Brown L, Ramsay CR, Cook A, Armstrong D, Altman D, Vale LD. Practical help for specifying the target difference in sample size calculations for RCTs: the DELTA 2 five-stage study, including a workshop. Health Technol Assess. 2019 Oct;23(60):1-88.
Kamper SJ. Interpreting outcomes 1-Change and difference: linking evidence to practice. J Orthop Sports Phys Ther. 2019 May;49(5):357-8.
Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care. 2003 May;41(5):582-92.
Salkind NJ, editor. Encyclopedia of Research Design. SAGE Publications; 2010. Sampling error.