Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials.
P-hacking
Randomised controlled trials
Statistical analysis
Statistical analysis plan
Transparency
Journal
BMC medicine
ISSN: 1741-7015
Titre abrégé: BMC Med
Pays: England
ID NLM: 101190723
Informations de publication
Date de publication:
29 05 2020
29 05 2020
Historique:
received:
28
01
2020
accepted:
09
04
2020
entrez:
30
5
2020
pubmed:
30
5
2020
medline:
15
12
2020
Statut:
epublish
Résumé
Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as 'p-hacking') can bias the results of randomised trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomised trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed. A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial, we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan and compared this to the trial publication. Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%), it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n = 31, 35%) and analysis population (n = 28, 31%), followed by the use of covariates (n = 23, 26%) and the approach for handling missing data (n = 16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed. Unexplained discrepancies in the statistical methods of randomised trials are common. Increased transparency is required for proper evaluation of results.
Sections du résumé
BACKGROUND
Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as 'p-hacking') can bias the results of randomised trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomised trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed.
METHODS
A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial, we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan and compared this to the trial publication.
RESULTS
Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%), it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n = 31, 35%) and analysis population (n = 28, 31%), followed by the use of covariates (n = 23, 26%) and the approach for handling missing data (n = 16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed.
CONCLUSIONS
Unexplained discrepancies in the statistical methods of randomised trials are common. Increased transparency is required for proper evaluation of results.
Identifiants
pubmed: 32466758
doi: 10.1186/s12916-020-01590-1
pii: 10.1186/s12916-020-01590-1
pmc: PMC7257229
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
137Subventions
Organisme : Medical Research Council
ID : MC_UU_12023/21
Pays : United Kingdom
Références
J Clin Epidemiol. 2018 Sep;101:53-60
pubmed: 29864539
BMJ. 2003 May 31;326(7400):1171-3
pubmed: 12775615
Stat Methods Med Res. 2005 Oct;14(5):515-24
pubmed: 16248351
J Eval Clin Pract. 2002 Aug;8(3):353-9
pubmed: 12164983
BMJ. 2010 Mar 23;340:c869
pubmed: 20332511
Ann Intern Med. 2001 Apr 17;134(8):663-94
pubmed: 11304107
BMJ. 2019 Aug 28;366:l4898
pubmed: 31462531
PLoS One. 2008 Aug 28;3(8):e3081
pubmed: 18769481
Ann Intern Med. 2013 Feb 5;158(3):200-7
pubmed: 23295957
BMJ. 2013 Jul 12;347:f4313
pubmed: 23851720
N Engl J Med. 1990 Nov 8;323(19):1355
pubmed: 2215630
J Clin Epidemiol. 2013 May;66(5):524-37
pubmed: 23337785
PLoS Med. 2014 Jun 24;11(6):e1001666
pubmed: 24959719
JAMA Netw Open. 2019 Jul 3;2(7):e197242
pubmed: 31322690
BMJ. 2005 Apr 2;330(7494):753
pubmed: 15681569
J Clin Epidemiol. 2007 Jul;60(7):663-9
pubmed: 17573981
BMJ. 2008 Dec 04;337:a2299
pubmed: 19056791
Trials. 2019 Feb 14;20(1):118
pubmed: 30760329
N Engl J Med. 2002 Jan 24;346(4):289-90
pubmed: 11807157
BMJ. 2019 Nov 21;367:l6365
pubmed: 31753846
BMJ. 2013 Jan 08;346:e7586
pubmed: 23303884
F1000Res. 2014 Mar 26;3:77
pubmed: 25075294
N Engl J Med. 2009 Nov 12;361(20):1963-71
pubmed: 19907043
BMC Med Res Methodol. 2018 Jan 11;18(1):9
pubmed: 29325533
PLoS Med. 2008 Nov 25;5(11):e217; discussion e217
pubmed: 19067477
Cochrane Database Syst Rev. 2011 Jan 19;(1):MR000031
pubmed: 21249714
Clin Trials. 2020 Feb;17(1):99-105
pubmed: 31450958
Int J Clin Pract. 2008 Aug;62(8):1147-58
pubmed: 18564199
JAMA. 2017 Dec 19;318(23):2337-2343
pubmed: 29260229
JAMA. 2004 May 26;291(20):2457-65
pubmed: 15161896
N Engl J Med. 2014 Dec 25;371(26):2452-5
pubmed: 25539105