Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study.
Agreement of effect estimates
Cohort studies
Meta-epidemiological study
Randomized controlled trials
Journal
BMC medicine
ISSN: 1741-7015
Titre abrégé: BMC Med
Pays: England
ID NLM: 101190723
Informations de publication
Date de publication:
11 05 2022
11 05 2022
Historique:
received:
01
12
2021
accepted:
06
04
2022
entrez:
10
5
2022
pubmed:
11
5
2022
medline:
14
5
2022
Statut:
epublish
Résumé
Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess the treatment effects of medical interventions. To evaluate the agreement of effect estimates between bodies of evidence (BoE) from randomized controlled trials (RCTs) and cohort studies and to identify factors associated with disagreement. Systematic reviews were published in the 13 medical journals with the highest impact factor identified through a MEDLINE search. BoE-pairs from RCTs and cohort studies with the same medical research question were included. We rated the similarity of PI/ECO (Population, Intervention/Exposure, Comparison, Outcome) between BoE from RCTs and cohort studies. The agreement of effect estimates across BoE was analyzed by pooling ratio of ratios (RoR) for binary outcomes and difference of mean differences for continuous outcomes. We performed subgroup analyses to explore factors associated with disagreements. One hundred twenty-nine BoE pairs from 64 systematic reviews were included. PI/ECO-similarity degree was moderate: two BoE pairs were rated as "more or less identical"; 90 were rated as "similar but not identical" and 37 as only "broadly similar". For binary outcomes, the pooled RoR was 1.04 (95% CI 0.97-1.11) with considerable statistical heterogeneity. For continuous outcomes, differences were small. In subgroup analyses, degree of PI/ECO-similarity, type of intervention, and type of outcome, the pooled RoR indicated that on average, differences between both BoE were small. Subgroup analysis by degree of PI/ECO-similarity revealed high statistical heterogeneity and wide prediction intervals across PI/ECO-dissimilar BoE pairs. On average, the pooled effect estimates between RCTs and cohort studies did not differ. Statistical heterogeneity and wide prediction intervals were mainly driven by PI/ECO-dissimilarities (i.e., clinical heterogeneity) and cohort studies. The potential influence of risk of bias and certainty of the evidence on differences of effect estimates between RCTs and cohort studies needs to be explored in upcoming meta-epidemiological studies.
Sections du résumé
BACKGROUND
Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess the treatment effects of medical interventions. To evaluate the agreement of effect estimates between bodies of evidence (BoE) from randomized controlled trials (RCTs) and cohort studies and to identify factors associated with disagreement.
METHODS
Systematic reviews were published in the 13 medical journals with the highest impact factor identified through a MEDLINE search. BoE-pairs from RCTs and cohort studies with the same medical research question were included. We rated the similarity of PI/ECO (Population, Intervention/Exposure, Comparison, Outcome) between BoE from RCTs and cohort studies. The agreement of effect estimates across BoE was analyzed by pooling ratio of ratios (RoR) for binary outcomes and difference of mean differences for continuous outcomes. We performed subgroup analyses to explore factors associated with disagreements.
RESULTS
One hundred twenty-nine BoE pairs from 64 systematic reviews were included. PI/ECO-similarity degree was moderate: two BoE pairs were rated as "more or less identical"; 90 were rated as "similar but not identical" and 37 as only "broadly similar". For binary outcomes, the pooled RoR was 1.04 (95% CI 0.97-1.11) with considerable statistical heterogeneity. For continuous outcomes, differences were small. In subgroup analyses, degree of PI/ECO-similarity, type of intervention, and type of outcome, the pooled RoR indicated that on average, differences between both BoE were small. Subgroup analysis by degree of PI/ECO-similarity revealed high statistical heterogeneity and wide prediction intervals across PI/ECO-dissimilar BoE pairs.
CONCLUSIONS
On average, the pooled effect estimates between RCTs and cohort studies did not differ. Statistical heterogeneity and wide prediction intervals were mainly driven by PI/ECO-dissimilarities (i.e., clinical heterogeneity) and cohort studies. The potential influence of risk of bias and certainty of the evidence on differences of effect estimates between RCTs and cohort studies needs to be explored in upcoming meta-epidemiological studies.
Identifiants
pubmed: 35538478
doi: 10.1186/s12916-022-02369-2
pii: 10.1186/s12916-022-02369-2
pmc: PMC9092682
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
174Informations de copyright
© 2022. The Author(s).
Références
NPJ Digit Med. 2020 May 11;3:67
pubmed: 32411828
JAMA. 2018 May 8;319(18):1914-1931
pubmed: 29801018
BMJ. 2005 Apr 16;330(7496):895-7
pubmed: 15831878
BMJ. 2019 Feb 21;364:l352
pubmed: 30792179
BMJ. 2016 Oct 13;355:i5170
pubmed: 27737834
Cochrane Database Syst Rev. 2010 Feb 17;(2):CD004876
pubmed: 20166072
World J Cardiol. 2015 Nov 26;7(11):808-16
pubmed: 26635929
J Clin Epidemiol. 2019 Jul;111:105-114
pubmed: 29432858
Cochrane Database Syst Rev. 2014 Jul 01;(7):CD007331
pubmed: 24980859
BMJ. 2014 Apr 15;348:g2366
pubmed: 24736555
Ann Intern Med. 2013 Feb 19;158(4):261-70
pubmed: 23420234
Res Synth Methods. 2016 Mar;7(1):55-79
pubmed: 26332144
Cochrane Database Syst Rev. 2017 Oct 30;10:CD006491
pubmed: 29083100
BMJ. 2015 Aug 30;351:h4451
pubmed: 26321114
PLoS One. 2011;6(6):e20811
pubmed: 21687681
Cancer Epidemiol Biomarkers Prev. 2014 Dec;23(12):2924-35
pubmed: 25258014
Ann Intern Med. 2016 Sep 6;165(5):334-44
pubmed: 27272666
Ann Surg. 2014 Jan;259(1):18-25
pubmed: 24096758
Cochrane Database Syst Rev. 2015 Jul 27;(7):CD011242
pubmed: 26214844
Ann Intern Med. 2013 Jul 16;159(2):123-9
pubmed: 23712381
Evid Based Ment Health. 2019 Nov;22(4):153-160
pubmed: 31563865
BMJ. 2014 Jan 24;348:f7450
pubmed: 24464277
Ann Intern Med. 2019 Oct 1;171(7):485-495
pubmed: 31476770
BMJ. 2012 Feb 03;344:e553
pubmed: 22306479
BMJ. 2021 Sep 15;374:n1864
pubmed: 34526355
BMJ. 2011 Dec 01;343:d7102
pubmed: 22134967
BMJ. 2014 Oct 24;349:g6269
pubmed: 25954988
Cochrane Database Syst Rev. 2017 Aug 25;8:CD010969
pubmed: 28841235
PLoS Med. 2018 Jul 3;15(7):e1002595
pubmed: 29969463
BMJ. 2003 Mar 1;326(7387):469
pubmed: 12609940
JAMA. 2012 Dec 19;308(23):2507-16
pubmed: 23287889
Cochrane Database Syst Rev. 2014 Apr 29;(4):MR000034
pubmed: 24782322
Cochrane Database Syst Rev. 2012 Aug 15;(8):CD009350
pubmed: 22895989
BMJ. 2014 Apr 01;348:g1903
pubmed: 24690623
Arch Orthop Trauma Surg. 2004 Jan;124(1):10-6
pubmed: 14576955
Cochrane Database Syst Rev. 2018 Jan 29;1:CD005195
pubmed: 29376219
CMAJ. 2017 Jul 17;189(28):E929-E939
pubmed: 28716847
Res Synth Methods. 2013 Mar;4(1):49-62
pubmed: 26053539
Eur Heart J. 2015 Jul 21;36(28):1831-8
pubmed: 25939649
BMJ. 2014 Apr 09;348:g2467
pubmed: 24922745
BMJ. 2012 Oct 30;345:e6698
pubmed: 23112118
Ann Intern Med. 2019 Nov 19;171(10):756-764
pubmed: 31569235
Ann Intern Med. 2011 Dec 20;155(12):827-38
pubmed: 22184690
BMJ. 2012 Jan 15;346:e7492
pubmed: 23321486
Cochrane Database Syst Rev. 2012 Aug 15;(8):CD004879
pubmed: 22895945
JAMA Intern Med. 2015 Nov;175(11):1752-1760
pubmed: 26365989
Trials. 2015 Nov 03;16:495
pubmed: 26530985
Ann Intern Med. 2014 Mar 18;160(6):398-406
pubmed: 24723079
PLoS One. 2013 Dec 26;8(12):e85035
pubmed: 24416098
BMJ. 2003 Jan 25;326(7382):219
pubmed: 12543843
JAMA. 2001 Aug 15;286(7):821-30
pubmed: 11497536
JAMA. 2006 Feb 8;295(6):655-66
pubmed: 16467234
J Palliat Med. 2009 Jan;12(1):77-82
pubmed: 19284267
J Clin Epidemiol. 2018 Jun;98:33-40
pubmed: 29452221
Am J Epidemiol. 2016 Apr 15;183(8):758-64
pubmed: 26994063
N Engl J Med. 2000 Jun 22;342(25):1878-86
pubmed: 10861324
BMJ. 2018 Oct 25;363:k4029
pubmed: 30361202
BMJ. 2013 Jun 13;346:f2743
pubmed: 23766464
BMJ. 2016 Oct 12;355:i4919
pubmed: 27733354
Ann Intern Med. 2016 Jun 7;164(11):752-63
pubmed: 27042809
Ann Intern Med. 2011 Aug 16;155(4):234-45
pubmed: 21844549
Lancet. 2010 Nov 6;376(9752):1552-7
pubmed: 20951422
BMJ. 2015 Sep 29;351:h4580
pubmed: 26420387
J Clin Epidemiol. 2020 Jul;123:127-130
pubmed: 32247656
Cochrane Database Syst Rev. 2019 Oct 3;10:ED000142
pubmed: 31643080
Evid Based Med. 2017 Aug;22(4):139-142
pubmed: 28701372
BMJ. 2015 Jun 22;350:h3163
pubmed: 26101226
Lancet. 2013 Oct 12;382(9900):1268-77
pubmed: 23870817
BMJ. 2019 Aug 28;366:l4898
pubmed: 31462531
Cochrane Database Syst Rev. 2010 Feb 17;(2):CD005187
pubmed: 20166073
Dtsch Arztebl Int. 2011 Sep;108(39):663-8
pubmed: 22013494
Cochrane Database Syst Rev. 2013 Apr 30;(4):CD009153
pubmed: 23633367
Hypertension. 2008 Nov;52(5):847-55
pubmed: 18824662
Cochrane Database Syst Rev. 2017 Apr 25;4:CD012200
pubmed: 28440858
BMJ. 2010 Jun 11;340:c2332
pubmed: 20543010
BMJ. 2016 Feb 17;352:i610
pubmed: 26888822
PLoS Med. 2018 Jan 23;15(1):e1002494
pubmed: 29360829
Int J Epidemiol. 2005 Feb;34(1):79-87
pubmed: 15561753
Mayo Clin Proc. 2017 Jan;92(1):15-29
pubmed: 28062061
BMC Med. 2017 Apr 21;15(1):84
pubmed: 28427392
BMJ. 2011 Feb 10;342:d549
pubmed: 21310794
Ann Intern Med. 2010 Mar 2;152(5):307-14
pubmed: 20194237
BMJ. 2019 Jan 07;364:k5120
pubmed: 30617123
BMJ. 2017 Mar 16;356:j1000
pubmed: 28302601
BMJ. 1991 Feb 2;302(6771):249-50
pubmed: 1998787
JAMA Intern Med. 2015 Jun;175(6):931-939
pubmed: 25822657
J Clin Epidemiol. 2011 Oct;64(10):1076-84
pubmed: 21482068
Cochrane Database Syst Rev. 2010 Feb 17;(2):CD006756
pubmed: 20166087
Ann Intern Med. 2016 Oct 04;165(7):491-500
pubmed: 27428849
Ann Intern Med. 2016 Dec 20;165(12):856-866
pubmed: 27776363
BMC Med. 2021 Dec 6;19(1):307
pubmed: 34865623
PLoS Med. 2012;9(7):e1001270
pubmed: 22911011
J Clin Epidemiol. 2020 Jul;123:133-134
pubmed: 32251682
BMJ. 2014 Sep 09;349:g5133
pubmed: 25208953
J Res Natl Bur Stand (1977). 1982 Sep-Oct;87(5):377-385
pubmed: 34566088
Ann Intern Med. 2011 Apr 19;154(8):529-40
pubmed: 21502651
Ann Intern Med. 2013 Dec 3;159(11):746-757
pubmed: 24297191
BMJ. 2018 Oct 08;363:k3845
pubmed: 30297439
BMJ. 2013 Apr 03;346:f1326
pubmed: 23558163
Circulation. 2021 Mar 9;143(10):1002-1013
pubmed: 33327727
Cochrane Database Syst Rev. 2010 Feb 17;(2):CD005953
pubmed: 20166076
JAMA. 2019 Sep 24;322(12):1195-1205
pubmed: 31550037
Evid Based Med. 2016 Aug;21(4):125-7
pubmed: 27339128
J Clin Epidemiol. 2011 Apr;64(4):383-94
pubmed: 21195583
Cochrane Database Syst Rev. 2013 Jan 31;(1):CD000422
pubmed: 23440780
PLoS Med. 2019 Dec 9;16(12):e1002983
pubmed: 31815931
Ann Intern Med. 2013 May 7;158(9):676-685
pubmed: 23648949
J Clin Epidemiol. 2010 May;63(5):513-23
pubmed: 19595577
BMJ. 2003 Sep 6;327(7414):557-60
pubmed: 12958120