A new method for testing reproducibility in systematic reviews was developed, but needs more testing.
Data extraction
Information storage and retrieval
Methodological quality
Reproducibility of Results
Risk of bias
Systematic reviews
Journal
BMC medical research methodology
ISSN: 1471-2288
Titre abrégé: BMC Med Res Methodol
Pays: England
ID NLM: 100968545
Informations de publication
Date de publication:
29 07 2021
29 07 2021
Historique:
received:
21
12
2020
accepted:
28
06
2021
entrez:
30
7
2021
pubmed:
31
7
2021
medline:
10
8
2021
Statut:
epublish
Résumé
To develop and test an approach to test reproducibility of SRs. Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review. We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory. Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.
Sections du résumé
BACKGROUND
To develop and test an approach to test reproducibility of SRs.
METHODS
Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review.
RESULTS
We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory.
CONCLUSION
Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.
Identifiants
pubmed: 34325650
doi: 10.1186/s12874-021-01342-6
pii: 10.1186/s12874-021-01342-6
pmc: PMC8323273
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
157Informations de copyright
© 2021. The Author(s).
Références
Int J Environ Res Public Health. 2020 Mar 19;17(6):
pubmed: 32204395
J Clin Epidemiol. 2017 Jun;86:39-50
pubmed: 28529187
BMC Res Notes. 2013 Dec 17;6:539
pubmed: 24344873
J Clin Epidemiol. 2016 Aug;76:238
pubmed: 27034096
BMJ. 2017 Sep 21;358:j4008
pubmed: 28935701
J Clin Epidemiol. 2019 Feb;106:121-135
pubmed: 30312656
BMJ. 2009 Aug 13;339:b3128
pubmed: 19679616
J Med Libr Assoc. 2009 Jan;97(1):21-9
pubmed: 19158999
J Clin Epidemiol. 2019 Nov;115:64-76
pubmed: 31295514
J Clin Epidemiol. 2017 Nov;91:95-110
pubmed: 28842290
BMJ. 2012 Nov 01;345:e7031
pubmed: 23118303
J Clin Epidemiol. 2018 Jun;98:53-61
pubmed: 29476922
Milbank Q. 2016 Sep;94(3):485-514
pubmed: 27620683
PLoS One. 2016 Sep 26;11(9):e0163309
pubmed: 27669416
Syst Rev. 2016 Sep 29;5(1):164
pubmed: 27686611
PLoS Med. 2016 May 24;13(5):e1002028
pubmed: 27218655
BMC Psychol. 2016 May 31;4(1):24
pubmed: 27241618
JAMA. 2007 Jul 25;298(4):430-7
pubmed: 17652297
BMC Med Res Methodol. 2017 Nov 28;17(1):152
pubmed: 29179685
PLoS One. 2020 May 27;15(5):e0233107
pubmed: 32459806
Acad Med. 2011 Aug;86(8):1049-54
pubmed: 21694568
BMC Med Res Methodol. 2017 Apr 19;17(1):64
pubmed: 28420349
BMJ Open. 2019 Apr 1;9(4):e028382
pubmed: 30940766
Science. 2011 Dec 2;334(6060):1226-7
pubmed: 22144613
J Clin Epidemiol. 2020 Apr;120:25-32
pubmed: 31866473
Stat Med. 2002 Jun 15;21(11):1635-40
pubmed: 12111924
J Clin Epidemiol. 2018 Feb;94:8-18
pubmed: 29113936
J Clin Epidemiol. 2019 Jun;110:96
pubmed: 30772455
BMJ. 2021 Mar 29;372:n160
pubmed: 33781993
Perspect Psychol Sci. 2014 Sep;9(5):552-5
pubmed: 26186757
Clin Nutr. 2020 Feb;39(2):358-368
pubmed: 30905498
J Clin Epidemiol. 2005 Jul;58(7):741-2
pubmed: 15939227
J Clin Epidemiol. 2008 Aug;61(8):748-54
pubmed: 18586178
Br J Anaesth. 2019 Jun;122(6):e79-e81
pubmed: 30916036
Lancet. 2016 Apr 9;387(10027):1573-1586
pubmed: 26423180
BMC Med Res Methodol. 2016 Sep 26;16(1):127
pubmed: 27670136
J Clin Epidemiol. 2015 Sep;68(9):1076-84
pubmed: 26279401
Am J Gastroenterol. 2010 Feb;105(2):280-8
pubmed: 19920807
BMC Med Res Methodol. 2019 Jun 28;19(1):132
pubmed: 31253092