Crowdsourcing the identification of studies for COVID-19-related Cochrane Rapid Reviews.
Journal
Research synthesis methods
ISSN: 1759-2887
Titre abrégé: Res Synth Methods
Pays: England
ID NLM: 101543738
Informations de publication
Date de publication:
Sep 2022
Sep 2022
Historique:
revised:
15
03
2022
received:
21
07
2021
accepted:
06
04
2022
pubmed:
12
4
2022
medline:
14
9
2022
entrez:
11
4
2022
Statut:
ppublish
Résumé
Utilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID-19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID-19 research. To test whether a crowd could accurately assess study eligibility for reviews under time constraints. time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus. We created four crowd tasks, corresponding to four Cochrane COVID-19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID-19 task were invited to participate. Each task was live for 48-h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard. Across all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48-h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews. Crowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID-19 research for rapid evidence syntheses.
Sections du résumé
BACKGROUND
BACKGROUND
Utilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID-19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID-19 research.
AIMS
OBJECTIVE
To test whether a crowd could accurately assess study eligibility for reviews under time constraints.
OUTCOME MEASURES
METHODS
time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus.
METHODS
METHODS
We created four crowd tasks, corresponding to four Cochrane COVID-19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID-19 task were invited to participate. Each task was live for 48-h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard.
RESULTS
RESULTS
Across all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48-h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews.
CONCLUSION
CONCLUSIONS
Crowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID-19 research for rapid evidence syntheses.
Identifiants
pubmed: 35403367
doi: 10.1002/jrsm.1559
pmc: PMC9088532
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
585-594Informations de copyright
© 2022 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd.
Références
BMC Med Res Methodol. 2021 Apr 26;21(1):88
pubmed: 33906604
Cochrane Database Syst Rev. 2020 Sep 15;9:CD013718
pubmed: 33502003
Proc Natl Acad Sci U S A. 2004 Apr 20;101(16):6146-51
pubmed: 15071187
J Clin Epidemiol. 2021 Feb;130:13-22
pubmed: 33068715
Res Synth Methods. 2017 Sep;8(3):366-386
pubmed: 28677322
J Clin Epidemiol. 2020 Oct;126:116-121
pubmed: 32615208
J Med Internet Res. 2020 Jun 26;22(6):e19659
pubmed: 32558655
J Clin Epidemiol. 2021 Feb;130:23-31
pubmed: 33007457
Personal Disord. 2017 Jan;8(1):26-34
pubmed: 28045305
Nature. 2020 Aug;584(7821):425-429
pubmed: 32604404
J Clin Epidemiol. 2021 May;133:130-139
pubmed: 33476769
J Nurs Res. 2005 Mar;13(1):58-65
pubmed: 15977136
Cochrane Database Syst Rev. 2020 May 14;5:CD013600
pubmed: 32406927
Res Synth Methods. 2022 Sep;13(5):585-594
pubmed: 35403367
Cochrane Database Syst Rev. 2020 Apr 8;4:CD013574
pubmed: 32267544
Nature. 2020 Dec;588(7839):553
pubmed: 33328621
Cochrane Database Syst Rev. 2020 Apr 21;4:CD013582
pubmed: 32315451
J Clin Epidemiol. 2020 Nov;127:142-150
pubmed: 32798713
J Clin Epidemiol. 2020 Jun;122:160-162
pubmed: 32142884
Eur J Clin Microbiol Infect Dis. 2019 Aug;38(8):1383-1385
pubmed: 31049744
J Med Internet Res. 2019 Apr 29;21(4):e12953
pubmed: 31033444
J Clin Epidemiol. 2020 May;121:20-28
pubmed: 31972274
Emerg Infect Dis. 2020 Jul;26(7):
pubmed: 32275498