Accessible data curation and analytics for international-scale citizen science datasets.
Journal
Scientific data
ISSN: 2052-4463
Titre abrégé: Sci Data
Pays: England
ID NLM: 101640192
Informations de publication
Date de publication:
22 11 2021
22 11 2021
Historique:
received:
03
12
2020
accepted:
29
09
2021
entrez:
23
11
2021
pubmed:
24
11
2021
medline:
15
12
2021
Statut:
epublish
Résumé
The Covid Symptom Study, a smartphone-based surveillance study on COVID-19 symptoms in the population, is an exemplar of big data citizen science. As of May 23rd, 2021, over 5 million participants have collectively logged over 360 million self-assessment reports since its introduction in March 2020. The success of the Covid Symptom Study creates significant technical challenges around effective data curation. The primary issue is scale. The size of the dataset means that it can no longer be readily processed using standard Python-based data analytics software such as Pandas on commodity hardware. Alternative technologies exist but carry a higher technical complexity and are less accessible to many researchers. We present ExeTera, a Python-based open source software package designed to provide Pandas-like data analytics on datasets that approach terabyte scales. We present its design and capabilities, and show how it is a critical component of a data curation pipeline that enables reproducible research across an international research group for the Covid Symptom Study.
Identifiants
pubmed: 34811392
doi: 10.1038/s41597-021-01071-x
pii: 10.1038/s41597-021-01071-x
pmc: PMC8608807
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
297Subventions
Organisme : British Heart Foundation
Pays : United Kingdom
Organisme : Wellcome Trust (Wellcome)
ID : WT101957/Z/13/Z
Organisme : Chronic Disease Research Foundation (CDRF)
ID : CDRF-22/2020
Organisme : Wellcome Trust
ID : WT213038/Z/18/Z
Pays : United Kingdom
Organisme : RCUK | Engineering and Physical Sciences Research Council (EPSRC)
ID : NS/A000049/1
Organisme : Wellcome Trust
Pays : United Kingdom
Organisme : Wellcome Trust (Wellcome)
ID : WT203148/Z/16/Z
Organisme : RCUK | Engineering and Physical Sciences Research Council (EPSRC)
ID : NS/A000027/1
Informations de copyright
© 2021. The Author(s).
Références
Front Med (Lausanne). 2021 Jan 20;7:607786
pubmed: 33553204
Proc Natl Acad Sci U S A. 2019 Apr 23;116(17):8089-8092
pubmed: 31015357
Thorax. 2021 Jul;76(7):714-722
pubmed: 33402392
Nat Med. 2020 Jul;26(7):1037-1040
pubmed: 32393804
Lancet Public Health. 2020 Sep;5(9):e475-e483
pubmed: 32745512
Trends Ecol Evol. 2009 Sep;24(9):467-71
pubmed: 19586682
Thorax. 2021 Jul;76(7):723-725
pubmed: 33376145
PLoS One. 2017 Mar 16;12(3):e0173987
pubmed: 28301533
Science. 2020 Jun 19;368(6497):1362-1367
pubmed: 32371477
Lancet Public Health. 2021 Jan;6(1):e21-e29
pubmed: 33278917
Nature. 2020 Sep;585(7825):357-362
pubmed: 32939066
Age Ageing. 2021 Jan 8;50(1):40-48
pubmed: 32986799
Sci Adv. 2021 Mar 19;7(12):
pubmed: 33741586
PLoS One. 2015 Nov 23;10(11):e0143687
pubmed: 26600041
PLoS One. 2021 Sep 10;16(9):e0257051
pubmed: 34506535