Understanding action concepts from videos and brain activity through subjects' consensus.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
09 11 2022
09 11 2022
Historique:
received:
21
01
2022
accepted:
25
10
2022
entrez:
9
11
2022
pubmed:
10
11
2022
medline:
15
11
2022
Statut:
epublish
Résumé
In this paper, we investigate brain activity associated with complex visual tasks, showing that electroencephalography (EEG) data can help computer vision in reliably recognizing actions from video footage that is used to stimulate human observers. Notably, we consider not only typical "explicit" video action benchmarks, but also more complex data sequences in which action concepts are only referred to, implicitly. To this end, we consider a challenging action recognition benchmark dataset-Moments in Time-whose video sequences do not explicitly visualize actions, but only implicitly refer to them (e.g., fireworks in the sky as an extreme example of "flying"). We employ such videos as stimuli and involve a large sample of subjects to collect a high-definition, multi-modal EEG and video data, designed for understanding action concepts. We discover an agreement among brain activities of different subjects stimulated by the same video footage. We name it as subjects consensus, and we design a computational pipeline to transfer knowledge from EEG to video, sharply boosting the recognition performance.
Identifiants
pubmed: 36351956
doi: 10.1038/s41598-022-23067-2
pii: 10.1038/s41598-022-23067-2
pmc: PMC9646846
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
19073Subventions
Organisme : HORIZON EUROPE European Research Council
ID : ERC-2016-StG-715058
Informations de copyright
© 2022. The Author(s).
Références
Phys Rev E Stat Nonlin Soft Matter Phys. 2012 Oct;86(4 Pt 2):046206
pubmed: 23214662
IEEE Trans Biomed Eng. 2000 Sep;47(9):1185-94
pubmed: 11008419
IEEE Trans Pattern Anal Mach Intell. 2020 Feb;42(2):502-508
pubmed: 30802849
Comput Intell Neurosci. 2011;2011:156869
pubmed: 21253357
Neural Comput. 1997 Nov 15;9(8):1735-80
pubmed: 9377276
Neuropsychologia. 2004;42(3):379-86
pubmed: 14670576
IEEE Trans Biomed Eng. 2004 Jun;51(6):1034-43
pubmed: 15188875
PLoS One. 2010 Dec 30;5(12):e14465
pubmed: 21209937
Front Neurorobot. 2017 Apr 10;11:19
pubmed: 28443015
PLoS Comput Biol. 2018 Mar 12;14(3):e1005938
pubmed: 29529062
Curr Biol. 2011 Oct 11;21(19):1641-6
pubmed: 21945275
Front Hum Neurosci. 2016 Jun 27;10:306
pubmed: 27445747
Sci Rep. 2021 Dec 3;11(1):23383
pubmed: 34862442
Annu Int Conf IEEE Eng Med Biol Soc. 2013;2013:6627-30
pubmed: 24111262
IEEE Trans Neural Syst Rehabil Eng. 2006 Jun;14(2):174-9
pubmed: 16792287
IEEE Trans Biomed Eng. 2004 Jun;51(6):1044-51
pubmed: 15188876
Annu Int Conf IEEE Eng Med Biol Soc. 2017 Jul;2017:3224-3227
pubmed: 29060584
Front Neurosci. 2018 Mar 19;12:162
pubmed: 29615853
J Neurosci Methods. 2007 May 15;162(1-2):8-13
pubmed: 17254636
Sci Rep. 2021 Aug 26;11(1):17274
pubmed: 34446791
Sleep Med. 2001 Nov;2(6):537-53
pubmed: 14592270
IEEE Trans Pattern Anal Mach Intell. 2020 Oct;42(10):2581-2593
pubmed: 31331879
Annu Int Conf IEEE Eng Med Biol Soc. 2013;2013:5263-6
pubmed: 24110923
Biol Psychol. 2000 Mar;52(2):95-111
pubmed: 10699350
IEEE Trans Pattern Anal Mach Intell. 2021 Nov;43(11):3833-3849
pubmed: 32750768
Comput Biol Med. 2016 Dec 1;79:205-214
pubmed: 27810626
Neuroimage. 2011 Feb 14;54(4):3028-39
pubmed: 21040796
IEEE Trans Pattern Anal Mach Intell. 2020 Nov 19;PP:
pubmed: 33211652