EEG Dataset for the Recognition of Different Emotions Induced in Voice-User Interaction.
Journal
Scientific data
ISSN: 2052-4463
Titre abrégé: Sci Data
Pays: England
ID NLM: 101640192
Informations de publication
Date de publication:
03 Oct 2024
03 Oct 2024
Historique:
received:
18
07
2022
accepted:
17
09
2024
medline:
4
10
2024
pubmed:
4
10
2024
entrez:
3
10
2024
Statut:
epublish
Résumé
Electroencephalography (EEG)-based open-access datasets are available for emotion recognition studies, where external auditory/visual stimuli are used to artificially evoke pre-defined emotions. In this study, we provide a novel EEG dataset containing the emotional information induced during a realistic human-computer interaction (HCI) using a voice user interface system that mimics natural human-to-human communication. To validate our dataset via neurophysiological investigation and binary emotion classification, we applied a series of signal processing and machine learning methods to the EEG data. The maximum classification accuracy ranged from 43.3% to 90.8% over 38 subjects and classification features could be interpreted neurophysiologically. Our EEG data could be used to develop a reliable HCI system because they were acquired in a natural HCI environment. In addition, auxiliary physiological data measured simultaneously with the EEG data also showed plausible results, i.e., electrocardiogram, photoplethysmogram, galvanic skin response, and facial images, which could be utilized for automatic emotion discrimination independently from, as well as together with the EEG data via the fusion of multi-modal physiological datasets.
Identifiants
pubmed: 39362909
doi: 10.1038/s41597-024-03887-9
pii: 10.1038/s41597-024-03887-9
doi:
Types de publication
Dataset
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
1084Subventions
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Organisme : National Research Foundation of Korea (NRF)
ID : 2020R1A4A1017775
Informations de copyright
© 2024. The Author(s).
Références
Dadebayev, D., Goh, W. W. & Tan, E. X. EEG-based emotion recognition: Review of commercial EEG devices and machine learning techniques. J. King. Saud. Univ. – Comput. Inf. Sci. 1, 29–40 (2021).
Suhaimi, N. S., Mountstephens, J. & Teo, J. EEG-based emotion recognition: a state-of-the-art review of current trends and opportunities. Comput. Intell. Neurosci. 8875426, 1–19 (2020).
doi: 10.1155/2020/8875426
Chunawale, A. & Bedekar, D. Human Emotion Recognition using Physiological Signals: A Survey. Proceeding of 2nd International Conference on Communication & Information Processing (ICCIP). 3645402 (2020).
Sarkar, P. & Etemad, A. Self-supervised learning for ECG-based emotion recognition. Proceeding of ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1, 3217–3221 (2020).
doi: 10.1109/ICASSP40776.2020.9053985
Raheel, A., Majid, M., Alnowami, M. & Anwar, S. M. Physiological sensors based emotion recognition while experiencing tactile enhanced multimedia. Sensors. 20, 4037 (2020).
pubmed: 32708056
pmcid: 7411620
doi: 10.3390/s20144037
Liu, Y. J. et al. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE. Trans. Affect. Comput. 9, 550–562 (2017).
doi: 10.1109/TAFFC.2017.2660485
Li, M. & Lu, B. L. Emotion classification based on gamma-band EEG. Proceeding of Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 1, 1223–1226 (2009).
Hassan, M. M. et al. Human emotion recognition using deep belief network architecture. Inf. Fusion. 51, 10–18 (2019).
doi: 10.1016/j.inffus.2018.10.009
Wiem, M. B. H. & Lachiri, Z. Emotion classification in arousal valence model using MAHNOB-HCI database. Int. J. Adv. Comput. 8, 1–6 (2017).
Reuderink, B., Mühl, C. & Poel, M. Valence arousal and dominance in the EEG during game play. Int. J. Auton. Adapt. 6, 45–62 (2013).
Chen, J. X. et al. Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access. 7, 44317–44328 (2019).
doi: 10.1109/ACCESS.2019.2908285
Li, X. et al. Exploring EEG features in cross-subject emotion recognition. Front. Neurosci. 12, 00162 (2018).
doi: 10.3389/fnins.2018.00162
Zheng, W. L., Zhu, J. Y. & Lu, B. L. Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans. Affect. Comput. 10, 417–429 (2017).
doi: 10.1109/TAFFC.2017.2712143
Huang, X. Multi-modal emotion analysis from facial expressions and electroencephalogram. Comput. Vis. Image. Underst. 147, 114–124 (2016).
doi: 10.1016/j.cviu.2015.09.015
Chai, X. Unsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition. Comput. Biol. Med. 79, 205–214 (2016).
pubmed: 27810626
doi: 10.1016/j.compbiomed.2016.10.019
Zheng, W. L. & Lu, B. L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Mental. Develop. 7, 162–175 (2015).
doi: 10.1109/TAMD.2015.2431497
Shin, J. G., Kim, J. B. & Kim, S. H. A framework to identify critical design parameters for enhancing user’s satisfaction in human-AI interactions. J. Phys. Conf. Ser. 1284, 237–243 (2019).
doi: 10.1088/1742-6596/1284/1/012036
Shin, J. G., Choi, G. Y., Hwang, H. J. & Kim, S. H. Evaluation of emotional satisfaction using questionnaires in voice-based human–AI interaction. Appl. Sci. 11, 1920 (2021).
doi: 10.3390/app11041920
Choi, S. I., Lee, J. Y., Lim, K. M. & Hwang, H. J. Evaluation of Real-Time Endogenous Brain-Computer Interface Developed Using Ear-Electroencephalography. Front. Neurosci. 16, 842635 (2022).
pubmed: 35401092
pmcid: 8987155
doi: 10.3389/fnins.2022.842635
Shim, M. et al. Altered functional networks of alpha and low-beta bands during upper limb movement and association with motor impairment in chronic stroke. Brain Connect. (2021).
Choi, S. I. & Hwang, H. J. Effects of different re-referencing methods on spontaneously generated ear-EEG. Front. Neurosci. 13, 822 (2019).
pubmed: 31440129
pmcid: 6692921
doi: 10.3389/fnins.2019.00822
Nagamachi, M. kansei engineering: a new ergonomic consumer-oriented technology for product development. Int. J. Ind. Ergon. 15, 3–11 (1995).
doi: 10.1016/0169-8141(94)00052-5
Osgood, C. E., Suci, G. J. & Tannenbaum, P. H. The measurement of meaning, 1st ed. University of Illinois Press. 1, 18–30 (1957).
Choi, G. Y., Han, C. H., Jung, Y. J. & Hwang, H. J. A multi-day and multi-band dataset for a steady-state visual-evoked potential–based brain-computer interface. Gigascience 8, giz133 (2019).
pubmed: 31765472
pmcid: 6876666
doi: 10.1093/gigascience/giz133
Han, C. H., Choi, G. Y. & Hwang, H. J. deep convolutional neural network based eye states classification using ear-EEG. Expert. Syst. Appl. 192, 116443 (2022).
doi: 10.1016/j.eswa.2021.116443
Han, C. H., Müller, K. R. & Hwang, H. J. Brain-switches for asynchronous brain–computer interfaces: A systematic review. Electronics. 9, 422 (2020).
doi: 10.3390/electronics9030422
Choi, S. I. & Hwang, H. J. Effects of different re-referencing methods on spontaneously generated Ear-EEG. Front. Neurosci. 1, 822 (2016).
Choi, S. I. et al. On the feasibility of using an ear-EEG to develop an endogenous brain-computer interface. Sensors. 18, 2856 (2018).
pubmed: 30158505
pmcid: 6165202
doi: 10.3390/s18092856
Kaiser, H. F. An index of factorial simplicity. Psychometrika. 39, 31–36 (1974).
doi: 10.1007/BF02291575
Watson, J. C. Establishing evidence for internal structure using exploratory factor analysis. Meas. Eval. Couns. Dev. 50, 232–238 (2017).
doi: 10.1080/07481756.2017.1336931
Yang, K. et al. High gamma band EEG closely related to emotion: evidence from functional network. Front. Hum. Neurosci. 14, 89 (2020).
pubmed: 32265674
pmcid: 7107011
doi: 10.3389/fnhum.2020.00089
Gao, Z., Cui, X., Wan, W. & Gu, Z. Recognition of emotional states using multiscale information analysis of high frequency EEG oscillations. Entropy. 21, 609 (2019).
pubmed: 33267323
pmcid: 7515095
doi: 10.3390/e21060609
Viola, F. C., Debener, S., Thorne, J. & Schneider, T. R. Using ICA for the analysis of multi-channel EEG data. Simultaneous EEG and fMRI: recording, analysis, and application. Oxford University. 1, 121–133 (2010).
Brown, K. W., Goodman, R. J. & Inzlicht, M. Dispositional mindfulness and the attenuation of neural responses to emotional stimuli. Soc. Cogn. Affect. Neurosci. 8, 93–99 (2013).
pubmed: 22253259
doi: 10.1093/scan/nss004
Arruda, J. E., Walker, K. A., Weiler, M. D. & Valentino, D. A. Validation of a right hemisphere vigilance system as measured by principal component and factor analyzed quantitative electroencephalogram. Int. J. Psychophysiol. 32, 119–128 (1999).
pubmed: 10380946
doi: 10.1016/S0167-8760(99)00006-9
Choi, G-Y. et al. EEG Dataset for the Recognition of Different Emotions Naturally Induced in Voice-User Interaction, figshare, https://doi.org/10.6084/m9.figshare.c.7207839 (2024).
Wang, Z., Tong, Y. & Heng, X. Phase-locking value based graph convolutional neural networks for emotion recognition. IEEE Access. 7, 93711–93722 (2019).
doi: 10.1109/ACCESS.2019.2927768
Gonuguntla, V., Shafiq, G., Wang, Y. & Veluvolu, K. C. EEG classification of emotions using emotion-specific brain functional network. Proceeding of 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 1, 2896–2899 (2015).
Sammler, D., Grigutsch, M., Fritz, T. & Koelsch, S. Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology. 44, 293–304 (2007).
pubmed: 17343712
doi: 10.1111/j.1469-8986.2007.00497.x
Balasubramanian, G., Kanagasabai, A., Mohan, J. & Seshadri, N. G. Music induced emotion using wavelet packet decomposition—An EEG study. Biomed. Signal. Process. Control. 42, 115–128 (2018).
doi: 10.1016/j.bspc.2018.01.015
Hwang, H. J., Kwon, K. & Im, C. H. Neurofeedback-based motor imagery training for brain-computer interface (BCI). J. Neurosci. Methods. 179, 150–156 (2009).
pubmed: 19428521
doi: 10.1016/j.jneumeth.2009.01.015
Deligani, R. J., Borgheai, S. B., McLinden, J. & Shahriari, Y. Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework. Biomed. Opt. Express 12, 1635 (2021).
pubmed: 33796378
pmcid: 7984774
doi: 10.1364/BOE.413666
Aghajani, H. & Omurtag, A. Assessment of mental workload by EEG+ fNIRS. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 1, 3773–3776 (2016).
Shim, M., Lee, S. H. & Hwang, H. J. Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection. Sci. Rep. 11, 7980 (2021).
pubmed: 33846489
pmcid: 8042090
doi: 10.1038/s41598-021-87157-3
Dasdemir, Y., Yildirim, E. & Yildirim, S. Analysis of functional brain connections for positive–negative emotions using phase locking value. Cogn. Neurodyn. 11, 487–500 (2017).
pubmed: 29147142
pmcid: 5670085
doi: 10.1007/s11571-017-9447-z
Uusberg, A., Thiruchselvam, R. & Gross, J. J. Using distraction to regulate emotion: Insights from EEG theta dynamics. J. Psychophysiol. 91, 254–260 (2014).
doi: 10.1016/j.ijpsycho.2014.01.006
Tandle, A., Jog, N., Dharmadhikari, A. & Jaiswal, S. Estimation of valence of emotion from musically stimulated EEG using frontal theta asymmetry. Proceeding of 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery. 1, 63–68 (2016).
Schubring, D. & Schupp, H. T. Emotion and brain oscillations: high arousal is associated with decreases in alpha-and lower beta-band power. Cereb. Cortex. 31, 597–1608 (2021).
doi: 10.1093/cercor/bhaa312
Schubring, D. & Schupp, H. T. Affective picture processing: Alpha‐and lower beta‐band desynchronization reflects emotional arousal. Psychophysiology. 56, e13386 (2019).
pubmed: 31026079
doi: 10.1111/psyp.13386
Ray, W. J. & Cole, H. W. EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science. 228, 750–752 (1985).
pubmed: 3992243
doi: 10.1126/science.3992243
Reisch, L. M., Wegrzyn, M., Woermann, F. G., Bien, C. G. & Kissler, J. Negative content enhances stimulus‐specific cerebral activity during free viewing of pictures, faces, and words. Hum. Brain Mapp. 41, 4332–4354 (2020).
pubmed: 32633448
pmcid: 7502837
doi: 10.1002/hbm.25128
Herbert, C. et al. Amygdala activation during reading of emotional adjectives—an advantage for pleasant content. Soc. Cogn. Affect. Neurosci. 4, 35–49 (2009).
pubmed: 19015080
doi: 10.1093/scan/nsn027
Holtgraves, T. & Felton, A. Hemispheric asymmetry in the processing of negative and positive words: A divided field study. Cogn. Emot. 25, 691–699 (2011).
pubmed: 21547769
doi: 10.1080/02699931.2010.493758
Goshvarpour, A., Abbasi, A. & Goshvarpour, A. An accurate emotion recognition system using ECG and GSR signals and matching pursuit method. Biomed J. 40, 355–368 (2017).
pubmed: 29433839
doi: 10.1016/j.bj.2017.11.001
Xu, Y. & Liu, G. Y. A method of emotion recognition based on ECG signal. Proceeding in 2009 International Conference on Computational Intelligence and Natural Computing. 1, 202–205 (2009).
doi: 10.1109/CINC.2009.102
Udovičić, G., Ðerek, J., Russo, M. & Sikora, M. Wearable emotion recognition system based on GSR and PPG signals. Proceedings of the 2nd International Workshop on Multimedia for Personal Health and Health Care. 1, 53–59 (2017).
doi: 10.1145/3132635.3132641
Sepúlveda, A., Castillo, F., Palma, C. & Rodriguez-Fernandez, M. Emotion recognition from ECG signals using wavelet scattering and machine learning. Appl Sci. 11, 4945 (2021).
doi: 10.3390/app11114945
Kartali, A., Roglić, M., Barjaktarović, M., Durić-Jovičić, M. & Janković, M. M. Real-time algorithms for facial emotion recognition: A comparison of different approaches. Proceeding in 2018 14th Symposium on Neural Networks and Applications (NEUREL). 1, 1–4 (2018).
Han, C. H., Müller, K. R. & Hwang, H. J. Enhanced performance of a brain switch by simultaneous use of EEG and NIRS data for asynchronous brain-computer interface. IEEE Trans. Neural. Syst. Rehabil. Eng. 28, 2102–2112 (2020).
pubmed: 32804653
doi: 10.1109/TNSRE.2020.3017167
Shin, J., Kim, D. W., Müller, K. R. & Hwang, H. J. Improvement of information transfer rates using a hybrid EEG-NIRS brain-computer interface with a short trial length: Offline and pseudo-online analyses. Sensors. 18, 1827 (2018).
pubmed: 29874804
pmcid: 6021956
doi: 10.3390/s18061827
Shin, J., Müller, K. R. & Hwang, H. J. Eyes-closed hybrid brain-computer interface employing frontal brain activation. PLoS. One. 13, e0196359 (2018).
pubmed: 29734383
pmcid: 5937739
doi: 10.1371/journal.pone.0196359
Han, C. H., Hwang, H. J., Lim, J. H. & Im, C. H. Assessment of user voluntary engagement during neurorehabilitation using functional near-infrared spectroscopy: a preliminary study. J. Neuroeng. Rehabil. 15, 1–10 (2018).
doi: 10.1186/s12984-018-0365-z
Won, D. O., Hwang, H. J., Kim, D. M., Müller, K. R. & Lee, S. W. Motion-based rapid serial visual presentation for gaze-independent brain-computer interfaces. IEEE Trans. Neural. Syst. Rehabil. Eng. 26, 334–343 (2017).
Lim, J. H. et al. An emergency call system for patients in locked‐in state using an SSVEP‐based brain switch. Psychophysiology. 54, 1632–1643 (2017).
pubmed: 28696536
doi: 10.1111/psyp.12916
Shin, J., Müller, K. R., Schmitz, C. H., Kim, D. W. & Hwang, H. J. Evaluation of a compact hybrid brain-computer interface system. Biomed. Res. Int. 1, 6820482 (2017).
Hwang, H. J. et al. Clinical feasibility of brain‐computer interface based on steady‐state visual evoked potential in patients with locked‐in syndrome: Case studies. Psychophysiology. 54, 444–451 (2017).
pubmed: 27914171
doi: 10.1111/psyp.12793
Shin, J., Müller, K. R. & Hwang, H. J. Near-infrared spectroscopy (NIRS)-based eyes-closed brain-computer interface (BCI) using prefrontal cortex activation due to mental arithmetic. Sci. Rep. 6, 1–11 (2016).
doi: 10.1038/srep36203
Hwang, H. J. et al. Toward more intuitive brain–computer interfacing: classification of binary covert intentions using functional near-infrared spectroscopy. J. Biomed. Opt. 21, 091303 (2016).
pubmed: 27050535
doi: 10.1117/1.JBO.21.9.091303
Won, D. O., Hwang, H. J., Dähne, S., Müller, K. R. & Lee, S. W. Effect of higher frequency on the classification of steady-state visual evoked potentials. J. Neural. Eng. 13, 016014 (2015).
pubmed: 26695712
doi: 10.1088/1741-2560/13/1/016014
Hwang, H. J. et al. A gaze independent brain-computer interface based on visual stimulation through closed eyelids. Sci. Rep. 5, 1–11 (2015).
doi: 10.1038/srep15890
Lim, J. H., Lee, J. H., Hwang, H. J., Kim, D. H. & Im, C. H. Development of a hybrid mental spelling system combining SSVEP-based brain–computer interface and webcam-based eye tracking. Biomed. Signal. Process. Control. 21, 99–104 (2015).
doi: 10.1016/j.bspc.2015.05.012
Hwang, H. J., Lim, J. H., Kim, D. W. & Im, C. H. Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces. J. Biomed. Opt. 19, 077005 (2014).
doi: 10.1117/1.JBO.19.7.077005
Han, C. H., Hwang, H. J. & Im, C. H. Classification of visual stimuli with different spatial patterns for single-frequency, multi-class SSVEP BCI. Electron. Lett. 49, 1374–1376 (2013).
doi: 10.1049/el.2013.2792
Hwang, H. J., Kim, S., Choi, S. & Im, C. H. EEG-based brain-computer interfaces: a thorough literature survey. Int. J. Hum-Comput. Interact. 29, 814–826 (2013).
doi: 10.1080/10447318.2013.780869
Hwang, H. J., Kim, D. H., Han, C. H. & Im, C. H. A new dual-frequency stimulation method to increase the number of visual stimuli for multi-class SSVEP-based brain–computer interface (BCI). Brain. Res. 1515, 66–77 (2013).
pubmed: 23587933
doi: 10.1016/j.brainres.2013.03.050
Park, S. et al. Evaluation of feature extraction methods for EEG-based brain–computer interfaces in terms of robustness to slight changes in electrode locations. Med. Biol. Eng. Comput. 51, 571–579 (2013).
pubmed: 23325145
doi: 10.1007/s11517-012-1026-1
Lim, J. H., Hwang, H. J., Han, C. H., Jung, K. Y. & Im, C. H. Classification of binary intentions for individuals with impaired oculomotor function: ‘eyes-closed’ SSVEP-based brain–computer interface (BCI). J. Neural. Eng. 10, 026021 (2013).
pubmed: 23528484
doi: 10.1088/1741-2560/10/2/026021
Hwang, H. J. et al. Development of an SSVEP-based BCI spelling system adopting a QWERTY-style LED keyboard. J. Neurosci. Methods. 208, 59–65 (2012).
pubmed: 22580222
doi: 10.1016/j.jneumeth.2012.04.011
Im, C. H., Hwang, H. J., Che, H. & Lee, S. An EEG-based real-time cortical rhythmic activity monitoring system. Physiol. Meas. 28, 1101 (2007).
pubmed: 17827657
doi: 10.1088/0967-3334/28/9/011