Realness of face images can be decoded from non-linear modulation of EEG responses.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
07 Mar 2024
07 Mar 2024
Historique:
received:
02
08
2023
accepted:
01
03
2024
medline:
8
3
2024
pubmed:
8
3
2024
entrez:
7
3
2024
Statut:
epublish
Résumé
Artificially created human faces play an increasingly important role in our digital world. However, the so-called uncanny valley effect may cause people to perceive highly, yet not perfectly human-like faces as eerie, bringing challenges to the interaction with virtual agents. At the same time, the neurocognitive underpinnings of the uncanny valley effect remain elusive. Here, we utilized an electroencephalography (EEG) dataset of steady-state visual evoked potentials (SSVEP) in which participants were presented with human face images of different stylization levels ranging from simplistic cartoons to actual photographs. Assessing neuronal responses both in frequency and time domain, we found a non-linear relationship between SSVEP amplitudes and stylization level, that is, the most stylized cartoon images and the real photographs evoked stronger responses than images with medium stylization. Moreover, realness of even highly similar stylization levels could be decoded from the EEG data with task-related component analysis (TRCA). Importantly, we also account for confounding factors, such as the size of the stimulus face's eyes, which previously have not been adequately addressed. Together, this study provides a basis for future research and neuronal benchmarking of real-time detection of face realness regarding three aspects: SSVEP-based neural markers, efficient classification methods, and low-level stimulus confounders.
Identifiants
pubmed: 38454099
doi: 10.1038/s41598-024-56130-1
pii: 10.1038/s41598-024-56130-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
5683Informations de copyright
© 2024. The Author(s).
Références
McDonnell, R., & Breidt, M. Face reality: Investigating the uncanny valley for virtual faces. ACM SIGGRAPH ASIA 2010 Sketches. 1–2 (2010).
Adolphs, R. Recognizing emotion from facial expressions: Psychological and neurological mechanisms. Behav. Cogn. Neurosci. Rev. 1(1), 21–62 (2002).
doi: 10.1177/1534582302001001003
Caharel, S. et al. ERPs associated with familiarity and degree of familiarity during face recognition. Int. J. Neurosci. 112(12), 1499–1512 (2002).
doi: 10.1080/00207450290158368
Calder, A. J. & Young, A. W. Understanding the recognition of facial identity and facial expression. Nat. Rev. Neurosci. 6(8), 641–651 (2005).
doi: 10.1038/nrn1724
Bruce, V. & Young, A. Understanding face recognition. Br. J. Psychol. 77(3), 305–327 (1986).
doi: 10.1111/j.2044-8295.1986.tb02199.x
Young, A. W., Hellawell, D. & Hay, D. C. Configurational information in face perception. Perception 42(11), 1166–1178 (2013).
doi: 10.1068/p160747n
Moshel, M. L., Robinson, A. K., Carlson, T. A. & Grootswagers, T. Are you for real? Decoding realistic AI-generated faces from neural activity. Vis. Res. 199, 108079 (2022).
doi: 10.1016/j.visres.2022.108079
Wang, T-C. et al. High-resolution image synthesis and semantic manipulation with conditional gans. Proceedings of the IEEE conference on computer vision and pattern recognition (2018).
Karras, T., Laine, S., & Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 4401–4410 (2019).
Westerlund, M. The emergence of deepfake technology: A review. Technol. Innov. Manag. Rev. 9(11) (2019).
Nightingale, S. J. & Farid, H. AI-synthesized faces are indistinguishable from real faces and more trustworthy. Proc. Natl. Acad. Sci. 119(8), e2120481119 (2022).
doi: 10.1073/pnas.2120481119
Mori, M., MacDorman, K. F. & Kageki, N. The uncanny valley [from the field]. IEEE Robot. Autom. Mag. 19(2), 98–100 (2012).
doi: 10.1109/MRA.2012.2192811
Burleigh, T. J., Schoenherr, J. R. & Lacroix, G. L. Does the uncanny valley exist? An empirical test of the relationship between eeriness and the human likeness of digitally created faces. Comput. Hum. Behav. 29(3), 759–771 (2013).
doi: 10.1016/j.chb.2012.11.021
Geller, T. Overcoming the uncanny valley. IEEE Comput. Graph. Appl. 28(4), 11–17 (2008).
doi: 10.1109/MCG.2008.79
Kätsyri, J. et al. A review of empirical evidence on different uncanny valley hypotheses: Support for perceptual mismatch as one road to the valley of eeriness. Front. Psychol. 6, 390 (2015).
doi: 10.3389/fpsyg.2015.00390
Kätsyri, J., de Gelder, B. & Takala, T. Virtual faces evoke only a weak uncanny valley effect: An empirical investigation with controlled virtual face images. Perception 48(10), 968–991 (2019).
doi: 10.1177/0301006619869134
Złotowski, J. A. et al. Persistence of the uncanny valley: The influence of repeated interactions and a robot’s attitude on its perception. Front. Psychol. 6, 883 (2015).
doi: 10.3389/fpsyg.2015.00883
Yamada, Y., Kawabe, T. & Ihaya, K. Categorization difficulty is associated with negative evaluation in the “uncanny valley” phenomenon. Jpn. Psychol. Res. 55(1), 20–32 (2013).
doi: 10.1111/j.1468-5884.2012.00538.x
Saygin, A. P. et al. The thing that should not be: Predictive coding and the uncanny valley in perceiving human and humanoid robot actions. Soc. Cogn. Affect. Neurosci. 7(4), 413–422 (2012).
doi: 10.1093/scan/nsr025
Urgen, B. A., Kutas, M. & Saygin, A. P. Uncanny valley as a window into predictive processing in the social brain. Neuropsychologia 114, 181–185 (2018).
doi: 10.1016/j.neuropsychologia.2018.04.027
Gray, K. & Wegner, D. M. Feeling robots and human zombies: Mind perception and the uncanny valley. Cognition 125(1), 125–130 (2012).
doi: 10.1016/j.cognition.2012.06.007
MacDorman, K. F. & Ishiguro, H. The uncanny advantage of using androids in cognitive and social science research. Interact. Stud. 7(3), 297–337 (2006).
doi: 10.1075/is.7.3.03mac
Wang, S., Lilienfeld, S. O. & Rochat, P. The uncanny valley: Existence and explanations. Rev. Gen. Psychol. 19(4), 393–407 (2015).
doi: 10.1037/gpr0000056
Moore, R. K. A Bayesian explanation of the ‘Uncanny Valley’effect and related psychological phenomena. Sci. Rep. 2(1), 1–5 (2012).
doi: 10.1038/srep00864
Vaitonytė, J., Alimardani, M., & Louwerse, M. M. Scoping review of the neural evidence on the uncanny valley. Comput. Hum. Behav. Rep. 100263 (2022).
Diel, A., Weigelt, S. & Macdorman, K. F. A meta-analysis of the uncanny valley’s independent and dependent variables. ACM Trans. Hum. Robot Interact. (THRI) 11(1), 1–33 (2021).
MacDorman, K. F. et al. Too real for comfort? Uncanny responses to computer generated faces. Comput. Hum. Behav. 25(3), 695–710 (2009).
doi: 10.1016/j.chb.2008.12.026
Seyama, J. & Nagayama, R. S. The uncanny valley: Effect of realism on the impression of artificial human faces. Presence 16(4), 337–351 (2007).
doi: 10.1162/pres.16.4.337
Mustafa, M. et al. How human am I? EEG-based evaluation of virtual characters. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (2017).
Bentin, S. et al. Electrophysiological studies of face perception in humans. J. Cogn. Neurosci. 8(6), 551–565 (1996).
doi: 10.1162/jocn.1996.8.6.551
Blau, V. C. et al. The face-specific N170 component is modulated by emotional facial expression. Behav. Brain Funct. 3(1), 1–13 (2007).
doi: 10.1186/1744-9081-3-7
Rossion, B. et al. The N170 occipito-temporal component is delayed and enhanced to inverted faces but not to inverted objects: An electrophysiological account of face-specific processes in the human brain. Neuroreport 11(1), 69–72 (2000).
doi: 10.1097/00001756-200001170-00014
Puce, A., Smith, A. & Allison, T. ERPs evoked by viewing facial movements. Cogn. Neuropsychol. 17(1–3), 221–239 (2000).
doi: 10.1080/026432900380580
Stephani, T. et al. Eye contact in active and passive viewing: Event-related brain potential evidence from a combined eye tracking and EEG study. Neuropsychologia 143, 107478 (2020).
doi: 10.1016/j.neuropsychologia.2020.107478
Latinus, M. et al. Social decisions affect neural activity to perceived dynamic gaze. Soc. Cogn. Affect. Neurosci. 10(11), 1557–1567 (2015).
doi: 10.1093/scan/nsv049
Itier, R. J. et al. Explicit versus implicit gaze processing assessed by ERPs. Brain Res. 1177, 79–89 (2007).
doi: 10.1016/j.brainres.2007.07.094
Schindler, S. et al. Differential effects of face-realism and emotion on event-related brain potentials and their implications for the uncanny valley theory. Sci. Rep. 7(1), 1–13 (2017).
doi: 10.1038/srep45003
Di Russo, F. et al. Spatiotemporal analysis of the cortical sources of the steady-state visual evoked potential. Hum. Brain Mapp. 28(4), 323–334 (2007).
doi: 10.1002/hbm.20276
Norcia, A. M. et al. The steady-state visual evoked potential in vision research: A review. J. Vis. 15(6), 4–4 (2015).
doi: 10.1167/15.6.4
Regan, D. Some characteristics of average steady-state and transient responses evoked by modulated light. Electroencephalogr. Clin. Neurophysiol. 20(3), 238–248 (1966).
doi: 10.1016/0013-4694(66)90088-5
Bosse, S. et al. Assessing perceived image quality using steady-state visual evoked potentials and spatio-spectral decomposition. IEEE Trans. Circuits Syst. Video Technol. 28(8), 1694–1706 (2017).
doi: 10.1109/TCSVT.2017.2694807
Acqualagna, L. et al. EEG-based classification of video quality perception using steady state visual evoked potentials (SSVEPs). J. Neural Eng. 12(2), 026012 (2015).
doi: 10.1088/1741-2560/12/2/026012
Ajaj, T., Mϋller, K. R., Curio, G., Wieg, T., & Bosse, S. EEG-Based Assessment of Perceived Quality in Complex Natural Images. In 2020 IEEE International Conference on Image Processing (ICIP) 136–140 (IEEE, 2020).
Rossion, B. & Boremanse, A. Robust sensitivity to facial identity in the right human occipito-temporal cortex as revealed by steady-state visual-evoked potentials. J. Vis. 11(2), 16–16 (2011).
doi: 10.1167/11.2.16
Gruss, L. F. et al. Face-evoked steady-state visual potentials: Effects of presentation rate and face inversion. Front. Hum. Neurosci. 6, 316 (2012).
doi: 10.3389/fnhum.2012.00316
Kotlewska, I. et al. Present and past selves: A steady-state visual evoked potentials approach to self-face processing. Sci. Rep. 7(1), 1–9 (2017).
doi: 10.1038/s41598-017-16679-6
Bagdasarian, M. T. et al. EEG-based assessment of perceived realness in stylized face images. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX) (IEEE, 2020).
Zell, E. et al. To stylize or not to stylize? The effect of shape and material stylization on the perception of computer-generated faces. ACM Trans. Graph. (TOG) 34(6), 1–12 (2015).
doi: 10.1145/2816795.2818126
Alonso-Prieto, E. et al. The 6 Hz fundamental stimulation frequency rate for individual face discrimination in the right occipito-temporal cortex. Neuropsychologia 51(13), 2863–2875 (2013).
doi: 10.1016/j.neuropsychologia.2013.08.018
Bosse, S. et al. On the stimulation frequency in ssvep-based image quality assessment. In 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX) (IEEE, 2018).
Delorme, A. & Makeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134(1), 9–21 (2004).
doi: 10.1016/j.jneumeth.2003.10.009
Gramfort, A. et al. MEG and EEG data analysis with MNE-Python. Front. Neurosci. 267 (2013).
Nikulin, V. V., Nolte, G. & Curio, G. A novel method for reliable and fast extraction of neuronal EEG/MEG oscillations on the basis of spatio-spectral decomposition. NeuroImage 55(4), 1528–1535 (2011).
doi: 10.1016/j.neuroimage.2011.01.057
Haufe, S. et al. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87, 96–110 (2014).
doi: 10.1016/j.neuroimage.2013.10.067
Nakanishi, M. et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans. Biomed. Eng. 65(1), 104–112 (2017).
doi: 10.1109/TBME.2017.2694818
Tanaka, H., Katura, T. & Sato, H. Task-related component analysis for functional neuroimaging and application to near-infrared spectroscopy data. NeuroImage 64, 308–327 (2013).
doi: 10.1016/j.neuroimage.2012.08.044
Bin, G. et al. An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method. J. Neural Eng. 6(4), 046002 (2009).
doi: 10.1088/1741-2560/6/4/046002
Bates, D. et al. Fitting linear mixed-effects models using lme4. arXiv preprint arXiv:1406.5823 (2014).
Ihaka, R. & Gentleman, R. R: A language for data analysis and graphics. J. Comput. Graph. Stat. 5(3), 299–314 (1996).
Schindler, S. et al. Effects of low-level visual information and perceptual load on P1 and N170 responses to emotional expressions. Cortex 136, 14–27 (2021).
doi: 10.1016/j.cortex.2020.12.011
Grand, R. L. et al. Expert face processing requires visual input to the right hemisphere during infancy. Nat. Neurosci. 6(10), 1108–1112 (2003).
doi: 10.1038/nn1121
Rossion, B. & Caharel, S. ERP evidence for the speed of face categorization in the human brain: Disentangling the contribution of low-level visual cues from face perception. Vis. Res. 51(12), 1297–1311 (2011).
doi: 10.1016/j.visres.2011.04.003
Mouli, S., & Palaniappan, R. Eliciting higher SSVEP response from LED visual stimulus with varying luminosity levels. In 2016 International Conference for Students on Applied Engineering (ICSAE). (IEEE, 2016).
Widmann, A., Schröger, E. & Maess, B. Digital filter design for electrophysiological data–a practical approach. J. Neurosci. Methods 250, 34–46 (2015).
doi: 10.1016/j.jneumeth.2014.08.002
Eimer, M. The face-specific N170 component reflects late stages in the structural encoding of faces. Neuroreport 11(10), 2319–2324 (2000).
doi: 10.1097/00001756-200007140-00050
Allen-Davidian, Y. et al. Turning the face inversion effect on its head: Violated expectations of orientation, lighting, and gravity enhance N170 amplitudes. J. Cogn. Neurosci. 33(2), 303–314 (2021).
doi: 10.1162/jocn_a_01656
Rossion, B., & Jacques, C. The N170: Understanding the time course of face perception in the human brain. The Oxford Handbook of ERP Components 115–142 (2011).
Johnston, P. et al. Temporal and spatial localization of prediction-error signals in the visual brain. Biol. Psychol. 125, 45–57 (2017).
doi: 10.1016/j.biopsycho.2017.02.004
Robinson, J. E. et al. Dose-dependent modulation of the visually evoked N1/N170 by perceptual surprise: A clear demonstration of prediction-error signalling. Eur. J. Neurosci. 52(11), 4442–4452 (2020).
doi: 10.1111/ejn.13920
Capilla, A. et al. Steady-state visual evoked potentials can be explained by temporal superposition of transient event-related responses. PloS One 6(1), e14543 (2011).
doi: 10.1371/journal.pone.0014543
Idaji, M. J. et al. Harmoni: A method for eliminating spurious interactions due to the harmonic components in neuronal data. Neuroimage 252, 119053 (2022).
doi: 10.1016/j.neuroimage.2022.119053
Schaworonkow, N. & Nikulin, V. V. Spatial neuronal synchronization and the waveform of oscillations: Implications for EEG and MEG. PLoS Comput. Biol. 15(5), e1007055 (2019).
doi: 10.1371/journal.pcbi.1007055
Whalen, P. J. et al. Human amygdala responsivity to masked fearful eye whites. Science 306(5704), 2061–2061 (2004).
doi: 10.1126/science.1103617