Idiosyncratic fixation patterns generalize across dynamic and static facial expression recognition.
Individual differences – facial expressions of emotion – eye-movements
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
13 Jul 2024
13 Jul 2024
Historique:
received:
21
03
2024
accepted:
02
07
2024
medline:
14
7
2024
pubmed:
14
7
2024
entrez:
13
7
2024
Statut:
epublish
Résumé
Facial expression recognition (FER) is crucial for understanding the emotional state of others during human social interactions. It has been assumed that humans share universal visual sampling strategies to achieve this task. However, recent studies in face identification have revealed striking idiosyncratic fixation patterns, questioning the universality of face processing. More importantly, very little is known about whether such idiosyncrasies extend to the biological relevant recognition of static and dynamic facial expressions of emotion (FEEs). To clarify this issue, we tracked observers' eye movements categorizing static and ecologically valid dynamic faces displaying the six basic FEEs, all normalized for time presentation (1 s), contrast and global luminance across exposure time. We then used robust data-driven analyses combining statistical fixation maps with hidden Markov Models to explore eye-movements across FEEs and stimulus modalities. Our data revealed three spatially and temporally distinct equally occurring face scanning strategies during FER. Crucially, such visual sampling strategies were mostly comparably effective in FER and highly consistent across FEEs and modalities. Our findings show that spatiotemporal idiosyncratic gaze strategies also occur for the biologically relevant recognition of FEEs, further questioning the universality of FER and, more generally, face processing.
Identifiants
pubmed: 39003314
doi: 10.1038/s41598-024-66619-4
pii: 10.1038/s41598-024-66619-4
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
16193Subventions
Organisme : Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
ID : 10001C_201145
Informations de copyright
© 2024. The Author(s).
Références
Ekman, P. & Friesen, W. V. Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues (Ishk, 1975).
Izard, C. E. The face of Emotion (Appleton-Century-Crofts, 1971).
Darwin, C. The Expression of the Emotions in Man and Animals (John Murray, 1872).
doi: 10.1037/10001-000
Yitzhak, N., Pertzov, Y. & Aviezer, H. The elusive link between eye-movement patterns and facial expression recognition. Soc. Personal Psychol. Compass https://doi.org/10.1111/spc3.12621 (2021).
doi: 10.1111/spc3.12621
White, D. & Burton, A. M. Individual differences and the multidimensional nature of face perception. Nat. Rev. Psychol. 1, 287 (2022).
doi: 10.1038/s44159-022-00041-3
Blais, C. & Caldara, R. Culture Shapes Face Processing. In Oxford Handbook of Cultural Neuroscience and Global Mental Health (eds Joan, Y. et al.) (Oxford University Press, 2021).
Blais, C., Jack, R. E., Scheepers, C., Fiset, D. & Caldara, R. Culture shapes how we look at faces. PLoS ONE 3, e3022 (2008).
pubmed: 18714387
pmcid: 2515341
doi: 10.1371/journal.pone.0003022
Caldara, R. & Miellet, S. iMap: A novel method for statistical fixation mapping of eye movement data. Behav. Res. Methods 43, 864–878 (2011).
pubmed: 21512875
doi: 10.3758/s13428-011-0092-x
Caldara, R. Culture Reveals a Flexible System for Face Processing. Curr. Dir. Psychol. Sci. 26, 249 (2017).
doi: 10.1177/0963721417710036
Yuki, M., Maddux, W. W. & Masuda, T. Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. J. Exp. Soc. Psychol. 43, 303–311 (2007).
doi: 10.1016/j.jesp.2006.02.004
Masuda, T. et al. Placing the Face in Context: Cultural Differences in the Perception of Facial Emotion. J. Pers. Soc. Psychol. 94, 365–381 (2008).
pubmed: 18284287
doi: 10.1037/0022-3514.94.3.365
Jack, R. E., Blais, C., Scheepers, C., Schyns, P. G. & Caldara, R. Cultural Confusions Show that Facial Expressions Are Not Universal. Curr. Biol. 19, 1543–1548 (2009).
pubmed: 19682907
doi: 10.1016/j.cub.2009.07.051
Gendron, M., Roberson, D., Van der Vyver, J. M. & Feldman Barret, L. Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. Emotion 14, 251–262 (2014).
pubmed: 24708506
pmcid: 4752367
doi: 10.1037/a0036052
Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R. & Schyns, P. G. Facial expressions of emotion are not culturally universal. Proc. Natl. Acad. Sci. U S A 109, 7241 (2012).
pubmed: 22509011
pmcid: 3358835
doi: 10.1073/pnas.1200155109
Geangu, E. et al. Culture shapes 7-month-olds’ perceptual strategies in discriminating facial expressions of emotion. Curr. Biol. 26, R663–R664 (2016).
pubmed: 27458908
doi: 10.1016/j.cub.2016.05.072
Quesque, F. et al. Does Culture Shape Our Understanding of Others’ Thoughts and Emotions? An Investigation Across 12 Countries. Neuropsychology 36, 664–682 (2022).
pubmed: 35834208
pmcid: 11186050
doi: 10.1037/neu0000817
Stacchi, L., Ramon, M., Lao, J. & Caldara, R. Neural representations of faces are tuned to eye movements. J. Neurosci. 39, 4106 (2019).
doi: 10.1523/JNEUROSCI.2968-18.2019
Mehoudar, E., Arizpe, J., Baker, C. I. & Yovel, G. Faces in the eye of the beholder: Unique and stable eye scanning patterns of individual observers. J. Vis. 14, 6 (2014).
pubmed: 25057839
pmcid: 4062043
doi: 10.1167/14.7.6
Arizpe, J., Walsh, V., Yovel, G. & Baker, C. I. The categories, frequencies, and stability of idiosyncratic eye-movement patterns to faces. Vis. Res. 141, 191–203 (2017).
pubmed: 27940212
doi: 10.1016/j.visres.2016.10.013
Or, C. C. F., Peterson, M. F. & Eckstein, M. P. Initial eye movements during face identification are optimal and similar across cultures. J. Vis. 15, 1–25 (2015).
pmcid: 4633035
doi: 10.1167/15.13.12
Yitzhak, N., Pertzov, Y., Guy, N. & Aviezer, H. Many Ways to See Your Feelings: Successful Facial Expression Recognition Occurs With Diverse Patterns of Fixation Distributions. Emotion 22, 844–860 (2022).
pubmed: 32658507
doi: 10.1037/emo0000812
Hsiao, J. H., Lan, H., Zheng, Y. & Chan, A. B. Eye movement analysis with hidden Markov models (EMHMM) with co-clustering. Behav. Res. Methods 53, 2473–2486 (2021).
pubmed: 33929699
pmcid: 8613150
doi: 10.3758/s13428-021-01541-5
Lao, J., Miellet, S., Pernet, C., Sokhn, N. & Caldara, R. iMap4: An open source toolbox for the statistical fixation mapping of eye movement data with linear mixed modeling. Behav. Res. Methods 49, 559–575 (2017).
pubmed: 27142836
doi: 10.3758/s13428-016-0737-x
Blais, C., Fiset, D., Roy, C., Régimbald, C. S. & Gosselin, F. Eye fixation patterns for categorizing static and dynamic facial Expressions. Emotion 17, 1107 (2017).
pubmed: 28368152
doi: 10.1037/emo0000283
Blais, C., Roy, C., Fiset, D., Arguin, M. & Gosselin, F. The eyes are not the window to basic emotions. Neuropsychologia 50, 2830–2838 (2012).
pubmed: 22974675
doi: 10.1016/j.neuropsychologia.2012.08.010
Calvo, M. G. & Nummenmaa, L. Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cogn. Emot. 30, 1081–1106 (2016).
pubmed: 26212348
doi: 10.1080/02699931.2015.1049124
Fiorentini, C. & Viviani, P. Is there a dynamic advantage for facial expressions?. J. Vis. 11, 17–17 (2011).
pubmed: 21427208
doi: 10.1167/11.3.17
Gold, J. M. et al. The Efficiency of Dynamic and Static Facial Expression Recognition. J. Vis. 13, 23 (2013).
pubmed: 23620533
pmcid: 3666543
doi: 10.1167/13.5.23
Bernstein, M. & Yovel, G. Two neural pathways of face processing: A critical evaluation of current models. Neurosci. Biobehav. Rev. 55, 536–546 (2015).
pubmed: 26067903
doi: 10.1016/j.neubiorev.2015.06.010
Duchaine, B. & Yovel, G. A Revised Neural Framework for Face Processing. Annu. Rev. Vis. Sci. 1, 393–416 (2015).
pubmed: 28532371
doi: 10.1146/annurev-vision-082114-035518
Hoffmann, H., Traue, H. C., Bachmayr, F. & Kessler, H. Perceived realism of dynamic facial expressions of emotion: Optimal durations for the presentation of emotional onsets and offsets. Cogn. Emot. 24, 1369–1376 (2010).
doi: 10.1080/02699930903417855
Wingenbach, T. S. H., Ashwin, C. & Brosnan, M. Validation of the Amsterdam Dynamic Facial Expression Set ’ Bath Intensity Variations (ADFES-BIV): A Set of Videos Expressing Low, Intermediate, and High Intensity Emotions. PLoS One https://doi.org/10.1073/pnas.2201380119 (2016).
doi: 10.1073/pnas.2201380119
pubmed: 27977795
pmcid: 5158084
Binetti, N. et al. Genetic algorithms reveal profound individual differences in emotion recognition. Proc. Natl. Acad. Sci. https://doi.org/10.1073/pnas.2201380119 (2022).
doi: 10.1073/pnas.2201380119
pubmed: 36322724
pmcid: 9659399
de Lissa, P. et al. Rapid saccadic categorization of other-race faces. J. Vis. 21, 1–17 (2021).
pubmed: 34724530
pmcid: 8572436
doi: 10.1167/jov.21.12.1
Coutrot, A., Hsiao, J. H. & Chan, A. B. Scanpath modeling and classification with hidden Markov models. Behav. Res. Methods 50, 362–379 (2018).
pubmed: 28409487
doi: 10.3758/s13428-017-0876-8
Hsiao, J. H., An, J., Zheng, Y. & Chan, A. B. Do portrait artists have enhanced face processing abilities? Evidence from hidden Markov modeling of eye movements. Cognition 211, 104616 (2021).
pubmed: 33592393
doi: 10.1016/j.cognition.2021.104616
Rodger, H., Sokhn, N., Lao, J., Liu, Y. & Caldara, R. Developmental eye movement strategies for decoding facial expressions of emotion. J. Exp. Child Psychol. 229, 105622 (2023).
pubmed: 36641829
doi: 10.1016/j.jecp.2022.105622
Vaidya, A. R., Jin, C. & Fellows, L. K. Eye spy : The predictive value of fixation patterns in detecting subtle and extreme emotions from faces. Cognition 133, 443–456 (2014).
pubmed: 25151253
doi: 10.1016/j.cognition.2014.07.004
Smith, M. L., Cottrell, G. W., Gosselin, F. & Schyns, P. G. Transmitting and decoding facial expressions. Psychol. Sci. 16, 184 (2005).
pubmed: 15733197
doi: 10.1111/j.0956-7976.2005.00801.x
Nusseck, M., Cunningham, D. W., Wallraven, C. & Bülthoff, H. H. The contribution of different facial regions to the recognition of conversational expressions. J. Vis. 8, 1–23 (2008).
doi: 10.1167/8.8.1
Beaudry, O., Roy-Charland, A., Perron, M., Cormier, I. & Tapp, R. Featural processing in recognition of emotional facial expressions. Cogn. Emot. 28, 416–432 (2014).
pubmed: 24047413
doi: 10.1080/02699931.2013.833500
Schurgin, M. W. et al. Eye movements during emotion recognition in faces. J. Vis. 14, 14 (2014).
pubmed: 25406159
doi: 10.1167/14.13.14
Calvo, M. G., Fernández-Martín, A., Gutiérrez-García, A. & Lundqvist, D. Selective eye fixations on diagnostic face regions of dynamic emotional expressions: KDEF-dyn database. Sci. Rep. https://doi.org/10.1038/s41598-018-35259-w (2018).
doi: 10.1038/s41598-018-35259-w
pubmed: 30451919
pmcid: 6242984
Willenbockel, V. et al. Controlling low-level image properties: The SHINE toolbox. Behav. Res. Methods 42, 671–684 (2010).
pubmed: 20805589
doi: 10.3758/BRM.42.3.671
Richoz, A. R., Lao, J., Pascalis, O. & Caldara, R. Tracking the recognition of static and dynamic facial expressions of emotion across the life span. J. Vis. 18, 1–27 (2018).
Brainard, D. H. The Psychophysics Toolbox. Spat. Vi. 10, 433–436 (1997).
Kleiner, M. et al. What’s new in psychtoolbox-3. Perception 36, 1–16 (2007).
Cornelissen, F. W., Peters, E. M. & Palmer, J. The Eyelink Toolbox: Eye tracking with MATLAB and the Psychophysics Toolbox. Behav. Res. Methods Instr. Comput. 34, 613–617 (2002).
doi: 10.3758/BF03195489
Nyström, M. & Holmqvist, K. An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data. Behav. Res. Methods 42, 188–204 (2010).
pubmed: 20160299
doi: 10.3758/BRM.42.1.188
Xiao, N. G. & Lee, K. iTemplate: A template-based eye movement data analysis approach. Behav. Res. Methods 50, 2388–2398 (2018).
pubmed: 29423736
doi: 10.3758/s13428-018-1015-x
Chuk, T., Chan, A. B. & Hsiao, J. H. Understanding eye movements in face recognition using hidden Markov models. J. Vis. 14, 8 (2014).
pubmed: 25228627
doi: 10.1167/14.11.8
Kanan, C., Bseiso, D. N. F., Ray, N. A., Hsiao, J. H. & Cottrell, G. W. Humans have idiosyncratic and task-specific scanpaths for judging faces. Vis. Res. 108, 67–76 (2015).
pubmed: 25641371
doi: 10.1016/j.visres.2015.01.013
Coutrot, A., Binetti, N., Harrison, C., Mareschal, I. & Johnston, A. Face exploration dynamics differentiate men and women. J. Vis. 14, 16 (2016).
doi: 10.1167/16.14.16
Chan, C. Y. H., Chan, A. B., Lee, T. M. C. & Hsiao, J. H. Eye-movement patterns in face recognition are associated with cognitive decline in older adults. Psychon. Bull. Rev. 25, 2200–2207 (2018).
pubmed: 29313315
doi: 10.3758/s13423-017-1419-0
Zhang, J., Chan, A. B., Lau, E. Y. Y. & Hsiao, J. H. Individuals with insomnia misrecognize angry faces as fearful faces while missing the eyes: An eye-tracking study. Sleep https://doi.org/10.1093/sleep/zsy220 (2019).
doi: 10.1093/sleep/zsy220
pubmed: 31587046
pmcid: 6783889
R: A Language and Environment for Statistical Computing. Preprint at (2022).