Auditory stimuli degrade visual performance in virtual reality.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
23 07 2020
23 07 2020
Historique:
received:
15
04
2020
accepted:
07
07
2020
entrez:
25
7
2020
pubmed:
25
7
2020
medline:
25
7
2020
Statut:
epublish
Résumé
We report an auditory effect of visual performance degradation in a virtual reality (VR) setting, where the viewing conditions are significantly different from previous studies. With the presentation of temporally congruent but spatially incongruent sound, we can degrade visual performance significantly at detection and recognition levels. We further show that this effect is robust to different types and locations of both auditory and visual stimuli. We also analyze participants behavior with an eye tracker to study the underlying cause of the degradation effect. We find that the performance degradation occurs even in the absence of saccades towards the sound source, during normal gaze behavior. This suggests that this effect is not caused by oculomotor phenomena, but rather by neural interactions or attentional shifts.
Identifiants
pubmed: 32703981
doi: 10.1038/s41598-020-69135-3
pii: 10.1038/s41598-020-69135-3
pmc: PMC7378072
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
12363Subventions
Organisme : European Research Council
ID : 682080 (CHAMELEON)
Pays : International
Références
Van der Stoep, N., Serino, A., Farnè, A., Di Luca, M. & Spence, C. Depth: the forgotten dimension in multisensory research. Multisens. Res. 29, 493–524 (2016).
doi: 10.1163/22134808-00002525
Spence, C., Lee, J. & Van der Stoep, N. Responding to sounds from unseen locations: crossmodal attentional orienting in response to sounds presented from the rear. Eur. J. Neurosci. 51, 1137–1150 (2017).
doi: 10.1111/ejn.13733
Volkmann, F. C., Riggs, L. A. & Moore, R. K. Eyeblinks and visual suppression. Science 207, 900–902 (1980).
doi: 10.1126/science.7355270
Matin, E. Saccadic suppression: a review and an analysis. Psychol. Bull. 81, 899 (1974).
doi: 10.1037/h0037368
Ross, J., Morrone, M. C., Goldberg, M. E. & Burr, D. C. Changes in visual perception at the time of saccades. Trends Neurosci. 24, 113–121 (2001).
doi: 10.1016/S0166-2236(00)01685-4
Teichert, M. & Bolz, J. How senses work together: Cross-modal interactions between primary sensory cortices. Neural Plast. 2018, (2018).
Hidaka, S. & Ide, M. Sound can suppress visual perception. Sci. Rep. 5, 10483 (2015).
doi: 10.1038/srep10483
Hidaka, S., Suzuishi, Y., Ide, M. & Wada, M. Effects of spatial consistency and individual difference on touch-induced visual suppression effect. Sci. Rep. 8, 17018 (2018).
doi: 10.1038/s41598-018-35302-w
Ide, M. & Hidaka, S. Tactile stimulation can suppress visual perception. Sci. Rep. 3, 1–8 (2013).
doi: 10.1038/srep03453
Laurienti, P. J. et al. Deactivation of sensory-specific cortex by cross-modal stimuli. J. Cogn. Neurosci. 14, 420–429 (2002).
doi: 10.1162/089892902317361930
Kawashima, R., Osullivan, B. T. & Roland, P. E. Positron-emission tomography studies of cross-modality inhibition in selective attentional tasks: closing the minds eye. Proc. Natl. Acad. Sci. 92, 5969–5972 (1995).
doi: 10.1073/pnas.92.13.5969
Merabet, L. B. et al. Combined activation and deactivation of visual cortex during tactile sensory processing. J. Neurophysiol. 97, 1633–1641 (2007).
doi: 10.1152/jn.00806.2006
Ide, M., Hidaka, S., Ikeda, H. & Wada, M. Neural mechanisms underlying touch-induced visual perceptual suppression: an fmri study. Sci. Rep. 6, 1–9 (2016).
doi: 10.1038/srep37301
Sitzmann, V. et al. How do people explore virtual environments?. IEEE Trans. Vis.l Comput. Graphics 24, 1633–1642 (2018).
doi: 10.1109/TVCG.2018.2793599
Leone, L. M. & McCourt, M. E. The roles of physical and physiological simultaneity in audiovisual multisensory facilitation. i-Perception 4, 213–228 (2013).
doi: 10.1068/i0532
McDonald, J. J., Teder-SaÈlejaÈrvi, W. A. & Hillyard, S. A. Involuntary orienting to sound improves visual perception. Nature 407, 906–908 (2000).
doi: 10.1038/35038085
Rungta, A. et al. Effects of virtual acoustics on target-word identification performance in multi-talker environments. In Proceedings of the 15th ACM Symposium on Applied Perception, 1–8 (2018).
Gonzalez-Franco, M., Maselli, A., Florencio, D., Smolyanskiy, N. & Zhang, Z. Concurrent talking in immersive virtual reality: on the dominance of visual speech cues. Sci. Rep. 7, 1–11 (2017).
doi: 10.1038/s41598-016-0028-x
Siddig, A., Ragano, A., Jahromi, H. Z. & Hines, A. Fusion confusion: exploring ambisonic spatial localisation for audio-visual immersion using the mcgurk effect. In Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, 28–33 (2019).
Rothe, S., Buschek, D. & Hußmann, H. Guidance in cinematic virtual reality-taxonomy, research status and challenges. Multimodal Technol. Interact. 3, 19 (2019).
doi: 10.3390/mti3010019
Middlebrooks, J. C. & Green, D. M. Sound localization by human listeners. Annu. Rev. Psychol. 42, 135–159 (1991).
doi: 10.1146/annurev.ps.42.020191.001031
Frens, M. & Van Opstal, A. Auditory-evoked saccades in two dimensions: dynamical characteristics, influence of eye position and sound spectrum. Inf. Process. Underlying Gaze Control 12, 329 (1994).
doi: 10.1016/B978-0-08-042506-1.50037-0
Hessels, R. S., Niehorster, D. C., Kemner, C. & Hooge, I. T. Noise-robust fixation detection in eye movement data: Identification by two-means clustering (i2mc). Behav. Res. Methods 49, 1802–1823 (2017).
doi: 10.3758/s13428-016-0822-1
Battista, J., Kalloniatis, M. & Metha, A. Visual function: the problem with eccentricity. Clin. Exp. Optom. 88, 313–321 (2005).
doi: 10.1111/j.1444-0938.2005.tb06715.x
Frens, M. A., Van Opstal, A. J. & Van der Willigen, R. F. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept. Psychophys. 57, 802–816 (1995).
doi: 10.3758/BF03206796
Delong, P. et al. Invisible flashes alter perceived sound location. Sci. Rep. 8, 12376 (2018).
doi: 10.1038/s41598-018-30773-3
Shams, L., Kamitani, Y. & Shimojo, S. Illusions: what you see is what you hear. Nature 408, 788 (2000).
doi: 10.1038/35048669
Corneil, B., Van Wanrooij, M., Munoz, D. & Van Opstal, A. Auditory-visual interactions subserving goal-directed saccades in a complex scene. J. Neurophysiol. 88, 438–454 (2002).
doi: 10.1152/jn.2002.88.1.438
Holmes, N. P. & Spence, C. Multisensory integration: space, time and superadditivity. Curr. Biol. 15, R762–R764 (2005).
doi: 10.1016/j.cub.2005.08.058
Spence, C. & Driver, J. Audiovisual links in exogenous covert spatial orienting. Percept. Psychophys. 59, 1–22 (1997).
doi: 10.3758/BF03206843
Spence, C. & Ho, C. Tactile and multisensory spatial warning signals for drivers. IEEE Trans. Haptics 1, 121–129 (2008).
doi: 10.1109/TOH.2008.14
Macaluso, E., Frith, C. D. & Driver, J. Modulation of human visual cortex by crossmodal spatial attention. Science 289, 1206–1208 (2000).
doi: 10.1126/science.289.5482.1206
Spence, C. & Parise, C. Prior-entry: a review. Conscious. Cogn. 19, 364–379 (2010).
doi: 10.1016/j.concog.2009.12.001
Mozolic, J. L. et al. Cross-modal deactivations during modality-specific selective attention. BMC Neurol. 8, 35 (2008).
doi: 10.1186/1471-2377-8-35
Takemori, S. Visual suppression test. Ann. Otol. Rhinol. Laryngol. 86, 80–85 (1977).
doi: 10.1177/000348947708600113
Bailey, R., McNamara, A., Sudarsanam, N. & Grimm, C. Subtle gaze direction. ACM Trans. Graph. (TOG) 28, 100 (2009).
doi: 10.1145/1559755.1559757
Arabadzhiyska, E., Tursun, O. T., Myszkowski, K., Seidel, H.-P. & Didyk, P. Saccade landing position prediction for gaze-contingent rendering. ACM Trans. Graph. (TOG) 36, 50 (2017).
doi: 10.1145/3072959.3073642
Blake, R. A neural theory of binocular rivalry. Psychol. Rev. 96, 145 (1989).
doi: 10.1037/0033-295X.96.1.145
Simons, D. J. & Levin, D. T. Change blindness. Trends Cognit. Sci. 1, 261–267 (1997).
doi: 10.1016/S1364-6613(97)01080-2
Sun, Q. et al. Towards virtual reality infinite walking: dynamic saccadic redirection. ACM Trans. Graphics (TOG) 37, 67 (2018).
doi: 10.1145/3197517.3201294
Bolte, B. & Lappe, M. Subliminal reorientation and repositioning in immersive virtual environments using saccadic suppression. IEEE Trans. Visual Comput. Graphics 21, 545–552 (2015).
doi: 10.1109/TVCG.2015.2391851
Hecht, D., Reiner, M. & Halevy, G. Multimodal virtual environments: response times, attention, and presence. Presence Teleoper. Virtual Environm.15, 515–523 (2006).
Creten, W., Vanpeperstraete, P., Van Camp, K. & Doclo, J. An experimental study on diphasic acoustic reflex patterns in normal ears. Scand. Audiol. 5, 3–8 (1976).
doi: 10.3109/01050397609043089