Crossmodal correspondence of elevation/pitch and size/pitch is driven by real-world features.
Multisensory integration
Neural mechanisms
Scene perception
Journal
Attention, perception & psychophysics
ISSN: 1943-393X
Titre abrégé: Atten Percept Psychophys
Pays: United States
ID NLM: 101495384
Informations de publication
Date de publication:
26 Oct 2024
26 Oct 2024
Historique:
accepted:
09
10
2024
medline:
27
10
2024
pubmed:
27
10
2024
entrez:
27
10
2024
Statut:
aheadofprint
Résumé
Crossmodal correspondences are consistent associations between sensory features from different modalities, with some theories suggesting they may either reflect environmental correlations or stem from innate neural structures. This study investigates this question by examining whether retinotopic or representational features of stimuli induce crossmodal congruency effects. Participants completed an auditory pitch discrimination task paired with visual stimuli varying in their sensory (retinotopic) or representational (scene integrated) nature, for both the elevation/pitch and size/pitch correspondences. Results show that only representational visual stimuli produced crossmodal congruency effects on pitch discrimination. These results support an environmental statistics hypothesis, suggesting crossmodal correspondences rely on real-world features rather than on sensory representations.
Identifiants
pubmed: 39461934
doi: 10.3758/s13414-024-02975-7
pii: 10.3758/s13414-024-02975-7
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2024. The Author(s).
Références
Barbiere, J. M., Vidal, A., & Zellner, D. A. (2007). The color of music: Correspondence through emotion. Empirical Studies of the Arts, 25(2), 193–208.
doi: 10.2190/A704-5647-5245-R47P
Ben-Artzi, E., & Marks, L. E. (1995). Visual-auditory interaction in speeded classification: Role of stimulus difference. Perception & Psychophysics, 57(8), 1151–1162.
doi: 10.3758/BF03208371
Bernstein, I. H., & Edelstein, B. A. (1971). Effects of some variations in auditory input upon visual choice reaction time. Journal of Experimental Psychology, 87(2), 241–247.
pubmed: 5542226
doi: 10.1037/h0030524
Bien, N., Ten Oever, S., Goebel, R., & Sack, A. T. (2012). The sound of size: Crossmodal binding in pitch-size synesthesia: A combined TMS. EEG and Psychophysics Study. Neuroimage, 59(1), 663–672.
pubmed: 21787871
Brainard, D. H., & Vision, S. (1997). The psychophysics toolbox. Spatial Vision, 10(4), 433–436.
pubmed: 9176952
doi: 10.1163/156856897X00357
Brunetti, R., Indraccolo, A., Del Gatto, C., Spence, C., & Santangelo, V. (2018). Are crossmodal correspondences relative or absolute? Sequential effects on speeded classification. Attention, Perception, & Psychophysics, 80, 527–534.
doi: 10.3758/s13414-017-1445-z
Carnevale, M. J., & Harris, L. R. (2016). Which direction is up for a high pitch? Multisensory Research, 29(1–3), 113–132.
pubmed: 27311293
doi: 10.1163/22134808-00002516
Chen, J., Sperandio, I., Henry, M. J., & Goodale, M. A. (2019). Changing the real viewing distance reveals the temporal evolution of size constancy in visual cortex. Current Biology, 29(13), 2237–2243.
pubmed: 31257140
doi: 10.1016/j.cub.2019.05.069
Chiou, R., & Rich, A. N. (2012). Cross-modality correspondence between pitch and spatial location modulates attentional orienting. Perception, 41(3), 339–353.
pubmed: 22808586
doi: 10.1068/p7161
Clark, H. H., & Brownell, H. H. (1976). Position, direction, and their perceptual integrality. Perception & Psychophysics, 19, 328–334.
doi: 10.3758/BF03204238
Connell, L. (2019). What have labels ever done for us? The linguistic shortcut in conceptual processing. Language, Cognition and Neuroscience, 34(10), 1308–1318.
doi: 10.1080/23273798.2018.1471512
Deroy, O., Fernandez-Prieto, I., Navarra, J., & Spence, C. (2018). Unravelling the paradox of spatial pitch. In T. L. Hubbard (Ed.), Spatial Biases in Perception and Cognition (pp. 77–93). Cambridge University Press.
doi: 10.1017/9781316651247.006
Diener, H. C., & Dichgans, J. (1988). On the role of vestibular, visual and somatosensory information for dynamic postural control in humans. Progress in Brain Research, 76, 253–262.
pubmed: 3064150
doi: 10.1016/S0079-6123(08)64512-4
Duhamel, J. R., Bremmer, F., Ben Hamed, S., & Graf, W. (1997). Spatial invariance of visual receptive fields in parietal cortex neurons. Nature, 389(6653), 845–848.
pubmed: 9349815
doi: 10.1038/39865
Ernst, M. O. (2007). Learning to integrate arbitrary signals from vision and touch. Journal of Vision, 7(5), 1–14.
doi: 10.1167/7.5.7
Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10(1), 1–12.
Fernandez-Prieto, I., Spence, C., Pons, F., & Navarra, J. (2017). Does language influence the vertical representation of auditory pitch and loudness? i-Perception, 8(3), 2041669517716183.
Fifer, J. M., Barutchu, A., Shivdasani, M. N., & Crewther, S. G. (2013). Verbal and novel multisensory associative learning in adults. F1000 Research, 2, 34–34. https://doi.org/10.12688/f1000research.2-34.v2
Gallace, A., & Spence, C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size. Perception & Psychophysics, 68, 1191–1203.
doi: 10.3758/BF03193720
Galletti, C., Battaglini, P. P., & Fattori, P. (1993). Parietal neurons encoding spatial locations in craniotopic coordinates. Experimental Brain Research, 96, 221–229.
pubmed: 8270019
doi: 10.1007/BF00227102
Hamilton-Fletcher, G., Pisanski, K., Reby, D., Stefańczyk, M., Ward, J., & Sorokowska, A. (2018). The role of visual experience in the emergence of cross-modal correspondences. Cognition, 175, 114–121.
pubmed: 29502009
doi: 10.1016/j.cognition.2018.02.023
Helbig, H. B., & Ernst, M. O. (2008). Visual-haptic cue weighting is independent of modality-specific attention. Journal of Vision, 8(1), 1–16.
doi: 10.1167/8.1.21
Hibbard, P. B., Goutcher, R., Hornsey, R. L., Hunter, D. W., & Scarfe, P. (2023). Luminance contrast provides metric depth information. Royal Society Open Science, 10(2), 220567.
pubmed: 36816842
pmcid: 9929495
doi: 10.1098/rsos.220567
Kanai, R., & Verstraten, F. A. (2006). Visual transients reveal the veridical position of a moving object. Perception, 35(4), 453–460.
pubmed: 16700288
doi: 10.1068/p5443
Koenderink, J. J., Van Doorn, A. J., Kappers, A. M., & Todd, J. T. (2001). Ambiguity and the ‘mental eye’ in pictorial relief. Perception, 30(4), 431–448.
pubmed: 11383191
doi: 10.1068/p3030
Korzeniowska, A. T., Root-Gutteridge, H., Simner, J., & Reby, D. (2019). Audio–visual crossmodal correspondences in domestic dogs (Canis familiaris). Biology Letters, 15(11), 20190564.
pubmed: 31718513
pmcid: 6892510
doi: 10.1098/rsbl.2019.0564
Korzeniowska, A. T., Simner, J., Root-Gutteridge, H., & Reby, D. (2022). High-pitch sounds small for domestic dogs: Abstract crossmodal correspondences between auditory pitch and visual size. Royal Society Open Science, 9(2), 211647.
pubmed: 35154798
pmcid: 8825985
doi: 10.1098/rsos.211647
Kovic, V., Plunkett, K., & Westermann, G. (2010). The shape of words in the brain. Cognition, 114(1), 19–28.
pubmed: 19828141
doi: 10.1016/j.cognition.2009.08.016
Levitan, C. A., Charney, S., Schloss, K. B., & Palmer, S. E. (2015, July). The smell of jazz: Crossmodal correspondences between music, odor, and emotion. In CogSci (Vol. 1, pp. 1326–1331).
Loconsole, M., Stancher, G., & Versace, E. (2023). Crossmodal association between visual and acoustic cues in a tortoise (Testudo hermanni). Biology Letters, 19(7), 20230265.
pubmed: 37465911
pmcid: 10354690
doi: 10.1098/rsbl.2023.0265
Longo, M. R., Azañón, E., & Haggard, P. (2010). More than skin deep: Body representation beyond primary somatosensory cortex. Neuropsychologia, 48(3), 655–668.
pubmed: 19720070
doi: 10.1016/j.neuropsychologia.2009.08.022
Ludwig, V. U., Adachi, I., & Matsuzawa, T. (2011). Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans. Proceedings of the National Academy of Sciences, 108(51), 20661–20665.
doi: 10.1073/pnas.1112605108
Marks, L. E. (1987). On cross-modal similarity: Auditory–visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception and Performance, 13(3), 384–394.
pubmed: 2958587
McEwan, J., Kritikos, A., & Zeljko, M. (2024). Involvement of the superior colliculi in crossmodal correspondences. Attention, Perception, & Psychophysics, 86(3), 931–941.
doi: 10.3758/s13414-024-02866-x
Melara, R. D. (1989). Dimensional interaction between color and pitch. Journal of Experimental Psychology: Human Perception and Performance, 15(1), 69–79.
pubmed: 2522534
Mondloch, C. J., & Maurer, D. (2004). Do small white balls squeak? Pitch-object correspondences in young children. Cognitive, Affective, & Behavioral Neuroscience, 4(2), 133–136.
doi: 10.3758/CABN.4.2.133
Motoki, K., Marks, L. E., & Velasco, C. (2023). Reflections on cross-modal correspondences: Current understanding and issues for future research. Multisensory Research, 37(1), 1–23.
pubmed: 37963487
doi: 10.1163/22134808-bja10114
Noviello, S., Songhorabadi, S. K., Deng, Z., Zheng, C., Chen, J., Pisani, A., Franchin, E., Pierotti, E., Tonolli, E., Monaco, S., Renoult, L., & Sperandio, I. (2024). Temporal features of size constancy for perception and action in a real-world setting: A combined EEG-kinematics study. Neuropsychologia, 193, 108746.
pubmed: 38081353
doi: 10.1016/j.neuropsychologia.2023.108746
Ozturk, O., Krehm, M., & Vouloumanos, A. (2013). Sound symbolism in infancy: Evidence for sound–shape cross-modal correspondences in 4-month-olds. Journal of Experimental Child Psychology, 114(2), 173–186.
pubmed: 22960203
doi: 10.1016/j.jecp.2012.05.004
Palmer, S. E., Schloss, K. B., Xu, Z., & Prado-León, L. R. (2013). Music–color associations are mediated by emotion. Proceedings of the National Academy of Sciences, 110(22), 8836–8841.
Parise, C. V. (2016). Crossmodal correspondences: Standing issues and experimental guidelines. Multisensory Research, 29(1–3), 7–28.
pubmed: 27311289
doi: 10.1163/22134808-00002502
Parise, C. V., & Spence, C. (2012). Audiovisual crossmodal correspondences and sound symbolism: A study using the implicit association test. Experimental Brain Research, 220, 319–333.
pubmed: 22706551
doi: 10.1007/s00221-012-3140-6
Parise, C., & Spence, C. (2008). Synesthetic congruency modulates the temporal ventriloquism effect. Neuroscience Letters, 442(3), 257–261.
pubmed: 18638522
doi: 10.1016/j.neulet.2008.07.010
Parise, C. V., Knorre, K., & Ernst, M. O. (2014). Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences, 111(16), 6104–6108.
doi: 10.1073/pnas.1322705111
Pelli, D. G. (1997). The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision, 10(4), 437–442.
pubmed: 9176953
doi: 10.1163/156856897X00366
Perdreau, F., & Cavanagh, P. (2013). Is artists’ perception more veridical? Frontiers in Neuroscience, 7, 6.
pubmed: 23386809
pmcid: 3560385
doi: 10.3389/fnins.2013.00006
Pitzalis, S., Bozzacchi, C., Bultrini, A., Fattori, P., Galletti, C., & Di Russo, F. (2013a). Parallel motion signals to the medial and lateral motion areas V6 and MT+. NeuroImage, 67, 89–100.
pubmed: 23186916
doi: 10.1016/j.neuroimage.2012.11.022
Pitzalis, S., Fattori, P., & Galletti, C. (2013b). The functional role of the medial motion area V6. Frontiers in Behavioral Neuroscience, 6, 91.
pubmed: 23335889
pmcid: 3546310
doi: 10.3389/fnbeh.2012.00091
Qian, J., & Yazdanbakhsh, A. (2015). A neural model of distance-dependent percept of object size constancy. PLoS ONE, 10(7), e0129377.
pubmed: 26132106
pmcid: 4489391
doi: 10.1371/journal.pone.0129377
Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia–a window into perception, thought and language. Journal of Consciousness Studies, 8(12), 3–34.
Riordan, B., & Jones, M. N. (2011). Redundancy in perceptual and linguistic experience: Comparing feature-based and distributional models of semantic representation. Topics in Cognitive Science, 3(2), 303–345.
pubmed: 25164298
doi: 10.1111/j.1756-8765.2010.01111.x
Samonds, J. M., Potetz, B. R., & Lee, T. S. (2012). Relative luminance and binocular disparity preferences are correlated in macaque primary visual cortex, matching natural scene statistics. Proceedings of the National Academy of Sciences, 109(16), 6313–6318.
doi: 10.1073/pnas.1200125109
Schifferstein, H. N., & Tanudjaja, I. (2004). Visualising fragrances through colours: The mediating role of emotions. Perception, 33(10), 1249–1266.
pubmed: 15693669
doi: 10.1068/p5132
Sciortino, P., & Kayser, C. (2023). Steady state visual evoked potentials reveal a signature of the pitch-size crossmodal association in visual cortex. NeuroImage, 273, 120093.
pubmed: 37028733
doi: 10.1016/j.neuroimage.2023.120093
Sloutsky, V. M., & Fisher, A. V. (2012). Linguistic labels: Conceptual markers or object features? Journal of Experimental Child Psychology, 111(1), 65–86.
pubmed: 21903223
doi: 10.1016/j.jecp.2011.07.007
Smith, C. P., & Reynolds, R. F. (2017). Vestibular feedback maintains reaching accuracy during body movement. The Journal of Physiology, 595(4), 1339–1349.
pubmed: 27730646
doi: 10.1113/JP273125
Smith, L. B., & Sera, M. D. (1992). A developmental analysis of the polar structure of dimensions. Cognitive Psychology, 24(1), 99–142.
pubmed: 1537233
doi: 10.1016/0010-0285(92)90004-L
Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73, 971–995.
doi: 10.3758/s13414-010-0073-7
Spence, C. (2019). On the relative nature of (pitch-based) crossmodal correspondences. Multisensory Research, 32(3), 235–265.
pubmed: 31059485
doi: 10.1163/22134808-20191407
Spence, C. (2020a). Assessing the role of emotional mediation in explaining crossmodal correspondences involving musical stimuli. Multisensory Research, 33(1), 1–29.
pubmed: 31648195
doi: 10.1163/22134808-20191469
Spence, C. (2020b). Simple and complex crossmodal correspondences involving audition. Acoustical Science and Technology, 41(1), 6–12.
doi: 10.1250/ast.41.6
Sperandio, I., & Chouinard, P. A. (2015). The mechanisms of size constancy. Multisensory Research, 28(3–4), 253–283.
pubmed: 26288899
doi: 10.1163/22134808-00002483
Sperandio, I., Chouinard, P. A., & Goodale, M. A. (2012). Retinotopic activity in V1 reflects the perceived and not the retinal size of an afterimage. Nature Neuroscience, 15(4), 540–542.
pubmed: 22406550
doi: 10.1038/nn.3069
Spence, C., & Deroy, O. (2013). How automatic are crossmodal correspondences? Consciousness and Cognition, 22(1), 245–260.
pubmed: 23370382
doi: 10.1016/j.concog.2012.12.006
Spence, C., & Di Stefano, N. (2022). Coloured hearing, colour music, colour organs, and the search for perceptually meaningful correspondences between colour and sound. i-Perception, 13(3), 1–42.
Stevens, S. S. (1957). On the psychophysical law. Psychological Review, 64(3), 153–181.
pubmed: 13441853
doi: 10.1037/h0046162
Tanaka, S., & Fujita, I. (2015). Computation of object size in visual cortical area V4 as a neural basis for size constancy. Journal of Neuroscience, 35(34), 12033–12046.
pubmed: 26311782
doi: 10.1523/JNEUROSCI.2665-14.2015
Turoman, N., Velasco, C., Chen, Y. C., Huang, P. C., & Spence, C. (2018). Symmetry and its role in the crossmodal correspondence between shape and taste. Attention, Perception, & Psychophysics, 80, 738–751.
doi: 10.3758/s13414-017-1463-x
Vanston, J. E., Boehm, A. E., Tuten, W. S., & Roorda, A. (2023). It’s not easy seeing green: The veridical perception of small spots. Journal of Vision, 23(5), 2–2.
pubmed: 37133838
pmcid: 10166115
doi: 10.1167/jov.23.5.2
Walker, L., & Walker, P. (2016). Cross-sensory mapping of feature values in the size–brightness correspondence can be more relative than absolute. Journal of Experimental Psychology: Human Perception and Performance, 42(1), 138–150.
pubmed: 26322684
Walker, L., Walker, P., & Francis, B. (2012). A common scheme for cross-sensory correspondences across stimulus domains. Perception, 41(10), 1186–1192.
pubmed: 23469700
doi: 10.1068/p7149
Walker, P., Bremner, J. G., Lunghi, M., Dolscheid, S., Barba, D., & B., & Simion, F. (2018). Newborns are sensitive to the correspondence between auditory pitch and visuospatial elevation. Developmental Psychobiology, 60(2), 216–223.
pubmed: 29355921
doi: 10.1002/dev.21603
Walker, P., Bremner, J. G., Mason, U., Spring, J., Mattock, K., Slater, A., & Johnson, S. P. (2010). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychological Science, 21(1), 21–25.
pubmed: 20424017
doi: 10.1177/0956797609354734
Walsh, V. (2003). A theory of magnitude: Common cortical metrics of time, space and quantity. Trends in Cognitive Sciences, 7(11), 483–488.
pubmed: 14585444
doi: 10.1016/j.tics.2003.09.002
Wells, A. (1980). Music and visual color: A proposed correlation. Leonardo, 13(2), 101–107.
doi: 10.2307/1577978
Walsh, V., & Kulikowski, J. (Eds.). (1998). Perceptual constancy: Why things look as they do. Cambridge University Press.
Zeljko, M., Grove, P. M., & Kritikos, A. (2021). The lightness/pitch crossmodal correspondence modulates the Rubin face/vase perception. Multisensory Research, 34(7), 763–783.
doi: 10.1163/22134808-bja10054
Zeljko, M., Kritikos, A., & Grove, P. M. (2019). Lightness/pitch and elevation/pitch crossmodal correspondences are low-level sensory effects. Attention, Perception, & Psychophysics, 81, 1609–1623.
doi: 10.3758/s13414-019-01668-w