The role of crossmodal correspondences in human vision

Abstract

As we interact with the world, the unified percept generated from our different senses is almost effortless and yet it is a substantial challenge for the brain. Multisensory research has shown that people typically pair particular features from one modality with those of another. These mappings have been termed crossmodal correspondences. In this thesis I examine three main empirical questions relating to crossmodal correspondences and their effects on human vision, using psychophysical methods. The first aim was to determine if crossmodal correspondences can reflect stimulus-specific mappings between featural dimensions of different modalities. This was investigated using a recently reported correspondence, between auditory modulation rate and visual spatial frequency, and the correspondence between tactile and visual spatial frequency. The experiments in Chapters 4 and 5 used a series of visual search tasks to demonstrate that the mapping in these correspondences is broad and relative, rather than stimulus-specific. The second aim was to investigate whether correspondences can produce attentional capture. This question was also examined using a series of visual search tasks. The results clearly demonstrate that even when participants' top-down goals match any existing bottom-up capture, stimulus-specific effects do not occur. Furthermore, when the crossmodal matches are made irrelevant to the task, a matching sound or tactile stimulus has no effect on search performance. This indicates that crossmodal matching affects visual selection through top-down guidance not bottom-up capture. The final aim was to determine whether correspondences can affect crossmodal binding. In Chapter 6, a well-known multisensory binding effect `temporal ventriloquism' was used to show that incongruent crossmodal pairings of auditory pitch and visual elevation impair crossmodal binding. However, when the auditory stimuli were made predictable the effect was eliminated, showing that the effect of crossmodal congruency is dependent on the saliency of the crossmodal mapping. In the last chapter, the results of the three experimental chapters are discussed together in the context of the role of crossmodal correspondences in unisensory and multisensory processing. Directions for future research into crossmodal correspondences are suggested and the role of causality in multisensory processing is discussed in relation to crossmodal correspondences.