Our first question draws on evidence demonstrating a nonuniform spatial distribution of reward effects across the visual cortex ( 21) and a sharpening of sensory representations through suppression of redundant signals elicited by reward/information predictive signals ( 10, 22). Owing to the low temporal resolution of functional MRI (fMRI) signals, here we manipulated only the spatial overlap of stimuli while keeping their temporal alignment constant (with simultaneous presentation). On this basis, we hypothesized that the effect of rewarded sounds on vision should be strongest when visual and auditory stimuli overlap in time and in space. Previous studies have shown that interactions between sensory modalities are strongest when stimuli are presented simultaneously and at contiguous spatial locations ( 1, 2, 11– 13), especially at the earliest stages of processing ( 2, 11). Furthermore, simultaneous and spatially overlapping stimuli are more likely to have a common source and thus a shared reward association. (No feedback was presented about the accuracy of perceptual judgments, and performance on the orientation discrimination task was not related to the payment subjects received at the end of the trial.) We hypothesized that psychophysical measures of visual orientation judgment ( d′ and performance accuracy) would show improvement in the presence of a sound with high (compared with low) reward association. Critically, these tones were task-irrelevant and bore no relationship either to the orientation of the visual stimulus or to the outcome of the trial. To study cross-modal transfer of value, we designed a visual orientation discrimination task in which visual stimuli were presented concurrently with one of two arbitrary pure tones previously associated with different levels of monetary reward. For example, one can easily imagine the importance of increased sensitivity to change in the visual scene to the concurrent sound of a tiger roaring or the mating call of a conspecific. This is important both because of what it reveals about the nature of cross-modal associations and because it might constitute an important perceptual mechanism in its own right. In particular, whether an association with increased reward, known to increase the accuracy of perceptual processing within that sensory modality ( 5– 10), can transfer between modalities for concurrently presented stimuli is unclear.

Surprisingly, we know little about how reward associations in one sensory modality influence processing in other modlities. One feature of environmental stimuli important for adaptive behavior is their rewarding or reward-predicting properties. This phenomenon has been the focus of a large and burgeoning theoretical and experimental literature ( 1– 4). The world is structured such that objects or events that cause sensations in one sensory modality influence those in another modality, a mechanism that underlies the near-ubiquitous phenomenon of multisensory interaction ( 1). We suggest that multisensory processing areas may mediate the transfer of value signals across senses. Our findings demonstrate a value-driven cross-modal interaction that affects perception and stimulus encoding, with a resemblance to well-described modulatory effects of attention. Univariate analysis showed that reward associations modulated responses in regions associated with multisensory processing in which the strength of modulation was a better predictor of the magnitude of the behavioral effect than the modulation in classical reward regions. This increased sensitivity correlates with a change in stimulus representation in the visual cortex, indexed by increased multivariate decoding accuracy in simultaneously acquired functional MRI data. Here we show that auditory stimuli with high-reward associations increase the sensitivity of visual perception, even when sounds and reward associations are both irrelevant for the visual task. What is unknown is whether reward associations in one sensory modality influence perception of stimuli in another modality. An important feature of many perceptual stimuli is their reward-predicting properties, the utilization of which is essential for adaptive behavior. Cross-modal interactions are very common in perception.

Cross modal perception definition