Using functional magnetic resonance imaging (fMRI), we investigated whether the response of auditory and visual cortex was modulated by attending selectively to either heard or seen numbers presented simultaneously. Alternating attention between modalities modulated fMRI signal within the corresponding sensory cortex. This study provides evidence that attention acts locally during early auditory cognitive sensory processing, and that modulation of auditory and visual sensory cortex by attention is modality-dependent.