We used functional magnetic resonance imaging (fMRI) to investigate the neural systems responding to the sight and to the sound of an action. Subjects saw a video of paper tearing in silence (V), heard the sound of paper tearing (A), and saw and heard the action simultaneously (A + V). Compared to a non-action control stimulus, we found that hearing action sounds (A) activated the anterior inferior frontal gyrus and middle frontal gyrus in addition to primary auditory cortex. The anterior inferior frontal gyrus, which is known to be activated by environmental sounds, also seems to be involved in recognizing actions by sound. Consistent with previous research, seeing an action video (V) compared with seeing a non-action video activated the premotor cortex, intraparietal cortex, and the pars opercularis of the inferior frontal gyrus. An A + V facilitation effect was found in the ventral premotor cortex on the border of areas 44, 6, 3a, and 3b for the action stimuli but not for the control stimuli. This region may be involved in integrating multimodal information about actions. These data provide evidence that the ventral premotor cortex may provide an action representation that abstracts across both agency (self and other) and sensory modality (hearing and seeing). This function may be an important precursor of language functions.