Previous studies have suggested differences in the neural substrates of recognition memory when the contributions of perceptual and semantic information are manipulated. In a within-subjects design PET study, we investigated the neural correlates of the following factors: material type (objects or faces), semantic knowledge (familiar or unfamiliar items), and perceptual similarity at study and test (identical or different pictures). There was consistent material-specific lateralization in frontal and temporal lobe regions when the retrieval of different types of nonverbal stimuli was compared, with objects activating bilateral areas and faces preferentially activating the right hemisphere. Retrieval of memories for nameable, familiar items was associated with increased activation in the left ventrolateral prefrontal cortex, while memory for unfamiliar items involved occipital regions. Recognition memory for different pictures of the same item at study and test produced blood flow increase in left inferior temporal cortex. These results have implications for our understanding of the neural correlates of perceptual and semantic contributions to recognition memory.