Problems in the perception of emotional material, in particular deficits in the recognition of negative stimuli, have been demonstrated in schizophrenia including in first-episode samples. However, it is largely unknown if emotion recognition impairment is present in people with subthreshold psychotic symptoms. Here, we examined the capacity to recognize facially expressed emotion and affective prosody in 79 individuals at ultra high-risk for psychosis, 30 clinically stable individuals with first-episode schizophrenia assessed as outpatients during the early recovery phase of illness, and 30 unaffected healthy control subjects. We compared (1) scores for a combined fear-sadness aggregate index across face and voice modalities, (2) summary scores of specific emotions across modalities, and (3) scores for specific emotions for each sensory modality. Findings supported deficits in recognition of fear and sadness across both modalities for the clinical groups (the ultra high-risk and first-episode group) as compared with the healthy controls. Furthermore, planned contrasts indicated that compared with the healthy control subjects, both clinical groups had a significant deficit for fear and sadness recognition in faces and for anger recognition in voices. Specific impairments in emotion recognition may be apparent in people at clinical high-risk for schizophrenia before the full expression of psychotic illness. The results suggest a trait deficit and an involvement of the amygdala in the pathology of ultra high-risk states.