The social motivation hypothesis posits that aberrant neural response to human faces in autism is attributable to atypical social development and consequently reduced exposure to faces. The specificity of deficits in neural specialization remains unclear, and alternative theories suggest generalized processing difficulties. The current study contrasted neural specialization for social information versus nonsocial information in 36 individuals with autism and 18 typically developing individuals matched for age, race, sex, handedness, and cognitive ability. Event-related potentials elicited by faces, inverted faces, houses, letters, and pseudoletters were recorded. Groups were compared on an electrophysiological marker of neural specialization (N170), as well as behavioral performance on standardized measures of face recognition and word reading/decoding. Consistent with prior results, individuals with autism displayed slowed face processing and decreased sensitivity to face inversion; however, they showed comparable brain responses to letters, which were associated with behavioral performance in both groups. Results suggest that individuals with autism display atypical neural specialization for social information but intact specialization for nonsocial information. Findings concord with the notion of specific dysfunction in social brain systems rather than nonspecific information-processing difficulties in autism.