The issue of the relationship between language and gesture processing and the partial overlap of their neural representations is of fundamental importance to neurology, psychology, and social sciences. Patients suffering from primary progressive aphasia, a clinical syndrome characterized by comparatively isolated language deficits, may provide direct evidence for anatomical and functional association between specific language deficits and gesture discrimination deficits. A consecutive series of 16 patients with primary progressive aphasia and 16 matched control subjects participated. Our nonverbal gesture discrimination task consisted of 19 trials. In each trial, participants observed three video clips showing the same gesture performed correctly in one clip and incorrectly in the other two. Subjects had to indicate which of the three versions was correct. Language and gesture production were evaluated by means of conventional tasks. All participants underwent high-resolution structural and diffusion tensor magnetic resonance imaging. Ten of the primary progressive aphasia patients showed a significant deficit on the nonverbal gesture discrimination task. A factor analysis revealed that this deficit clustered with gesture imitation, word and pseudoword repetition, and writing-to-dictation. Individual scores on this cluster correlated with volume in the left anterior inferior parietal cortex extending into the posterior superior temporal gyrus. Probabilistic tractography indicated this region comprised the cortical relay station of the indirect pathway connecting the inferior frontal gyrus and the superior temporal cortex. Thus, the left perisylvian temporoparietal area may underpin verbal imitative behavior, gesture imitation, and gesture discrimination indicative of a partly shared neural substrate for language and gesture resonance.