We used statistical modeling to investigate variability in the cortical auditory representations of 24 normal-hearing epilepsy patients undergoing electrocortical stimulation mapping (ESM). Patients were identified as normal or impaired listeners based on recognition accuracy for acoustically filtered words used to simulate everyday listening conditions. The experimental ESM task was a binary (same-different) auditory syllable discrimination paradigm that both listener groups performed accurately at baseline. Template mixture modeling of speech discrimination deficits during ESM showed larger and more variable cortical distributions for impaired listeners than normal listeners, despite comparable behavioral performances. These results demonstrate that individual differences in speech recognition abilities are reflected in the underlying cortical representations.