Findings in the domain of spoken word recognition have indicated that lexical representations contain both abstract and episodic information. It has been proposed that processing time determines when each source of information is recruited, with increased processing time being required to access lower-frequency episodic instantiations. The time-course hypothesis of specificity effects has thus identified a strong role for retrieval mechanisms mediating the use of abstract versus episodic information. Here we conducted three recognition memory experiments to examine whether the findings previously attributed to retrieval mechanisms might instead reflect attention during encoding. The results from Experiment 1 showed that talker-specificity effects emerged when subjects attended to the individual speakers, but not when they attended to lexical characteristics, during encoding, even though processing times at retrieval were equivalent. The results from Experiment 2 showed that talker-specificity effects emerged when listeners attended to talker gender but not when they attended to syntactic characteristics, even though the processing times at retrieval were significantly longer in the latter condition. The results from Experiment 3 showed no talker-specificity effects when all listeners attended to lexical characteristics, even when processing at retrieval was slowed by the addition of background noise. Collectively, these results suggest that when processing time during retrieval is decoupled from encoding factors, it fails to predict the emergence of talker-specificity effects. Rather, attention during encoding appears to be the putative variable.