In a game context, nonwords either were artificially associated with negative valence or were in some sense neutral or irrelevant. Subsequently, participants memorized target words in silence or while attempting to ignore the negatively valent, irrelevant, or neutral auditory distractor nonwords. The presence of distractor nonwords impaired recall performance, but negative distractor nonwords caused more disruption than neutral and irrelevant distractors, which did not differ in how much disruption they caused. These findings conceptually replicate earlier results showing disruption due to valence with natural language words and extend them by demonstrating that auditory features that may possibly be confounded with valence in natural language words cannot be the cause of the observed disruption. Explanations of the irrelevant speech effect within working memory models that specify an explicit role of attention in the maintenance of information for immediate serial recall can explain this pattern of results, whereas structural models of working memory cannot.