Participants memorized target words in silence or while ignoring neutral or valent (positive or negative) distractor words that could be either possessor-relevant or other-relevant. Distractor words impaired recall performance, but valent distractor words caused more disruption than neutral distractors, and negative distractors caused more disruption than positive distractors. The results are problematic for explanations of the irrelevant speech effect within working memory models that do not specify an explicit role of attention in the maintenance of information for immediate serial recall.