Deaf children who are native users of American Sign Language (ASL) and hearing children who are native English speakers performed three working memory tasks. Results indicate that language modality shapes the architecture of working memory. Digit span with forward and backward report, performed by each group in their native language, suggests that the language rehearsal mechanisms for spoken language and for sign language differ in their processing constraints. Unlike hearing children, deaf children who are native signers of ASL were as good at backward recall of digits as at forward recall, suggesting that serial order information for ASL is stored in a form that does not have a preferred directionality. Data from a group of deaf children who were not native signers of ASL rule out explanations in terms of a floor effect or a nonlinguistic visual strategy. Further, deaf children who were native signers outperformed hearing children on a nonlinguistic spatial memory task, suggesting that language expertise in a particular modality exerts an influence on nonlinguistic working memory within that modality. Thus, language modality has consequences for the structure of working memory, both within and outside the linguistic domain.