Working memory of familiar faces involves the coordination of multiple brain regions in sensory processing, attention and memory, and relies on robust representations in long-term memory. It is not clear how prior knowledge interacts with bottom-up visual processing at different phases of working memory. In this study, we collected functional magnetic resonance imaging (fMRI) data of 40 right-handed participants during the sequential memory task and recognition task of familiar celebrity faces. We observed strong left-lateralized neural activity in the language-processing areas and right-lateralized activity for visual processing in the dorsal stream. Yet, no obvious hemispheric lateralization was found in either face-selective (fusiform gyrus) or memory-specific (hippocampus) areas. Besides, the left lateralization of prefrontal activity and its task-evoked regulation on visual areas boost face memory performance, i.e., faster reaction and higher accuracy. These findings suggest that the top-down prefrontal regulation plays a critical role in the successful memory of familiar faces. Our study provides neural substrates underlying how familiarity boosts face memory by endorsing prior/common knowledge through left-lateralized language network.