A large body of evidence suggested that both emotion and self-referential processing can enhance memory. However, it remains unclear how these two factors influence directed forgetting. This study speculates that directed forgetting of negative self-referential memory is more difficult than forgetting of other-referential memory. To verify this speculation, we combined the directed forgetting paradigm with the self-reference task. The behavioral result suggested that although both self-referential and other-referential information can be directly forgotten, less self-referential information can be forgotten than other-referential information. At the neural level, the forget instruction strongly activated the frontal cortex, suggesting that directed forgetting is not memory decay but an active process. In addition, compared with the negative other-referential information, forgetting of the negative self-referential information were associated with a more widespread activation, including the orbital frontal gyrus (BA47), the inferior frontal gyrus (BA45, BA44), and the middle frontal gyrus. Our results suggest that forgetting of the self-referential information seems to be a more demanding and difficult process.