Although many studies have examined the neural bases of controlling cognitive responses, the neural systems for controlling conflicts between competing affective responses remain unclear. To address the neural correlates of affective conflict and their relationship to cognitive conflict, the present study collected whole-brain fMRI data during two versions of the Eriksen flanker task. For these tasks, participants indicated either the valence (affective task) or the semantic category (cognitive task) of a central target word while ignoring flanking words that mapped onto either the same (congruent) or a different (incongruent) response as the target. Overall, contrasts of incongruent > congruent trials showed that bilateral dorsal ACC, posterior medial frontal cortex, and dorsolateral pFC were active during both kinds of conflict, whereas rostral medial pFC and left ventrolateral pFC were differentially active during affective or cognitive conflict, respectively. Individual difference analyses showed that separate regions of rostral cingulate/ventromedial pFC and left ventrolateral pFC were positively correlated with the magnitude of response time interference. Taken together, the findings that controlling affective and cognitive conflicts depends upon both common and distinct systems have important implications for understanding the organization of control systems in general and their potential dysfunction in clinical disorders.