Measuring the cognitive and neural sequelae of switching between tasks permits a window into the flexible functioning of the executive control system. Prolonged reaction times (RTs) after task switches are accompanied by increases in brain activity in the anterior cingulate cortex (ACC) and dorsolateral prefrontal cortex (dlPFC), but the contribution made by these regions to task level control remains controversial. Here, subjects performed a hybrid spatial Stroop/task-switching paradigm, requiring them to respond with a joystick movement to congruent or incongruent spatial/verbal cues. Relative to the previous trial, the active task either switched or remained the same. Calculating switch costs as a function of current and previous trial congruency, we observed both a general RT increase on every switch trial, and additional slowing and impairment to performance when the switch occurred on the second of two successive incongruent trials (iI trials). Imaging data revealed corresponding neural concomitants of these two switch costs: the ACC was activated by task switches regardless of trial type (including congruent trials in which task-relevant and task-irrelevant information did not clash), whereas the caudal dlPFC exhibited a switch cost that was unique to iI trials. We argue that the ACC configures the priorities associated with a new task, whereas the dlPFC tackles interference from recently active, rivalrous task sets. These data contribute to a literature arguing that human cognitive flexibility benefits from the setting of new priorities for future action as well as the overcoming of interference from previously active task sets.