The human ability to flexibly alternate between tasks represents a central component of cognitive control. Neuroimaging studies have linked task switching with a diverse set of prefrontal cortex (PFC) regions, but the contributions of these regions to various forms of cognitive flexibility remain largely unknown. Here, subjects underwent functional brain imaging while they completed a paradigm that selectively induced stimulus, response, or cognitive set switches in the context of a single task decision performed on a common set of stimuli. Behavioral results indicated comparable reaction time costs associated with each switch type. Domain-general task-switching activation was observed in the inferior frontal junction and posterior parietal cortex, suggesting core roles for these regions in switching such as updating and representing task sets. In contrast, multiple domain-preferential PFC activations were observed across lateral and medial PFC, with progressively more rostral regions recruited as switches became increasingly abstract. Specifically, highly abstract cognitive set switches recruited anterior-PFC regions, moderately abstract response switches recruited mid-PFC regions, and highly constrained stimulus switches recruited posterior-PFC regions. These results demonstrate a functional organization across lateral and medial PFC according to the level of abstraction associated with acts of cognitive flexibility.