Whether and how the balance between plasticity and stability varies across the brain is an important open question. Within a processing hierarchy, it is thought that plasticity is increased at higher levels of cortical processing, but direct quantitative comparisons between low- and high-level plasticity have not been made so far. Here, we address this issue for the human cortical visual system. We quantify plasticity as the complement of the heritability of resting-state functional connectivity and thereby demonstrate a non-monotonic relationship between plasticity and hierarchical level, such that plasticity decreases from early to mid-level cortex, and then increases further of the visual hierarchy. This non-monotonic relationship argues against recent theory that the balance between plasticity and stability is governed by the costs of the "coding-catastrophe", and can be explained by a concurrent decline of short-term adaptation and rise of long-term plasticity up the visual processing hierarchy.