It remains unclear to what extent retinotopic maps can undergo large-scale plasticity following damage to human visual cortex. The literature has predominately focused on retinotopic changes in patients with retinal pathologies or congenital brain malformations. Yet, damage to the adult visual cortex itself is common in cases such as stroke, tumor, or trauma. To address this issue, we used a unique database of fMRI vision maps in patients with adult-onset (n=25) and congenital (n=2) pathology of the visual cortex. We identified atypical retinotopic organization in three patients (two with adult-onset, and one with congenital pathology) consisting of an expanded ipsilateral field representation that was on average 3.2 times greater than healthy controls. The expanded representations were located at the vertical meridian borders between visual areas such as V1/V2. Additionally, two of the three patients had apparently an ectopic (topographically inconsistent) representation of the ipsilateral field within lateral occipital cortex that is normally associated with visual areas V3/V3A (and possibly other areas). Both adult-onset cases had direct damage to early visual cortex itself (rather than to the afferent drive only), resulting in a mostly nonfunctional hemisphere. The congenital case had severe cortical malformation of the visual cortex and was acallosal. Our results are consistent with a competitive model in which unilateral damage to visual cortex or disruption of the transcallosal connections removes interhemispheric suppression from retino-geniculate afferents in intact visual cortex that represent the vertical meridian and ipsilateral visual field.