In the peripheral visual field, nearby objects can make one another difficult to recognize (crowding) in a manner that critically depends on their separation. We manipulated the apparent separation of objects using the illusory shifts in perceived location that arise from local motion to determine if crowding depends on physical or perceived location. Flickering Gabor targets displayed between either flickering or drifting flankers were used to (a) quantify the perceived target-flanker separation and (b) measure discrimination of the target orientation or spatial frequency as a function of physical target-flanker separation. Relative to performance with flickering targets, we find that flankers drifting away from the target improve discrimination, while those drifting toward the target degrade it. When plotted as a function of perceived separation across conditions, the data collapse onto a single function indicating that it is perceived and not physical location that determines the magnitude of visual crowding. There was no measurable spatial distortion of the target that could explain the effects. This suggests that crowding operates predominantly in extrastriate visual cortex and not in early visual areas where the response of neurons is retinotopically aligned with the physical position of a stimulus.