Visual search is slowed for cast shadows lit from above, as compared to the same search items inverted and so not interpreted as shadows (R. A. Rensink & P. Cavanagh, 2004). The underlying mechanisms for such impaired shadow processing are still not understood. Here we investigated the processing levels at which this shadow-related slowing might operate, by examining its interaction with a range of different phenomena including eye movements, perceptual learning, and stimulus presentation context. The data demonstrated that the shadow mechanism affects the number of saccades during the search rather than the duration until first saccade onset and can be overridden by prolonged training, which then transfers from one type of shadow stimulus to another. Shadow-related slowing did not differ for peripheral and central search items but was reduced when participants searched unilateral displays as compared to bilateral ones. Together our findings suggest that difficulties with perceiving shadows are due to visual processes linked to object recognition, rather than to shadow-specific identification and suppression mechanisms in low-level sensory visual areas. Findings are discussed in the context of the need for the visual system to distinguish between illumination and material.