Nearly all methods for analyzing and interpreting functional magnetic resonance imaging (fMRI) data assume that the fMRI signal behaves in an approximately linear manner. However, it has been shown that the mean fMRI response to a pair of briefly presented visual stimuli is significantly smaller than would be expected from the response to a single stimulus. This smaller response could be the result of either a nonlinearity in the fMRI signal or neuronal adaptation. We tested the neuronal adaptation hypothesis by measuring the fMRI response to sequential pairs of sinusoidal gratings that had either the same or orthogonal orientation. The adaptation hypothesis predicts that brain areas with orientation-selective neurons should show a more linear response when the stimulus pair is orthogonal than when the pair is identical. Our results show no orientation-specific adaptation effects in primary visual cortex (V1) but increasing effects along the hierarchy of visual areas (V2, V3, and V4V). A psychophysical contrast detection experiment, using similar oriented gratings as adapters, shows evidence of orientation-specific adaptation in the visual system. These results have implications for the interpretation of rapid event-related fMRI experiments, as well as for recently developed methods that use adaptation as a tool to measure the response properties of underlying neuronal subpopulations.