A challenging issue in relating brain function to perception and cognition concerns the functional interpretation of brain responses. For example, while there is agreement that the N170 component of event-related potentials is sensitive to face processing, there is considerable debate about whether its response reflects a structural encoder for faces, a feature (e.g. eye) detector, or something else. We introduce a principled approach to determine the stimulus features driving brain responses. Our analyses on two observers resolving different face categorization tasks (gender and expressive or not) reveal that the N170 responds to the eyes within a face irrespective of task demands. This suggests a new methodology to attribute function to different components of the neural system for perceiving complex stimuli.