Increasing brain volume may impose constraints, through longer information transfer delays, on the distributed networks supporting language. Here, we assessed the relative effects of brain volume and other putative predictors of the functional variability of perisylvian language areas, as probed with PET, during both a language comprehension and a language production task. In the case of language comprehension (story listening), a linear combination of planum temporale surface, brain volume and handedness could explain almost 60% of the functional asymmetry observed in the perisylvian area. Without brain volume, the goodness of fit was significantly decreased (39%, P < 0.05), and furthermore, the effect of handedness was not detected anymore. This was due to the fact that in our sample, left-handers (n = 12) had a significantly larger brain volume as compared to right-handers (n = 8, P = 0.03). As for language production (verb generation), brain volume and the planum temporale also played a role. However, in this case, the main predictor of functional variability was handedness, where a greater degree of right-handedness was associated with larger activation of left inferior frontal regions. Depending on the language component of interest, these results support different (yet compatible) theories on hemispheric specialization. Left specialization for comprehension could be attributed to the constraints of processing speech stimuli, while a gestural origin of language is mostly supported by the relation we observed between left specialization for production and right-handedness.