There are substantial individual differences in the size and shape of the corpus callosum and such differences are thought to relate to behavioral lateralization. We report findings from a large scale investigation of relationships between brain anatomy and behavioral asymmetry on a battery of visual word recognition tasks. A sample of 200 individuals was divided into groups on the basis of sex and consistency of handedness. We investigated differences between sex/handedness groups in callosal area and relationships between callosal area and behavioral predictors. Sex/handedness groups did not show systematic differences in callosal area or behavioral asymmetry. However, the groups differed in the relationships between area of the corpus callosum and behavioral asymmetry. Among consistent-handed males, callosal area was negatively related to behavioral laterality. Among mixed-handed males and consistent-handed females, behavioral laterality was not predictive of callosal area. The most robust relationship was observed in mixed-handed females, in whom behavioral asymmetry was positively related to callosal area. Our study demonstrates the importance of considering brain/behavior relationships within sub-populations, as relationships between behavioral asymmetry and callosal anatomy varied across subject groups.