Cognitive neuroscience exploring the architecture of semantics has shown that coherent supramodal concepts are computed in the anterior temporal lobes (ATL), but it is unknown how/where modular information implemented by posterior cortices (word/object/face forms) is conveyed to the ATL hub. We investigated the semantic module-hub network in healthy adults (n = 19) and in semantic dementia patients (n = 28) by combining semantic assessments of verbal and nonverbal stimuli and MRI-based fiber tracking using seeds in three module-related cortices implementing (i) written word forms (visual word form area), (ii) abstract lexical representations (posterior-superior temporal cortices), and (iii) face/object representations (face form area). Fiber tracking revealed three key tracts linking the ATL with the three module-related cortices. Correlation analyses between tract parameters and semantic scores indicated that the three tracts subserve semantics, transferring modular verbal or nonverbal object/face information to the left and right ATL, respectively. The module-hub tracts were functionally and microstructurally damaged in semantic dementia, whereas damage to non-module-specific ATL tracts (inferior longitudinal fasciculus, uncinate fasciculus) had more limited impact on semantic failure. These findings identify major components of the white matter module-hub network of semantics, and they corroborate/materialize claims of cognitive models positing direct links between modular and semantic representations. In combination with modular accounts of cognition, they also suggest that the currently prevailing "hub-and-spokes" model of semantics could be extended by incorporating an intermediate module level containing invariant representations, in addition to "spokes," which subserve the processing of a near-unlimited number of sensorimotor and speech-sound features.