The purpose of this study is to explore the possibility of progressive neuropsychological decline in chronic temporal lobe epilepsy (TLE) and determine how this vulnerability may be associated with the neurodevelopmental impact of the disorder. 53 patients with TLE and 62 healthy controls underwent quantitative MRI volumetric imaging of total brain tissue and hippocampal volumes as well as assessment of intelligence and memory function. In addition to reduced hippocampal volume, childhood onset (< 14 years) but not adult onset TLE was associated with significantly reduced total brain tissue that was generalized in nature and extended into extratemporal regions. In addition to this adverse impact on brain structure, there was significantly reduced intellectual status as well as memory function in childhood onset TLE patients, consistent with the generalized nature of the MRI volumetric abnormalities. Finally, cross-sectional correlational analyses indicated that increasing duration of epilepsy in childhood onset patients was associated with declining performance across both intellectual and memory measures, suggestive of progressive cognitive effects. We propose that childhood onset TLE is associated with an adverse neurodevelopmental impact on brain structure and function which represents an early acquired vulnerability, effectively reducing cerebral reserve, placing patients at risk for progressive cognitive decline in the context of chronic and unremitting epilepsy.