The mesial temporal lobe (MTL) is typically understood as a memory structure in clinical settings, with the sine qua non of MTL damage in epilepsy being memory impairment. Recent models, however, understand memory as one of a number of higher cognitive functions that recruit the MTL through their reliance on more fundamental processes, such as "self-projection" or "association formation". We examined how damage to the left MTL influences these fundamental processes through the encoding of elemental spatial and temporal associations. We used a novel fMRI task to image the encoding of simple visual stimuli, either rich or impoverished, in spatial or spatial plus temporal information. Participants included 14 typical adults (36.4 years, sd. 10.5 years) and 14 patients with left mesial temporal lobe damage as evidenced by a clinical diagnosis of left temporal lobe epilepsy (TLE) and left MTL impairment on imaging (34.3 years, sd. 6.6 years). In-scanner behavioral performance was equivalent across groups. In the typical group whole-brain analysis revealed highly significant bilateral parahippocampal activation (right > left) during spatial associative processing and left hippocampal/parahippocampal deactivation in joint spatial-temporal associative processing. In the left TLE group identical analyses indicated patients used MTL structures contralateral to the seizure focus differently and relied on extra-MTL regions to a greater extent. These results are consistent with the notion that epileptogenic MTL damage is followed by reorganization of networks underlying elemental associative processes. In addition, they provide further evidence that task-related fMRI deactivation can meaningfully index brain function. The implications of these findings for clinical and cognitive neuropsychological models of MTL function in TLE are discussed.