Cytotoxic T-lymphocyte (CTL) memory to viruses has traditionally been studied in an isolated setting. However, recent experiments have indicated that the presence of antigenically heterologous challenges can result in the attrition of CTL memory. Here we use mathematical models in order to explore the consequence of these dynamics for the ability of the immune system in controlling multiple infections. Mathematical models suggest that antigen-independent persistence of CTL memory is required in order to resolve and clear an infection. This ensures strong immunological pressure at low loads when the virus population declines towards extinction. If the number of antigenic stimuli exposed to the immune system crosses a threshold, we find that immunological pressure is significantly reduced at low loads and this can prevent virus clearance and reduces overall control of viral replication. Hence, exposure to many heterologous challenges reduces the ability of CTL memory to contribute to virus control. The higher the number of infections present in the host, the higher the overall virus load and the higher the total number of memory CTLs. Beyond a given threshold, addition of new viruses to the system results in accelerated loss of virus control which eventually leads to a reduction in the overall memory CTL population. These dynamics might contribute to the progressively weaker immunity observed as a result of ageing. In this context, antigenically variable pathogens expose the immune system to many heterologous challenges within a short period of time and this could result in accelerated ageing of the immune system. These results have important implications for vaccination and treatment strategies directed against viral infections.