The relationship between human cytomegalovirus (HCMV) infections and accelerated immune senescence is controversial. Whereas some studies reported a CMV-associated impaired capacity to control heterologous infections at old age, other studies could not confirm this. We hypothesized that these discrepancies might relate to the variability in the infectious dose of CMV occurring in real life. Here, we investigated the influence of persistent CMV infection on immune perturbations and specifically addressed the role of the infectious dose on the contribution of CMV to accelerated immune senescence. We show in experimental mouse models that the degree of mouse CMV (MCMV)-specific memory CD8+ T cell accumulation and the phenotypic T cell profile are directly influenced by the infectious dose, and data on HCMV-specific T cells indicate a similar connection. Detailed cluster analysis of the memory CD8+ T cell development showed that high-dose infection causes a differentiation pathway that progresses faster throughout the life span of the host, suggesting a virus–host balance that is influenced by aging and infectious dose. Importantly, short-term MCMV infection in adult mice is not disadvantageous for heterologous superinfection with lymphocytic choriomeningitis virus (LCMV). However, following long-term CMV infection the strength of the CD8+ T cell immunity to LCMV superinfection was affected by the initial CMV infectious dose, wherein a high infectious dose was found to be a prerequisite for impaired heterologous immunity. Altogether our results underscore the importance of stratification based on the size and differentiation of the CMV-specific memory T cell pools for the impact on immune senescence, and indicate that reduction of the latent/lytic viral load can be beneficial to diminish CMV-associated immune senescence.