Cycling stability of the photochromic effect in rare-earth oxyhydride thin films is of great importance for long-term applications such as smart windows. However, an increasingly slower bleaching rate upon photochromic cycling was found in yttrium oxyhydride thin films; the origin of this memory effect is yet unclear. In this work, the microstructural changes under six photodarkening-bleaching cycles in YHxOy and GdHxOy thin films are investigated by in situ illumination Doppler broadening positron annihilation spectroscopy, complemented by positron annihilation lifetime spectroscopy (PALS) investigations on YHxOy films before and after one cycle. For the first three cycles, the Doppler broadening S parameter after bleaching increases systematically with photodarkening-bleaching cycle, and correlates with the bleaching time constant extracted from optical transmittance measurements. This suggests that the microstructural evolution that leads to progressively slower bleaching involves vacancy creation and agglomeration. PALS suggests that during a photodarkening-bleaching cycle, divacancies are formed that are possibly composed of illumination-induced hydrogen vacancies and preexisting yttrium monovacancies, and vacancy clusters grow, which might be due to local removal of hydrogen. If bleaching is a diffusion-related process, the formed vacancy defects induced by illumination might affect the diffusion time by reducing the diffusion coefficient. Hydrogen loss could also be a key factor in the reduced bleaching kinetics. Other microstructural origins including domain growth, or formation of OH- hydroxide groups, are also discussed with respect to the slower bleaching kinetics. During the fourth to sixth photodarkening-bleaching cycle, reversible shifts in the Doppler S and W parameters are seen that are consistent with the reversible formation of metallic-like domains, previously proposed as a key factor in the mechanism for the photochromic effect.