In-depth understanding of the synergetic effect between the various incorporating constituents in asphalt binders (e.g., polymers, fillers) is needed to design durable paving materials with desired properties. In this research, the focus was first on the effect of the reactivity of fillers on the evolution of adhesive strength between stone aggregates and epoxy modified asphalt mastics during the epoxy polymerization. Uniaxial tensile tests were performed on different combinations of fillers and binders with and without the epoxy-based polymer, and at different modification levels. Based on the results of the tensile tests, the increase of the adhesive strength of mastic with aggregates was generally lower when reactive filler particles (i.e., hydrated lime) were added than of epoxy binders with non-reactive filler. In other words, the non-reactive fillers did not influence the adhesion process and were thus selected for the next step studies on aging. The chemo-mechanical changes of epoxy modified asphalt mastics were analysed after pressure aging vessel and oven-conditioning after various aging times by means of Fourier transform infrared spectroscopy and dynamic shear rheometer. Less sulfoxides formed and higher modulus levels were measured with increasing the epoxy polymer in mastics over oven- and PAV-aging conditions. Due to the pressure difference, the rate of modulus increases and phase angle decrease was higher when the materials were conditioned in PAV than in oven.