High-frequency phonons hold great promise as carriers of quantum information on chip and as quantum memories. Because of their coherent interaction with several systems, their compact mode volume, and slow group velocity, multiple experiments have recently demonstrated coherent transport of information on chip using phonon modes, interconnecting distinct quantum devices. Strongly confined phonons in waveguidelike geometries are particularly interesting because of their long lifetime. However, spectral diffusion has been observed to substantially limit their coherence times [S. M. Meenehan et al., Silicon optomechanical crystal resonator at millikelvin temperatures, Phys. Rev. A 90, 011803(R) (2014), A. Wallucks et al., A quantum memory at telecom wavelengths, Nat. Phys. 16, 772 (2020), and G. S. MacCabe et al., Nano-acoustic resonator with ultralong phonon lifetime, Science 370, 840 (2020)]. Coupling to two-level systems is suspected to be a major contributor to the diffusion; however, to date, the origin and underlying mechanisms are still not fully understood. Here, we perform a time-domain study on two adjacent mechanical modes (separated by around 5 MHz) and show that the frequency positions of the two modes are not correlated in time, in agreement with our theoretical model and Monte Carlo simulations. This result is an important step in fully understanding the microscopic mechanisms of dephasing in mechanical quantum buses and memories.