We propose a frequency-domain state representation to improve the performance and reduce the computation and data requirements of reinforcement learning. This approach is tailored to tracking tasks of periodic trajectories. We apply the proposed methodology to an active knee prosthesis application. Using the high-fidelity simulator MuJoCo, we demonstrate significant performance improvements (in terms of Bellman error) for the proposed frequency-domain state representation relative to the current state-of-the-art time-domain state representation used in these applications.