This work seeks to resolve an outstanding problem in the use of reinforcement-learning methods for the simulation of economically-rational agents. We discuss the problem of non-stationarity, and how this subsequently limits market simulation capabilities. After explicating and isolating the source of the problem for a day-ahead electricity market, we demonstrate the application of methods which resolve this problem in simple test-cases, and prove conditions under which similar methods will work in general. Subsequently, we illustrate how these techniques can be used to solve a restricted market-design problem, in the process introducing a framework for discussing adversarial market-design for electricity markets in general. It is hoped that, insofar as they provide a new feedback-loop for market-design, these results will facilitate the design of more complex electricity-markets suitable for the energy transition.