The integration of pipeline energy storage in the control of a district heating system can lead to profit gain, for example by adjusting the electricity production of a combined heat and power (CHP) unit to the fluctuating electricity price. The uncertainty from the environment, the computational complexity of an accurate model, and the scarcity of placed sensors in a district heating system make the operational use of pipeline energy storage challenging. A vast majority of previous works determined a control strategy by a decomposition of a mixed-integer nonlinear model and significant simplifications. To mitigate consequential stability, feasibility, and computational complexity challenges, we model CHP economic dispatch as a Markov decision process. We use a reinforcement learning (RL) algorithm to estimate the system’s dynamics through interactions with the simulation environment. The RL approach is compared with a detailed nonlinear mathematical optimizer on day-ahead and real-time electricity markets and two district heating grid models. The proposed method achieves moderate profit impacted by environment stochasticity. The advantages of the RL approach are reflected in three aspects: stability, feasibility, and time scale flexibility. From this, it can be concluded that RL is a promising alternative for real-time control of complex, nonlinear industrial systems.

District heating system (DHS) optimization is becoming an increasingly important problem because of the unused potential in flexibility that could allow less energy being wasted and the integration of renewable energy. While new optimization methods are proposed every year to tackle this problem, the literature lacks a good way to benchmark newly proposed methods. To address this problem, we introduce GridPenguin, an open-source computational simulator for the physics of district heating networks. It provides flexibility in usage by providing building blocks with which the user can build any grid he wants. The detailed simulation of the physical world with a focus on the heat balance and average flow rate and temperature allows for fast and accurate simulation. By explaining the physical equations and computational model as well as the comparison to existing software, we lay a solid foundation for the performance of the simulator. We present GridPenguin as a metric to evaluate optimization methods as well as a tool for easy integration of advanced machine learning methods into DHS optimization. The source code of our project can be found on https://github.com/ftbv/grid-penguin.