The operation of a community energy storage system (CESS) is challenging due to the volatility of photovoltaic distributed generation, electricity consumption, and energy prices. Selecting the optimal CESS setpoints during the day is a sequential decision problem under uncertainty, which can be solved using dynamic learning methods. This paper proposes a reinforcement learning (RL) technique based on temporal difference learning with eligibility traces (ET). It aims to minimize the day-ahead energy costs while maintaining the technical limits at the grid coupling point. The performance of the RL is compared against an oracle based on a deterministic mixed-integer second-order constraint program (MISOCP). The use of ET boosts the RL agent learning rate for the CESS operation problem. The ET effectively assigns credit to the action sequences that bring the CESS to a high state of charge before the peak prices, reducing the training time. The case study shows that the proposed method learns to operate the CESS effectively and ten times faster than common RL algorithms applied to energy systems such as Tabular Q-learning and Fitted-Q. Also, the RL agent operates the CESS 94% near the optimal, reducing the energy costs for the end-user up to 12%.

The power grid is rapidly experiencing a transformation driven by renewable and climate targets which pose a huge challenge to maintain the system stability and reliability, thus balancing services become more crucial now than ever. To provide balancing services in a cost-efficient way, it is necessary to develop predictive models which can optimize power portfolio in an online manner. This paper presents the development of an Online Predictive Dispatch Optimizer and its connection with a grid model that simulates power and frequency control within interconnected power systems. The performance of the dispatch optimizer and its connection with the grid model is tested by simulating several cases where the adequacy of the model is confirmed, especially regarding its ability to manage energy imbalance in real-time. The outcomes and flexibility of this framework can be used to quantitatively evaluate the operation of power portfolio owners in the power grid.