Reinforcement learning (RL) has become a promising approach for optimizing the dispatch of energy storage systems (ESSs) in distributed energy systems. Utilizing linear methods in the Q-representation of RL often struggles to balance accuracy and efficiency, while neural network (NN) performs well but falls short in terms of explainability and interpretability. To address these challenges, we developed and evaluated a deep-Q-symbolic-network (DQSN) framework, which integrates a symbolic network (SN) into the deep-Q-network (DQN) architecture for optimal dispatch of ESS. We benchmarked the performance of DQSN against DQN using mixed-integer linear programming (MILP) results, focusing on algorithm convergence, training duration, and operational cost accuracy. Our findings indicate that DQSN achieves slightly superior rewards and reduced operational costs with a modest increase in training time. Additionally, while DQN demonstrates superior generalization to unseen scenarios, DQSN excels in accurately fitting training data, enabling DQSN to be a viable alternative to DQN, particularly in applications requiring explainability and interpretability.

Deep Reinforcement Learning (DRL) presents a promising avenue for optimizing Energy Storage Systems (ESSs) dispatch in distribution networks. This paper introduces RL-ADN, an innovative open-source library specifically designed for solving the optimal ESSs dispatch in active distribution networks. RL-ADN offers unparalleled flexibility in modeling distribution networks, and ESSs, accommodating a wide range of research goals. A standout feature of RL-ADN is its data augmentation module, based on Gaussian Mixture Model and Copula (GMC) functions, which elevates the performance ceiling of DRL agents, achieving an average performance improvement of 21.43%, 1.08%, 2.76%, by augmenting five-year, one-year and three-month data, respectively. Additionally, RL-ADN incorporates the Tensor Power Flow solver, significantly reducing the computational burden of power flow calculations during training without sacrificing accuracy, maintaining voltage magnitude with an average error not exceeding 0.0001%. The effectiveness of RL-ADN is demonstrated using distribution networks with size varying, showing marked performance improvements in the adaptability of DRL algorithms for ESS dispatch tasks. Furthermore, RL-ADN achieves a tenfold increase in computational efficiency during training, making it highly suitable for large-scale network applications. The library sets a new benchmark in DRL-based ESSs dispatch in distribution networks and it is poised to advance DRL applications in distribution network operations significantly. RL-ADN is available at: https://github.com/ShengrenHou/RL-ADN and https://github.com/distributionnetworksTUDelft/RL-ADN.

Reinforcement Learning (RL) has emerged as a promising solution for defining the optimal dispatch of Energy Storage Systems (ESS) in distributed energy systems. However, a notable gap exists in the literature: a lack of comprehensive and fair comparisons between different RL algorithms, particularly between linear and nonlinear approaches. This study critically evaluates the trade-offs between computational efficiency and operational accuracy among various Linear RL (LRL) strategies and compares them against the nonlinear Deep-Q-Network (DQN) algorithm. Through a comprehensive analysis, this study benchmarks the model-based Mixed-Integer Linear Programming (MILP) results to assess and compare these algorithms' convergence, training efficiency, and optimization accuracy. Results indicate that while LRL approaches the operational cost accuracy of DQN, it faces significant trade-offs in computational efficiency and struggles with generalization across larger and varied datasets. The results illuminate critical areas for further development in LRL methodologies, particularly in enhancing their adaptability and generalization capabilities.