The optimal dispatch of energy storage systems (ESSs) in distribution networks poses significant challenges, primarily due to uncertainties of dynamic pricing, fluctuating demand, and the variability inherent in renewable energy sources. By exploiting the generalization capabilities of deep neural networks (DNNs), the deep reinforcement learning (DRL) algorithms can learn good-quality control models that adapt to the stochastic nature of distribution networks. Nevertheless, the practical deployment of DRL algorithms is often hampered by their limited capacity for satisfying operational constraints in real time, which is a crucial requirement for ensuring the reliability and feasibility of control actions during online operations. This paper introduces an innovative framework, named mixed-integer programming based deep reinforcement learning (MIP-DRL), to overcome these limitations. The proposed MIP-DRL framework can rigorously enforce operational constraints for the optimal dispatch of ESSs during the online execution. This framework involves training a Q-function with DNNs, which is subsequently represented in a mixed-integer programming (MIP) formulation. This unique combination allows for the seamless integration of operational constraints into the decision-making process. The effectiveness of the proposed MIP-DRL framework is validated through numerical simulations, demonstrating its superior capability to enforce all operational constraints and achieve high-quality dispatch decisions and showing its advantage over existing DRL algorithms.

The integration of distributed energy resources (DERs) has escalated the challenge of voltage magnitude regulation in distribution networks. Model-based approaches, which rely on complex sequential mathematical formulations, cannot meet the real-time demand. Deep reinforcement learning (DRL) offers an alternative by utilizing offline training with distribution network simulators and then executing online without computation. However, DRL algorithms fail to enforce voltage magnitude constraints during training and testing, potentially leading to serious operational violations. To tackle these challenges, we introduce a novel safe-guaranteed reinforcement learning algorithm, the DistFlow safe reinforcement learning (DF-SRL), designed specifically for real-time voltage magnitude regulation in distribution networks. The DF-SRL algorithm incorporates a DistFlow linearization to construct an expert-knowledge-based safety layer. Subsequently, the DF-SRL algorithm overlays this safety layer on top of the agent policy, recalibrating unsafe actions to safe domains through a quadratic programming formulation. Simulation results show the DF-SRL algorithm consistently ensures voltage magnitude constraints during training and real-time operation (test) phases, achieving faster convergence and higher performance, which differentiates it apart from (safe) DRL benchmark algorithms.

The penetration of renewable energy in power systems is continually increasing as the global energy system transitions to low-carbon, leading to greater uncertainty intensifying the clearing pressure in electricity markets and the dispatching pressure in power systems. These factors pose challenges to the selection of power-balancing mechanisms. This paper focuses on the typical decentralized decision-making balancing mechanism exemplified by the Balancing Group mechanism in Germany. It analyzes its advantages, such as enhanced autonomy of market participants, reduced aggregation of system uncertainty, and lowered dispatch pressure. The paper first introduces the mechanism's fundamental structure and operational methods, highlighting the role of balancing groups as a key component. Next, it analyzes the characteristics of the German Balancing Group mechanism rules and the supporting mechanisms in actual operations and explores the decentralized nature of this mechanism. Finally, it explores the issues and adaptability faced in introducing related market-oriented balancing mechanisms in China, offering recommendations on dispatch levels, market participation, market integration, and technical support.

Deep Reinforcement Learning (DRL) presents a promising avenue for optimizing Energy Storage Systems (ESSs) dispatch in distribution networks. This paper introduces RL-ADN, an innovative open-source library specifically designed for solving the optimal ESSs dispatch in active distribution networks. RL-ADN offers unparalleled flexibility in modeling distribution networks, and ESSs, accommodating a wide range of research goals. A standout feature of RL-ADN is its data augmentation module, based on Gaussian Mixture Model and Copula (GMC) functions, which elevates the performance ceiling of DRL agents, achieving an average performance improvement of 21.43%, 1.08%, 2.76%, by augmenting five-year, one-year and three-month data, respectively. Additionally, RL-ADN incorporates the Tensor Power Flow solver, significantly reducing the computational burden of power flow calculations during training without sacrificing accuracy, maintaining voltage magnitude with an average error not exceeding 0.0001%. The effectiveness of RL-ADN is demonstrated using distribution networks with size varying, showing marked performance improvements in the adaptability of DRL algorithms for ESS dispatch tasks. Furthermore, RL-ADN achieves a tenfold increase in computational efficiency during training, making it highly suitable for large-scale network applications. The library sets a new benchmark in DRL-based ESSs dispatch in distribution networks and it is poised to advance DRL applications in distribution network operations significantly. RL-ADN is available at: https://github.com/ShengrenHou/RL-ADN and https://github.com/distributionnetworksTUDelft/RL-ADN.

Residential load profiles (RLPs) play an increasingly important role in the optimal operation and planning of distribution systems, particularly with the rising integration of low-carbon energy resources such as PV systems, electric vehicles, small-scale batteries, etc. Despite the prevalence of various data-driven models for generating consumption profiles, there is a lack of clear conclusions about their relative strengths and weaknesses. This study undertakes a comprehensive comparison of frequently used data-driven models in recent research, including Generative Adversarial Networks (GANs), Variational Autoencoders (VAE), Wasserstein GANs (WGAN), WGANs with Gradient Penalty (WGANGP), Gaussian Mixture Models (GMMs), and Gaussian Mixture Copulas (GMC). The presented comparison explores the effectiveness of the above-mentioned models on transformer- and consumer-level consumption profiles, as well as for different time resolutions (15-min, 30-min, and 60-min). The objective of this research is to elucidate the respective advantages and drawbacks of these models, thereby providing valuable insights for subsequent research in this field.

Reinforcement Learning (RL) has emerged as a promising solution for defining the optimal dispatch of Energy Storage Systems (ESS) in distributed energy systems. However, a notable gap exists in the literature: a lack of comprehensive and fair comparisons between different RL algorithms, particularly between linear and nonlinear approaches. This study critically evaluates the trade-offs between computational efficiency and operational accuracy among various Linear RL (LRL) strategies and compares them against the nonlinear Deep-Q-Network (DQN) algorithm. Through a comprehensive analysis, this study benchmarks the model-based Mixed-Integer Linear Programming (MILP) results to assess and compare these algorithms' convergence, training efficiency, and optimization accuracy. Results indicate that while LRL approaches the operational cost accuracy of DQN, it faces significant trade-offs in computational efficiency and struggles with generalization across larger and varied datasets. The results illuminate critical areas for further development in LRL methodologies, particularly in enhancing their adaptability and generalization capabilities.

The massive integration of renewable-based distributed energy resources (DERs) inherently increases the energy system’s complexity, especially when it comes to defining its operational schedule. Deep reinforcement learning (DRL) algorithms arise as a promising solution due to their data-driven and model-free features. However, current DRL algorithms fail to enforce rigorous operational constraints (e.g., power balance, ramping up or down constraints) limiting their implementation in real systems. To overcome this, in this paper, a DRL algorithm (namely MIP-DQN) is proposed, capable of strictly enforcing all operational constraints in the action space, ensuring the feasibility of the defined schedule in real-time operation. This is done by leveraging recent optimization advances for deep neural networks (DNNs) that allow their representation as a MIP formulation, enabling further consideration of any action space constraints. Comprehensive numerical simulations show that the proposed algorithm outperforms existing state-of-the-art DRL algorithms, obtaining a lower error when compared with the optimal global solution (upper boundary) obtained after solving a mathematical programming formulation with perfect forecast information; while strictly enforcing all operational constraints (even in unseen test days).

Taking advantage of their data-driven and model-free features, Deep Reinforcement Learning (DRL) algorithms have the potential to deal with the increasing level of uncertainty due to the introduction of renewable-based generation. To deal simultaneously with the energy systems’ operational cost and technical constraints (e.g, generation-demand power balance) DRL algorithms must consider a trade-off when designing the reward function. This trade-off introduces extra hyperparameters that impact the DRL algorithms’ performance and capability of providing feasible solutions. In this paper, a performance comparison of different DRL algorithms, including DDPG, TD3, SAC, and PPO, are presented. We aim to provide a fair comparison of these DRL algorithms for energy systems optimal scheduling problems. Results show DRL algorithms’ capability of providing in real-time good-quality solutions, even in unseen operational scenarios, when compared with a mathematical programming model of the energy system optimal scheduling problem. Nevertheless, in the case of large peak consumption, these algorithms failed to provide feasible solutions, which can impede their practical implementation.