Risk-Aware Day-Ahead Energy Market Bidding Using Reinforcement Learning and Imbalance Market Flexibility

Abstract

The increasing penetration of renewable energy sources (RES), worsening grid congestion and rising electricity price volatility are changing the operating conditions under which prosumers participate in electricity markets. In the Netherlands, these developments are accompanied by increasingly restrictive and time-varying grid connection limits, requiring energy management systems (EMSs) to reduce operational costs while maintaining feasible operation under uncertainty. This thesis develops and evaluates a risk-aware residual reinforcement learning (RRL) framework for day-ahead market (DAM) bidding by a grid-connected prosumer equipped with a commercial building load, a solar photovoltaic (PV) installation and a battery energy storage system (BESS). The proposed method combines a scenario-based optimisation baseline with a learning-based residual policy. This allows the optimisation layer to provide a constraint-aware reference schedule while the reinforcement learning (RL) agent learns corrective bidding and battery-scheduling actions. Time-varying grid limits are incorporated throughout the operational framework, constraining both the day-ahead schedule and the real-time imbalance market operation, where a rule-based controller operates the BESS and curtails PV generation during delivery. The proposed strategies are evaluated under Dutch electricity market conditions and compared against deterministic linear programming (LP) and scenario-based optimisation benchmarks in terms of operational cost, robustness and grid-limit violations.