Scalable Reinforcement Learning for Optimal Electric Vehicle Charging

Abstract

As the adoption of electric vehicles (EVs) accelerates, Charging Point Operators must continuously coordinate charging decisions for large fleets under uncertainty (arrivals, departures, heterogeneous energy needs and prices), while preserving user satisfaction and respecting distribution grid infrastructure limits. Existing optimization-based approaches can solve simpler problems, but their computational requirements grow rapidly with problem size and model realism.
In contrast, learning-based algorithms can offer faster decision-making, shifting most computation to training and enabling fast inference during operation. However, existing data-driven methods must overcome practical limitations, such as scalability to large networks, alignment with grid constraints, and robustness to changing operating conditions.
This PhD thesis addresses these challenges through a progression of contributions that aim to develop scalable, real-time, and grid-aware methods for EV smart charging.

First, EV2Gym, an open-source simulation and benchmarking platform for Reinforcement Learning (RL) based EV charging, is presented.
EV2Gym uses realistic system and EV behavioral models, constituting a unified evaluation pipeline with a diverse set of baselines. This establishes a reproducible experimental foundation and clarifies which aspects of the problem (uncertainty, partial observability, constraint coupling, and scale) most strongly influence algorithmic performance.

Building on the realistic simulation platform, this PhD thesis then proposes scalable RL formulations that exploit the structure of charging and distribution networks.
To leverage this structure, a graph-based RL approach is introduced that uses Graph Neural Networks to produce permutation-equivariant representations and a branch-pruning mechanism that focuses computation on relevant substructures (i.e., branches leading to active EV sessions).
This substantially improves RL's scalability and learning efficiency compared to fixed-size vector formulations, and enables real-time control in large instances where classical receding-horizon optimization becomes prohibitively slow.

A central theme of this PhD thesis is that grid awareness must be learned rather than only assumed.
Purely model-free RL can learn high rewards in simulation while still producing actions that stress the network, particularly when voltage constraints are spatially coupled, and the environment is uncertain.
To close this gap, the thesis introduces a physics-informed RL method that incorporates distribution network dynamics via differentiable power flow components and voltage-aware learning signals.
This integration improves sample efficiency and stability during RL training, resulting in policies that maintain high user satisfaction while reducing voltage magnitude violations, thereby bridging fast control with the physical feasibility requirements of distribution networks.

Finally, this PhD thesis investigates how to reduce reliance on online RL exploration and improve generalization in dynamic charging environments with sparse or delayed rewards.
To achieve this, a topology-aware offline sequence-modeling approach is proposed that learns charging decisions directly from expert demonstrations. The proposed graph formulation is combined with a powerful Large Language Model, enabling fast inference at deployment time without repeated interactions with the environment.
By capturing long-horizon dependencies and conditioning on desired outcomes, the approach can reach near-oracle performance across
diverse data regimes and generalize to variations in system dynamics and problem size.

Overall, this PhD thesis demonstrates that deployment-grade EV smart charging benefits from learning algorithms whose inductive biases align with the problem's operational and physical structure. Together, these contributions advance the state of the art toward city-wide, real-time, and grid-aware EV charging.