This project aimed to investigate reinforcement learning (RL) algorithms to improve water management policy development in the Nile Basin, with a focus on the Multi-Objective Natural Evolution Strategies (MONES) and Evolutionary Multi-Objective Direct Policy Search (EMODPS) algorithms. This project intended to refactor a Nile Basin simulation to be compatible with the MONES algorithm, which continues the exploration of different machine learning algorithms in water resource management. Additionally, the RL algorithms were aimed at training using two climate data sets: human-favourable and climate-varying conditions, and then evaluating on the satisfaction and regret metrics. The successful integration of the MONES framework shows the feasibility of utilizing advanced RL algorithms for water management problems. Initial results indicate that the MONES algorithm underperforms compared to the EMODPS algorithm according to hypervolume and diversity of solutions, however, further research is needed to test whether this claim holds. The EMODPS algorithm faced challenges in finding optimal solutions when dealing with variable climate conditions scenarios, accentuating the need for robust solutions, which consider a variety of possible climate outcomes. The observed sensitivity to variable climate conditions underlines the crucial importance of accurate and recent data, as well as the need to consider the climate change effects on water management. The study concluded with suggestions that future simulations of water management strategies may be improved if a broader set of external factors and a more realistic representation of objectives are included in the simulation model. These improvements stand to positively impact the accuracy, applicability and reliability of future simulations.

This study investigates the use of Multi-Objective Natural Evolution Strategies (MONES) to optimise water management control policies in the Nile River Basin, focusing on four key objectives: minimising irrigation deficits for Egypt and Sudan, maximising hydropower production for Ethiopia, and maintaining water level in the High Aswan Dam (HAD). The developed Nile River simulation was integrated with MONES and used to train an agent for making release decisions. Performance metrics including hypervolume, epsilon-indicator, and Inverted Generational Distance Plus (IGD+) were employed to compare MONES with the EMODPS baseline. The results indicate that while MONES identifies feasible solutions, it falls short in exploration and overall performance compared to EMODPS. The study contributes to the development of water management strategies by open-sourcing Nile River framework compatible with reinforcement learning and demonstrating the potential and limitations of MONES in multi-objective optimisations.

Efficient management of water resources is increasingly critical in the face of growing challenges such as climate change and population growth. This research paper introduces RL4Water, an adaptable framework for simulating water management systems using multi-objective reinforcement learning (MORL). Adhering to the Gymnasium API standard, RL4Water ensures seamless integration with existing MORL algorithms. The framework includes diverse facility classes to accurately model the physical components of water networks. Its generalizability is enhanced by allowing users to modify both the physical properties of these components and the key features of the MORL simulations. RL4Water's capabilities are demonstrated through two case studies: simulations of the Nile River and the Susquehanna River, validating its accuracy and flexibility in managing both large, distributed water systems and centralized systems with complex reservoirs. By bridging the gap between water management and reinforcement learning, RL4Water offers a unified platform for developing and researching water management simulations.

Water management systems (WMSs) are complex systems in which often multiple conflicting objectives are at stake. Reinforcement Learning (RL), where an agent learns through punishments and rewards, can find trade-offs between these objectives. This research studies three case studies of WMS simulations in the context of RL problems and notes their similarities and differences. Based on these, core properties of WMSs are defined and used to formulate a general WMS as a RL problem. This bottom-up approach uses Gymnasium to implement the RL problem. The result is compared to a simulation from one of the case studies and produces the same results. While maintaining this level of accuracy, it is applicable to a much wider range of WMSs. It thereby contributes to generalisation of WMSs in the context of RL, and removes the need to rewrite simulations each time.

This paper studies the simulation of the Nile River as a multi-objective reinforcement learning problem. The main goal of this essay is to develop and evaluate the visualization techniques to effectively present the results of reinforcement learning models. Using a multi-objective approach, visualizations are very important for understanding the trade-offs and complexities in managing the Nile River problem.

This study includes a user evaluation to compare different visualizations, analyzing their effectiveness in terms of their clarity and usefulness using ANOVA test. Additionally, the effectiveness of clustering and full data points will be analyzed using a chi-square test to choose which visualisation technique works the best.

According to the results, stacked bar chart and parallel coordinates plot performed the best, while the spider plot performed the worst. Additionally, there is no preference between clustered and full data points visualizations based on the user evaluation.