Z. Osika
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
6 records found
1
Learning a Policy from User Preferences
An Interactive Approach to Multi-Objective Reinforcement Learning
Many real-life problems are complex due to their multi-objective nature. Over the past decade, there has been growing research on Multi-Objective Reinforcement Learning (MORL) problems, which simulate the complexities of real-life scenarios. Because there are multiple objectives to be optimized, the majority of the MORL methods focus on providing a dense set of solutions called the Pareto Front as a result. The issues with the current approaches are that generating a large solution set requires high computational costs, and it can still be difficult for the user to find their most preferred solutions from a large solution set. In this research, we propose an interactive MORL method where the user is asked for their preferred solution in every iteration from the current solution set, and the algorithm utilizes this information to enhance its learning process to find preference-aligned solutions. This is achieved by bounding the solution space to only search for new policies that outperform the previously user-selected solution within these bounds. We evaluate our method using an artificial user function to simulate preferences, comparing it with non-interactive MORL methods. Metrics to compare the quality of solutions include the number of learning steps required to converge to a preferred solution, the value achieved on the artificial user function. The results demonstrate that the interactive method provides a dense set of solutions in the user’s region of interest, and it tends to converge faster towards the user’s preferred solution.
...
Many real-life problems are complex due to their multi-objective nature. Over the past decade, there has been growing research on Multi-Objective Reinforcement Learning (MORL) problems, which simulate the complexities of real-life scenarios. Because there are multiple objectives to be optimized, the majority of the MORL methods focus on providing a dense set of solutions called the Pareto Front as a result. The issues with the current approaches are that generating a large solution set requires high computational costs, and it can still be difficult for the user to find their most preferred solutions from a large solution set. In this research, we propose an interactive MORL method where the user is asked for their preferred solution in every iteration from the current solution set, and the algorithm utilizes this information to enhance its learning process to find preference-aligned solutions. This is achieved by bounding the solution space to only search for new policies that outperform the previously user-selected solution within these bounds. We evaluate our method using an artificial user function to simulate preferences, comparing it with non-interactive MORL methods. Metrics to compare the quality of solutions include the number of learning steps required to converge to a preferred solution, the value achieved on the artificial user function. The results demonstrate that the interactive method provides a dense set of solutions in the user’s region of interest, and it tends to converge faster towards the user’s preferred solution.
This study investigates the use of Multi-Objective Natural Evolution Strategies (MONES) to optimise water management control policies in the Nile River Basin, focusing on four key objectives: minimising irrigation deficits for Egypt and Sudan, maximising hydropower production for Ethiopia, and maintaining water level in the High Aswan Dam (HAD). The developed Nile River simulation was integrated with MONES and used to train an agent for making release decisions. Performance metrics including hypervolume, epsilon-indicator, and Inverted Generational Distance Plus (IGD+) were employed to compare MONES with the EMODPS baseline. The results indicate that while MONES identifies feasible solutions, it falls short in exploration and overall performance compared to EMODPS. The study contributes to the development of water management strategies by open-sourcing Nile River framework compatible with reinforcement learning and demonstrating the potential and limitations of MONES in multi-objective optimisations.
...
This study investigates the use of Multi-Objective Natural Evolution Strategies (MONES) to optimise water management control policies in the Nile River Basin, focusing on four key objectives: minimising irrigation deficits for Egypt and Sudan, maximising hydropower production for Ethiopia, and maintaining water level in the High Aswan Dam (HAD). The developed Nile River simulation was integrated with MONES and used to train an agent for making release decisions. Performance metrics including hypervolume, epsilon-indicator, and Inverted Generational Distance Plus (IGD+) were employed to compare MONES with the EMODPS baseline. The results indicate that while MONES identifies feasible solutions, it falls short in exploration and overall performance compared to EMODPS. The study contributes to the development of water management strategies by open-sourcing Nile River framework compatible with reinforcement learning and demonstrating the potential and limitations of MONES in multi-objective optimisations.
Climate-Resilient Water Management via Reinforcement Learning
Impact of varying climate conditions on water management of the Nile River Basin using Reinforcement Learning
This project aimed to investigate reinforcement learning (RL) algorithms to improve water management policy development in the Nile Basin, with a focus on the Multi-Objective Natural Evolution Strategies (MONES) and Evolutionary Multi-Objective Direct Policy Search (EMODPS) algorithms. This project intended to refactor a Nile Basin simulation to be compatible with the MONES algorithm, which continues the exploration of different machine learning algorithms in water resource management. Additionally, the RL algorithms were aimed at training using two climate data sets: human-favourable and climate-varying conditions, and then evaluating on the satisfaction and regret metrics. The successful integration of the MONES framework shows the feasibility of utilizing advanced RL algorithms for water management problems. Initial results indicate that the MONES algorithm underperforms compared to the EMODPS algorithm according to hypervolume and diversity of solutions, however, further research is needed to test whether this claim holds. The EMODPS algorithm faced challenges in finding optimal solutions when dealing with variable climate conditions scenarios, accentuating the need for robust solutions, which consider a variety of possible climate outcomes. The observed sensitivity to variable climate conditions underlines the crucial importance of accurate and recent data, as well as the need to consider the climate change effects on water management. The study concluded with suggestions that future simulations of water management strategies may be improved if a broader set of external factors and a more realistic representation of objectives are included in the simulation model. These improvements stand to positively impact the accuracy, applicability and reliability of future simulations.
...
This project aimed to investigate reinforcement learning (RL) algorithms to improve water management policy development in the Nile Basin, with a focus on the Multi-Objective Natural Evolution Strategies (MONES) and Evolutionary Multi-Objective Direct Policy Search (EMODPS) algorithms. This project intended to refactor a Nile Basin simulation to be compatible with the MONES algorithm, which continues the exploration of different machine learning algorithms in water resource management. Additionally, the RL algorithms were aimed at training using two climate data sets: human-favourable and climate-varying conditions, and then evaluating on the satisfaction and regret metrics. The successful integration of the MONES framework shows the feasibility of utilizing advanced RL algorithms for water management problems. Initial results indicate that the MONES algorithm underperforms compared to the EMODPS algorithm according to hypervolume and diversity of solutions, however, further research is needed to test whether this claim holds. The EMODPS algorithm faced challenges in finding optimal solutions when dealing with variable climate conditions scenarios, accentuating the need for robust solutions, which consider a variety of possible climate outcomes. The observed sensitivity to variable climate conditions underlines the crucial importance of accurate and recent data, as well as the need to consider the climate change effects on water management. The study concluded with suggestions that future simulations of water management strategies may be improved if a broader set of external factors and a more realistic representation of objectives are included in the simulation model. These improvements stand to positively impact the accuracy, applicability and reliability of future simulations.
Bottom-up Formulation of Water Management Systems as a Reinforcement Learning Problem
Generalisation of Water Management in the Context of Reinforcement Learning
Water management systems (WMSs) are complex systems in which often multiple conflicting objectives are at stake. Reinforcement Learning (RL), where an agent learns through punishments and rewards, can find trade-offs between these objectives. This research studies three case studies of WMS simulations in the context of RL problems and notes their similarities and differences. Based on these, core properties of WMSs are defined and used to formulate a general WMS as a RL problem. This bottom-up approach uses Gymnasium to implement the RL problem. The result is compared to a simulation from one of the case studies and produces the same results. While maintaining this level of accuracy, it is applicable to a much wider range of WMSs. It thereby contributes to generalisation of WMSs in the context of RL, and removes the need to rewrite simulations each time.
...
Water management systems (WMSs) are complex systems in which often multiple conflicting objectives are at stake. Reinforcement Learning (RL), where an agent learns through punishments and rewards, can find trade-offs between these objectives. This research studies three case studies of WMS simulations in the context of RL problems and notes their similarities and differences. Based on these, core properties of WMSs are defined and used to formulate a general WMS as a RL problem. This bottom-up approach uses Gymnasium to implement the RL problem. The result is compared to a simulation from one of the case studies and produces the same results. While maintaining this level of accuracy, it is applicable to a much wider range of WMSs. It thereby contributes to generalisation of WMSs in the context of RL, and removes the need to rewrite simulations each time.
Efficient management of water resources is increasingly critical in the face of growing challenges such as climate change and population growth. This research paper introduces RL4Water, an adaptable framework for simulating water management systems using multi-objective reinforcement learning (MORL). Adhering to the Gymnasium API standard, RL4Water ensures seamless integration with existing MORL algorithms. The framework includes diverse facility classes to accurately model the physical components of water networks. Its generalizability is enhanced by allowing users to modify both the physical properties of these components and the key features of the MORL simulations. RL4Water's capabilities are demonstrated through two case studies: simulations of the Nile River and the Susquehanna River, validating its accuracy and flexibility in managing both large, distributed water systems and centralized systems with complex reservoirs. By bridging the gap between water management and reinforcement learning, RL4Water offers a unified platform for developing and researching water management simulations.
...
Efficient management of water resources is increasingly critical in the face of growing challenges such as climate change and population growth. This research paper introduces RL4Water, an adaptable framework for simulating water management systems using multi-objective reinforcement learning (MORL). Adhering to the Gymnasium API standard, RL4Water ensures seamless integration with existing MORL algorithms. The framework includes diverse facility classes to accurately model the physical components of water networks. Its generalizability is enhanced by allowing users to modify both the physical properties of these components and the key features of the MORL simulations. RL4Water's capabilities are demonstrated through two case studies: simulations of the Nile River and the Susquehanna River, validating its accuracy and flexibility in managing both large, distributed water systems and centralized systems with complex reservoirs. By bridging the gap between water management and reinforcement learning, RL4Water offers a unified platform for developing and researching water management simulations.
RL4Water: Climate-Resilient Water Management via Reinforcement Learning
Investigation of Different Visualization Techniques for the Multi-Objective Reinforcement Learning Results
This paper studies the simulation of the Nile River as a multi-objective reinforcement learning problem. The main goal of this essay is to develop and evaluate the visualization techniques to effectively present the results of reinforcement learning models. Using a multi-objective approach, visualizations are very important for understanding the trade-offs and complexities in managing the Nile River problem.
This study includes a user evaluation to compare different visualizations, analyzing their effectiveness in terms of their clarity and usefulness using ANOVA test. Additionally, the effectiveness of clustering and full data points will be analyzed using a chi-square test to choose which visualisation technique works the best.
According to the results, stacked bar chart and parallel coordinates plot performed the best, while the spider plot performed the worst. Additionally, there is no preference between clustered and full data points visualizations based on the user evaluation.
...
This study includes a user evaluation to compare different visualizations, analyzing their effectiveness in terms of their clarity and usefulness using ANOVA test. Additionally, the effectiveness of clustering and full data points will be analyzed using a chi-square test to choose which visualisation technique works the best.
According to the results, stacked bar chart and parallel coordinates plot performed the best, while the spider plot performed the worst. Additionally, there is no preference between clustered and full data points visualizations based on the user evaluation.
...
This paper studies the simulation of the Nile River as a multi-objective reinforcement learning problem. The main goal of this essay is to develop and evaluate the visualization techniques to effectively present the results of reinforcement learning models. Using a multi-objective approach, visualizations are very important for understanding the trade-offs and complexities in managing the Nile River problem.
This study includes a user evaluation to compare different visualizations, analyzing their effectiveness in terms of their clarity and usefulness using ANOVA test. Additionally, the effectiveness of clustering and full data points will be analyzed using a chi-square test to choose which visualisation technique works the best.
According to the results, stacked bar chart and parallel coordinates plot performed the best, while the spider plot performed the worst. Additionally, there is no preference between clustered and full data points visualizations based on the user evaluation.
This study includes a user evaluation to compare different visualizations, analyzing their effectiveness in terms of their clarity and usefulness using ANOVA test. Additionally, the effectiveness of clustering and full data points will be analyzed using a chi-square test to choose which visualisation technique works the best.
According to the results, stacked bar chart and parallel coordinates plot performed the best, while the spider plot performed the worst. Additionally, there is no preference between clustered and full data points visualizations based on the user evaluation.