Optimization models are widely used in energy system planning to identify cost-effective investment strategies. However, relying solely on a single optimal solution can be misleading, as it fails to account for model uncertainty, competing objectives, and stakeholder preferences. To address this, near-optimal alternatives, solutions that are close in cost to the optimum but structurally different, are increasingly used to support robust and flexible decision-making.

This thesis explores the generation and evaluation of near-optimal alternatives within energy systems, with a focus on improving the decision relevance of the generated alternatives. This thesis introduces a unified analytical framework, formalizing existing Modeling to Generate Alternatives (MGA) methods using weight vector formulations. This formulation enables a clearer comparison of different techniques that generate these alternatives. This analysis highlights the limitations of current evaluation metrics, particularly their inability to distinguish decision-relevant alternatives from decision-irrelevant ones.

To overcome this gap, the thesis proposes a novel evaluation metric based on dominance relations from multi-objective optimization. This metric identifies non-dominated alternatives, those not strictly worse than any other across all decision variables, as decision-relevant. The thesis introduces a new method that uses Directionally Weighted Variables to generate alternatives aligned with this dominance criterion.

The proposed approach is evaluated using a stylized energy investment model and benchmarked against existing MGA techniques. Results show that traditional methods tend to generate fewer non-dominated alternatives, while the new method generates more non-dominated alternatives within the near-optimal space. This work contributes a new perspective on alternative generation, bridging the gap between mathematical optimality and practical decision support.

In this thesis, we investigate how representative periods can be used as a temporal reduction technique for stochastic programming formulations of large-scale energy models. We specifically apply this to generation expansion planning. The focus is on cost-efficient decisions as well as on robust decisions, especially since the latter is why uncertainty is incorporated in most energy models. For this, we build on recent work that uses instead of traditional clustering methods like $k$-means or $k$-medoids, a method to obtain a convex or bounded conical hull over the periods. We prove that when inter-period constraints are ignored and a perfect hull is found, we can ensure that a feasible solution for the original model exists, obtained from the optimal decision variables of the reduced model. The proof is suitable for both the traditional way of using representative periods in stochastic programming, as for an approach that shares representative periods across scenarios instead of making them scenario-dependent. We show that a greedy implementation of the hull approach can outperform standard clustering methods in terms of costs in small case studies when a valid hull is found. The near-optimal costs can be obtained with a low number of representatives, largely due to the absence of loss of load due to the hull methods. However, in more complex cases, such as a European-scale model with high renewable shares, finding such a hull with the greedy algorithm proves difficult. To address this, we propose adding extreme periods either beforehand or afterwards, and applying a blended weights approach to reduce conservativeness while maintaining feasibility. Our results on small case studies demonstrate that extreme representatives can significantly reduce loss of load, although not always at lower cost when applied to the large European case study. These findings suggest that targeted selection of extremes, a right weighting approach and improved hull approximations offer a promising direction for enabling scalable, robust planning under uncertainty. 

The transition towards renewable energy requires long-term energy system planning, which depends on solving constrained optimization (CO) problems. These CO problems are becoming increasingly complex, particularly due to the variability introduced by renewable energy sources. Traditional optimization methods struggle to scale with this growing model complexity. In contrast, machine learning approaches allow faster solution computation, but offer only approximate solutions with no guarantees on solution quality, thereby limiting reliability and interpretability in planning applications.

This research addresses these limitations by exploring the use of neural networks to predict feasible solution pairs for the primal and dual formulations of the CO problem, enabling approximate solutions accompanied by bounds on suboptimality. Self-supervised primal-dual learning (PDL) is adapted and extended to produce paired feasible solutions for both the primal and dual formulations of an economic dispatch problem. Feasibility in the primal network is enforced through differentiable repair and completion layers, including novel domain-specific extensions that adjust supply-demand balancing priorities, which were found to be essential for predictive accuracy. This then exposes a structural limitation of PDL: while repair and completion layers are essential for primal learning, they prevent the learning of meaningful dual predictions. To address this, the dual network is trained using a modified loss function that directly optimizes the dual problem, enabling the use of a completion layer adopted from the literature to ensure dual feasibility. An additional novel classification-based layer that incorporates prior knowledge on the dual variables further improves the dual prediction quality when applied to the economic dispatch problem. Finally, the trained networks are integrated into Benders decomposition, a technique that breaks CO problems into easier, independent problems. This enables a hybrid approach: approximate solutions with bounds on suboptimality can be obtained swiftly, whereas exact solving remains available if necessary, retaining all theoretical convergence and optimality guarantees and potentially reducing computational time.

The proposed framework is evaluated on a generation expansion planning problem, where the economic dispatch subproblems are solved iteratively using the trained networks. The results demonstrate a theoretically grounded and empirically validated proof of concept for producing solutions with quality guarantees using learning-based methods, which is generally applicable to any problem compatible with Benders decomposition where the resulting subproblem admits a conic formulation. However, the results do not show conclusive speed-ups due to a mismatch between the training data and the data encountered during Benders iterations. By demonstrating how neural networks can be used to generate approximate, feasible solutions accompanied by theoretical guarantees on solution quality, this research contributes to the advancement of scalable and reliable constrained optimization methods for energy systems.

This thesis investigates the application of alternating maximisation for active wake control in wind farms, focusing on both numerical static yaw optimisation and multi-agent deep reinforcement learning for dynamic yaw control. As the size and number of offshore wind farms continue to grow, effectively mitigating wake effects ---~where the airflow behind a turbine is disturbed, reducing the efficiency of downstream turbines~--- becomes increasingly important to maximise power output and ensure economic profitability.

Current numerical strategies are either sub-optimal, as they do not leverage the shape of the optimisation surface well, or expensive to optimise due to the computational demands of high-fidelity simulations. Similarly, current reinforcement learning approaches are too sample inefficient on contemporary wind farms of 40+ turbines to learn high-quality policies and likely suffer from credit assignment problems, other agents performing exploration, and relative over-generalisation. Alternating maximisation emerges as a promising candidate to address these challenges by providing an efficient and potentially more effective optimisation method.

In static yaw optimisation, the research analysed the effects of wind direction, wind speed, turbine layout, and yaw misalignment on the optimisation surface. Key findings indicate that Nash equilibria are transient and significantly influenced by these factors.

The sampling-based approach to alternating maximisation outperformed other methods, proving robust against variations in initial yaw configurations and optimisation orders. This approach required fewer samples and appears suitable for practical applications.

For dynamic yaw control, alternating maximisation with policy gradients demonstrated greater sample efficiency compared to distributed policy gradient methods. However, convergence to sub-optimal policies was common due to limited exploration capabilities. An oscillating exploration strategy enabled effective exploration of yaw angles, leading to high-quality policies.

The study primarily focused on farm power output, neglecting structural loading and leading-edge erosion. Future research should integrate these factors and explore dynamic yaw control under time-varying wind conditions.

Overall, alternating maximisation shows potential in enhancing wind farm performance through efficient optimisation strategies. Future work should refine these methods and address identified limitations to maximise practical applicability.

The increasing demand for electricity has lead to demand for more efficient energy production. One promising option is wind power, which currently provides an estimated 7.8% of the world’s energy production. One of the problems with wind energy is that a small percentage of the energy is lost due to the wake effect. The wake of a wind turbine is an area of low wind speed and high turbulence which is caused by the spinning of the turbine. This wake effect can mitigated by active wake control, which is a process by which the wake from a turbine is redirected away from downwind turbines, by changing the yaw of the turbine head. Calculating a policy for doing this is computationally expensive to do using numerical optimisation. Therefore, multi agent reinforcement learning is proposed to learn a policy which performs active wake control.
The proposed approach makes use of the popular reinforcement learning algorithm REINFORCE, and extends it using a variety of methods. First, a simplified version of the problem is treated, wherein the wind direction is fixed. Then the problem is made more realistic by introducing changing wind directions. The first extension of REINFORCE that is treated is difference rewards, a reward shaping strategy which seeks to solve the credit assignment problem, thereby improving cooperation between turbines. The second method uses training regimes, which train different agents at different times to stabilise the environment as much as possible. Next, role-based reinforcement learning is used to conteract the complexity of the problem by allowing each agent to specialise for a certain role. Finally, since roles cannot be manually determined for larger farms, influence-based abstraction is used to enable agents to learn the roles themselves, by abstracting spacial information and presenting it to the agent as an observation.
The results demonstrate that multi agent reinforcement learning can be used to perform active wake control in wind farms. Furthermore, the extensions proposed are shown to improve learning, and lead to greater energy output. While multi agent reinforcement learning is shown to be a promising way to tackle active wake control in wind farms, research is needed to improve the stability of the learned policies.

The close proximity of wind turbines to one another in a wind farm can lead to inefficiency in terms of power production due to wake effects. One technique to mitigate the losses is to veer from their individual optimal direction. As such, the wakes can be steered away from downstream turbines in order to increase the overall power output. Multi-Agent Reinforcement Learning (MARL) models the interactions between wind turbines and determines an optimal control strategy through agents that learn the collective consequences of their actions. To
analyse the benefit of multi-agent cooperation and centralised critic evaluation, I investigated the effect of Counterfactual Multi-Agent Policy Gradients (COMA) on Active Wake Control. Ultimately, experiments on wind farms of three and sixteen turbines indicate that the algorithm performs moderately, yet worse than single-agent Reinforcement Learning. In addition, high computation costs hinder its application on real-life environments.

Wind energy, generated by windfarms, is playing an increasingly critical role in meeting current and future energy demands. windfarms, however, face a challenge due to the inherent flaw of wake-induced power losses when turbines are located in close proximity. Wakes, characterized by regions of turbulence and lower wind speed, are created as air passes through the rotors, reducing the efficiency of downstream turbines. Wake losses can be reduced by yawing upstream turbines to steer the wake away from downstream turbines. While there are power losses associated with turning turbines off-wind, the gains in the subsequent turbines can outweigh these losses. Yawing turbines to increase overall power output is known as Active Wake Control, and the literature shows that single-agent reinforcement learning algorithms can be used to learn such control policies. However, these approaches are limited to a small number of turbines and do not scale well to larger windfarms. Multi agent reinforcement learning algorithms do scale to larger windfarms and this paper investigates the application of the DGN algorithm for windfarm active wake control. DGN is a fully cooperative algorithm that utilizes a graph representation of agents to encourage collaboration among neighboring turbines. The use of DGN is particularly interesting because windfarms naturally have a topological structure, and depending on the modeling choices, graphs can capture a significant amount of information. This paper demonstrates that DGN can learn useful policies for wake control. Although it does not outperform the DQN single-agent algorithm on small windfarms, its advantage becomes apparent in larger windfarms, where its performance remains consistent while DQN’s performance deteriorates.

When multiple wind turbines are positioned close to one another, such as in a wind farm, wind turbines located downwind of other turbines are not 100% efficient due to wakes, negatively affecting the total power output of the wind farm. A way to mitigate the loss of power is to steer the wake away from the next turbine, which lowers the current turbine's power output but increases the turbine's total power output. As the number of wind turbines increases, how complex it is to calculate the optimal steering increases exponentially. Reinforcement learning techniques have been a promising candidate to solve this problem. However, single-agent techniques are still very computationally expensive when the number of turbines is the same as an average wind farm. Therefore, this paper aims to see how the QMIX algorithm, a multi-agent reinforcement learning technique, can be efficiently applied to the problem of active wake control. QMIX will be compared to the FLORIS model and a single agent deep reinforcement learning technique TD3 to see if it achieves a higher average reward and converges faster. Finally, QMIX is tested on a larger wind farm to see if it achieves any results in a reasonable amount of time, showing that multi-agent reinforcement learning techniques are more suitable to the problem. This paper shows QMIX has the potential to outperform TD3; although FLORIS is better for smaller wind farms but more research has to be done in applying QMIX to the problem.

The wake effect which is turbulence behind a wind turbine created when it extracts energy negatively impacts the power output of the downstream turbines. Active Wake Control can mitigate this effect, by rotating some turbines away from the wind. Previous research applied single agent reinforcement learning to apply Active Wake Control, show- ing good results for small-scale layouts, that don’t scale for larger, practical wind farms.
To that extent, this study focuses on the application of mean-field multi-agent reinforcement learning to Active Wake Control, under constant wind conditions. This algorithm limits the computations to a limited set of neighbouring turbines, reducing their complexities. To build the answer to this question I will also study:
1. how to model the rewards to solve the lazy- agent problem, leveraging the nature of the Active Wake Control
2. how the view of the agent changes the results
3. how does it compare to a single-agent reinforcement learning algorithm, TD3
The experiments were done using the Floris Wake Simulator, with each turbine sharing the same agent, placed in tunnel layouts at real-life distances (6-7 rotor diameters), under constant wind conditions.
Results show that with the proper configuration of rewards and view space within wind tunnels, the mean-field algorithm finds near optimal configurations for Active Wake Control, within a small number of episodes. This shows a promising start for the application of mean-field multi-agent algorithms for the Active Wake Control problem, and provides insight into how to model the rewards, which might be applicable for the whole class of algorithms.

In wind farms wind turbines are often placed close to each other. Each turbine generates a turbulent wake field, this field negatively affects subsequent turbines. This can cost more than 12% efficiency. To decrease this loss we can steer the turbines away from the wind direction, this will decrease the individual turbine power output, but can increase the total power output of the farm. As the size of the farm increases the number of possible actions increase exponentially. Due to this a numerical solution is not feasible. A reinforcement learning technique has been proven useful in the past, but a standard single agent implementation is still very computationally expensive. We evaluate the effectiveness of MADDPG on the active wake control problem. MADDPG is a multi agent reinforcement learning algorithm. MADDPG will be compared to the numerical solver FLORIS and to the already implemented and proven TD3 (which is a variation on a single agent DDPG algorithm). We compare the eventual output power of the algorithms with MADDPG. From the results we can see that MADDPG does improve on the learning performance of TD3, but since MADDPG needs to manage more neural networks the overhead is larger. MADDPG reaches an optimum solution in less training steps, but these steps take significantly more time.

Automated asset trading is a crucial method used by financial entities such as investment firms or hedge funds. It allows them to allocate their capital in order to maximize their rate of returns. In scientific literature, there are multiple models suggested to solve this problem. However, these models either lack the complexity to understand the market or scalability for the market in general. On the other hand, deep reinforcement learning is a great framework that can solve these problem. In this study we aim to understand the performance of model-free deep reinforcement learning algorithms in terms of training speed, financial performance and generalizability by training and comparing them on a smaller representative market. Proximal Policy Approximation (PPO) and Twin Delayed Deep Deterministic Policy Gradient (TD3) were used as a representatives of policy approximation and QLearning algorithms respectively. Our study have found that while proximal policy algorithms offer higher speed due to smaller training data they use at each timestep, Q-Learning algorithms offer a better general performance in terms of stability and generalizability. With respect to financial performance on training stocks, this study did not find a statistically important difference in performances.

One of the most challenging types of environments for a Deep Reinforcement Learning agent to learn in are those with sparse reward functions. There exist algorithms that are designed to perform well in settings with sparse rewards, but they are often applied to continuous state-action spaces, since economically relevant problems like robotic control and stock trading fall under this category. This means the continuous version overshadows the discrete state-action version of the sparse reward problem. Furthermore, research that focuses on sparse rewards is lacking in comparisons of algorithms dedicated to performing in this type of setting with other state-of-the-art Deep Reinforcement Learning algorithms. We devise an experimental setup to test a selection of algorithms from three state-of-the-art Deep Reinforcement Learning approaches; Hindsight Experience Replay, Maximum Entropy Reinforcement Learning and Distributional Reinforcement Learning. We show that as the cardinality of the state spaces in sparse reward settings increase, Hindsight Experience Replay approaches are superior in sample efficiency compared to the other two approaches studied.

The current state-of-the-art solutions for playing Chess, are created using deep reinforcement learning. AlphaZero, the current world champion, uses ’policy networks’ and ’value network’ for selecting moves and evaluating positions respectively. However, the training of these networks are done using reinforcement learning from games of selfplay. There are many factors which determine the learning speed of reinforcement learning algorithms, where the size of the search space is a main one. In this research, we have tried to see the effect of the size of the search space on the time it takes the reinforcement learning agent to learn.