The goal of reinforcement learning is to train agents to perform tasks under little supervision. Tasks are specified by a reward function and transition function, which state how much reward the agent gets for its action in a state, and how the environment state changes based on the action the agent took. Typically reinforcement learning assumes no prior knowledge over the reward and transition function,meaning that agents need to explore the environment and learn essentially through trial and error. Model-free methods attempt to learn which actions lead to good outcomes without modeling the reward or environments itself. Efficiently selecting that actions are promising is an active research direction which can greatly reduce the number of total interactions needed for an agent to learn the task, potentially opening the door to new applications where trials or simulations are expensive or compute is limited.
Uncertainty quantification is a central mechanism in such efficient exploration methods. Provided with an estimate of how certain the agent is about the outcome of an action, it can intelligently weigh whether it is worth exploring. The Bayesian paradigm is one method to quantify uncertainty in machine learning. It models the uncertainty with a probability distributions over models, specifying how likely a model is based on the data the agent has collected.
We adopt a Bayesian point of view in model-free reinforcement learning, and develop a deeper understanding on when Bayesian reinforcement learning methods can be expected to work well and challenges that remain. To this end, in Chapter 2 we propose training ensembles through Sequential Monte Carlo, obtaining a sample from the posterior distribution of a deep Q-learning agent. We observe that agents are able to perform directed exploration, although not necessarily more efficiently than standard ensembles in every environment. Furthermore, in Chapter 3 we theoretically analyze existing Bayesian Deep model-Free Reinforcement Learning methods, and unify them into a single theoretical framework we call Epistemic Bellman Operators. We prove that these operators are contractions, establishing convergence of derived algorithms in a simplified setting. Finally, in Chapter 4 we analyze the likelihood and prior assumptions
in existing Bayesian deep model-free reinforcement learning methods, and find through statistical tests that the standard likelihood assumptions are violated on every benchmark we tested. We also find that we can improve performance of Bayesian model free reinforcement learning methods by picking different priors based on empirical data from unrelated tasks, which transfer to new environments.
This dissertation establishes several desirable properties of Bayesian Deep model free reinforcement learning, but also raises some key issues, most notably misspecification in Chapter 4. We hope our findings convince other Bayesian reinforcement learning researchers to give more attention to assumptions about priors and likelihoods.

This dissertation concerns the efficient quantification of uncertainty in the field of deep reinforcement learning. At the time of this writing, artificial intelligence is being adopted rapidly into the critical pipelines of numerous scientific and societal domains — from autonomous driving and medical diagnostics to scientific discovery. A particular class of machine learning models, deep neural networks, has been pivotal in this recent development due to their extraordinary scalability and expressive power. Such models learn by optimizing vast sets of parameters to shape predictions according to previous measurements, captured in large datasets. When we deploy such learned models for practical applications, however, they are asked to make predictions for novel inputs not represented in their training data. Such predictions are the result of inductive generalization — deriving insights about future situations from past experience — and are inherently subject to uncertainty. For these predictions to be actionable, they must often be accompanied by a reliable measure of confidence. This need to know what one does not know is addressed by the quantification of epistemic uncertainty, which arises from the imperfection of a learned model, often due to a lack of sufficient relevant data. This stands in contrast to aleatoric uncertainty — the irreducible, inherent randomness in a process — and it is this reducible, model-centric epistemic uncertainty that forms the central object of inquiry for this dissertation.

The challenge of epistemic uncertainty estimation becomes especially tangible in the context of sequential decision-making problems. In such settings, an agent’s actions can have long-term consequences that compound over time, shaping downstream outcomes and choices. Reinforcement learning, a paradigm in which agents learn such decision-making strategies through direct interaction with an environment, faces several fundamental challenges that hinge on reliable uncertainty estimation. Underpinning both efficient exploration and safe decision-making is a principled understanding of an agent’s own epistemic uncertainty — the central topic of this thesis.

Examining the current research landscape of uncertainty quantification in deep learning, we observe a persistent tension between theoretically well-motivated yet computationally expensive techniques on one hand, and computationally efficient yet less understood methods on the other. Bayesian inference, widely regarded as the gold standard for reasoning about epistemic uncertainty, is generally intractable for modern, large-scale neural networks. This has led to a spectrum of approximate methods — including deep ensembles, advanced sampling techniques, and variational inference — that navigate this trade-off to varying degrees. More pragmatic solutions often offer substantial computational savings but lack a deeper theoretical understanding of what their uncertainty estimates represent, or how they behave in practice. From this landscape, we derive the research mission for this dissertation: to develop and analyze uncertainty quantification methods that are both computationally tractable and theoretically well-motivated.

Our first line of inquiry investigates deep neural network ensembles. We hypothesize that their efficacy is constrained not merely by the number of constituent models but by the quality of their diversity. Focusing on distributional reinforcement learning, we show that architectural components such as projection operators can induce inductive biases that shape generalization. We develop diverse projection ensembles, which induce diversity through architecturally distinct members, and demonstrate empirically that this approach yields more robust uncertainty signals and improved exploration performance.

Our second line of inquiry pursues the goal of emulating ensemble uncertainty within a single, efficient model. We introduce contextual similarity distillation, a technique enabling epistemic uncertainty estimation with a single model trained via gradient descent. Using neural tangent kernel theory, we reinterpret ensemble variance estimation as a tractable kernel regression problem. Complementarily, we provide a theoretical foundation for random network distillation, showing that, in the infinite-width limit, its uncertainty corresponds to the predictive variance of a deep ensemble. We further develop a Bayesian variant whose error signal matches the posterior predictive variance of an infinitely wide Bayesian neural network.

Finally, we address the estimation of long-term, cumulative uncertainty in reinforcement learning. We propose a novel single-model method — universal value-function uncertainties — that quantifies uncertainty over entire trajectories. Grounded in neural tangent kernel theory, we show that this method yields estimates equivalent to those of a full ensemble, while retaining computational efficiency.

In conclusion, this dissertation advances uncertainty quantification in deep reinforcement learning by bridging the gap between theoretical rigor and computational practicality. It provides both methodological innovations and theoretical insights toward more reliable, uncertainty-aware autonomous agents.

Model-free deep reinforcement learning has shown remarkable promise in solving highly complex sequential decision-making problems. However, the widespread adoption of reinforcement learning algorithms has not materialized in real-world applications such as robotics. A primary challenge is the general assumption that environments remain stationary at deployment. This problem is exacerbated when agents rely on pixel-based observations, dramatically increasing task complexity. As a result, these algorithms often fail when environments change over time. The perspective that agents should learn a disentangled representation has already been shown to be effective in improving generalization to domain shifts. We extend previous work by introducing Generalized Disentanglement (GED), an auxiliary contrastive learning task that encourages pixel-based deep reinforcement learning algorithms to isolate factors of variation in the data by leveraging temporal structure. We show that our methodology can improve generalization to unseen domains in several environments.

Recent advancements in differential simulators offer a promising approach to enhancing the sim2real transfer of reinforcement learning (RL) agents by enabling the computation of gradients of the simulator’s dynamics with respect to its parameters. However, the application of these gradients is often limited to specific scenarios. In this thesis, we address these limitations by proposing methods to obtain accurate gradients through the use of a privileged value function. This approach provides valuable insights into the effectiveness of differential gradients and demonstrates that, in certain cases, it can significantly improve sim2real performance. To illustrate this, we develop an adversary that identifies the worst-case domain parameters for a given policy using local gradients. Our experiments are conducted on the Pendulum swing-up environment. This thesis forms the basis for the exploration of further possibilities of leveraging differential simulator gradients.

Recent work has shown that offline reinforcement learning (RL) does not generalize well to new environments compared to behavioral cloning (BC). We propose WSAC-N, an ensemble model of soft actor-critics with weights to de-emphasize actions with high variance. We compare the zero-shot generalization abilities of WSAC-N with the baseline BC in a four-room maze-like environment, testing on unseen tasks. Our findings indicate that WSAC-N has worse zero-shot generalization compared to BC, aligning with previous work. Additionally, we investigate the impact of dataset characteristics on generalization, finding that dataset size has a negligible impact, while the quality of trajectories generally has a positive effect. These results are consistent with prior research.

In the field of Artificial Intelligence (AI), techniques like Reinforcement Learning (RL) and Decision Transformer (DT) are utilized by machines to learn from experiences and solve problems. The distinction between offline and online learning determines whether the machine learns from a live environment or simply observes pre-recorded actions. The difference between single-task and multi-task settings indicates whether the machine can handle similar but not identical tasks. Multi-task, offline learning, the focus of this paper, allows machines to address a variety of related tasks, based on a pre-recorded set of experiences. This approach is particularly valuable in situations where traditional training methods are costly or challenging. For instance, in robotics, multi-task, offline learning enables robots to use experiences from various tasks, such as picking up objects, to solve new problems like placing them down. This research paper explores the effectiveness of Decision Transformers in multi-task environments through theoretical discussions and practical examples. It also tries to answer the question, of how introducing sub-optimal training data, affects the performance or generalisation ability of the model.

This paper examines the generalization capabilities of the Soft Actor-Critic (SAC) algorithm when combined with Behavioral Cloning (BC) in a MiniGrid Four-Room Environment. Reinforcement learning (RL), particularly offline, is important for tasks where interactions with the environments are risky or costly, and this research focuses on multi-task environments where generalizability to new tasks is crucial. Our findings indicate that SAC+BC can achieve generalization performance close to BC. Notably, while BC shows robustness across various dataset characteristics (quality, diversity, size), SAC alone struggles without integrating BC, highlighting the enhancement in generalization brought by this hybrid approach. Furthermore, an increased data size only enhances generalizability when introducing greater diversity. However, these results are constrained by hardware limitations, suggesting that further hyperparameter optimization and using more seeds could validate and possibly enhance our findings, demonstrating that SAC+BC is even more effective
than shown. The implementation details and the source code for this study are available on GitHub at https://github.com/AxelGeist/multi-task-offlinereinforcement-learning.

Offline Reinforcement Learning (Offline RL) involves learning policies from a static dataset without further interactions with the environment, making it suitable for high-stakes scenarios where data collection is costly or risky. This paper investigates the generalization capabilities of Implicit Q-Learning (IQL), an offline RL algorithm, compared to Behavioral Cloning (BC). We adapt the IQL algorithm for discrete control and evaluate both IQL and BC in a four-room environment using training datasets generated from different behavioral policies. Performance is assessed based on average rewards over various test seeds, on reachable and unreachable tasks, as well as the training set. Our results indicate that BC consistently outperforms IQL across all scenarios, although IQL reaches peak performance faster. This study highlights the need for further research into offline RL algorithms for better generalization and more robust performance in diverse environments. Full code available on GitHub.

Meta-learning is an important emerging paradigm in machine learning, aimed at improving data-efficiency and generalization performance across learning tasks. Challenges caused by noisy data has been extensively researched in traditional learning settings. However, its impact in the context of meta-learning, especially concerning label noise in meta-training, remains under-explored. Curriculum Learning (CL), is an approach where training data is ordered from easy to complex, and models learn from easier to harder samples. A type of CL , Self-Paced learning (SPL) offers adaptive data curriculum, where ordering is based on per sample model performance during training. Self-Paced Learning (SPL) has proven effective in enhancing model robustness and convergence under noisy data scenarios. However, its application in meta-learning under these conditions remains limited. In this paper, we use a Neural Process model on 1D sinusoidal function regression tasks, with different ratio of clean / noisy training data scenarios to empirically observe the same benefits SPL can potentially offer for noisy meta-learning. In line with findings in traditional learning settings, SPL improved overall training convergence, also lead to increase in generalization thus noise robustness. Furthermore SPL lead the model to be more robust to increasing scale of noise for tasks within the training data distribution.

Meta-Learning is an emerging field where the main challenge is to develop models capable of distilling previous experiences to efficiently learn new tasks. Curriculum Learning, a group of optimization strategies, structures data in a meaningful order which aids learning. However, the extent to which curriculum strategies can optimize the performance of meta-learners remains unclear. Here we study the separate and joint effects of a model-based (ScreenerNet) and a statistics-based (Active Bias) curriculum strategy on the training of a meta-learning model (Neural Processes) which solves 1-D function regression tasks. The findings show that ScreenerNet increases in-task accuracy and accelerates convergence, but decreases the generalization performance. Active Bias achieves mixed generalization results and significantly decreases training efficiency when trained on noisy data-sets. Combining them partially mitigates ScreenerNet's overfitting and stabilizes Active Bias' susceptibility to noise, but further research is necessary in order to achieve consistent improvements to the baseline.

We investigate whether a teacher-student curriculum learning approach using a teacher network with a simpler structure than the student network can achieve better results at meta-learning. The goal of meta-learning is to learn from a set of tasks, and then perform well on a new, structurally similar but unseen task with minimal retraining. Instead of sampling uniformly from all data to create the training batches, the curriculum-learning approach aims to create a sequence of mini-batches that enhances the training process, also known as a curriculum. During teacher-student curriculum learning a "teacher" network is trained in the standard manner, and then its outputs are used to order the training samples by difficulty and categorise them into mini-batches. This curriculum is then used to train the "student" network. Previous teacher-student models either had pre-trained more complex teachers, or teachers with the same structure as the student network. We investigate whether a teacher network with a simpler structure can also increase accuracy, while preserving computational resources. We find that using such a curriculum worsens performance compared to not using any curriculum at all.

Agents improve by interacting with an environment and planning. By leveraging information about what they don't know, they can learn better and faster, at least in environments that benefit from exploring. They do this by estimating the uncertainty in their predictions. There are choices for how to estimate the uncertainty, and in this work, we look at what effect this choice has on the exploration and strength of agents playing board games. We compare the effect of a source of uncertainty which perfectly tracks what the agent has seen, and a source which generalizes. We also describe the challenges associated with tuning uncertainty estimators and show what considerations have to be made when exploration is not all you need.

Reinforcement learning (RL) is a type of machine learning where a model learns by making an observation of the current state it is in, picking out an action to execute, and observing the reward of said action, after which it receives the next state and repeats the cycle until it reaches its goal. The traditional online training approach allows the agent to directly interact with the live environment, but that is not always possible due to the live environment possibly being too dangerous or costly to train in. In cases like these offline training, which instead trains the agent on already pre-collected datasets of previously mentioned interactions and tries to learn a better policy than the one used for collection, offers a viable alternative by employing Q-Learning methods like CQL. However, prior studies, such as Mediratta et al., have suggested that Behavior Cloning (BC), a type of imitation cloning, may outperform modern offline RL methods in the multi-task setting, where model generalization is tested on new or similar tasks rather than the ones trained on. Considering these results, it begs the question whether it is worthwhile to employ modern Q-Learning methods designed to derive a better policy than the one used to collect the data, especially when they are unable to outperform standard imitation learning.
This study seeks to reproduce and extend these findings within a custom environment.
The results reveal that, contrary to the aforementioned report, BC does not consistently outperform CQL. Both machine learning methods exhibit comparable performance across datasets varying in diversity and size. Additionally, incorporating more diverse data significantly enhances generalization performance.

This paper explores the application of evolutionary algorithms to enhance task generation for Neural Processes (NPs) in meta-learning. Meta-learning aims to develop models capable of rapid adaptation to new tasks with minimal data, a necessity in fields where data collection is costly or difficult. By integrating evolutionary strategies, we aim to enhance the efficiency and robustness of NPs. We evaluate our approach using 1-D function regression problems, where Genetic Algorithm generates diverse and challenging tasks. Our results show that the evolutionary approach improves learning efficiency and model performance, achieving lower Root Mean Squared Error (RMSE) compared to traditional methods.

In Reinforcement Learning (RL), an agent learns to make decisions by interacting with an environment and receiving feedback in the form of rewards. Multi-Task Reinforcement Learning (MTRL) extends this concept by training a single agent to perform multiple tasks simultaneously, allowing for more efficient use of resources and behavior sharing between tasks. Policy Distillation (PD) is a technique commonly used in MTRL, where policies from multiple single-task agents (teachers) are distilled into a single multi-task agent (student). This is done by merging common structure across tasks, while separating task-specific properties.

However, existing PD approaches require interactions with the environment during training. In this work, we investigate the effectiveness of PD in the offline setting, where the agent has no interaction with the environment before deployment and can only learn from previously collected data. Through a series of experiments, we demonstrate that a straightforward approach yields the highest performance. This approach involves first learning teacher policies using an existing offline RL algorithm, then distilling these policies into a student by sampling states from the offline data and applying a Mean Squared Error (MSE) loss between the teachers’ and student’s best actions. Moreover, we investigate the effect of a state distribution shift—a major challenge in offline RL—on our approach. We find that such shifts impact performance only slightly in cases of relatively small neural networks or substantial distribution shifts.

We also explore how PD can be enhanced to better capture common structure across related tasks, a key to improving efficiency in MTRL. To this end, we formally define common structure at two levels: the trajectory level and the computational level. To the best of our knowledge, we present the first attempt to quantify the amount of common structure shared across tasks. This measurement reveals that task commonalities are not fully exploited automatically. At the computational level, we attempt to improve sharing of common structure by reducing the network size and adding a regularization term to the loss function. To capture more common structure at the trajectory level, we argue that multi-task exploration is required, meaning that behaviors from one task must be evaluated in the context of another task. We propose two extensions to our approach that introduce multi-task exploration: Data Sharing (DS) and Offline Q-Switch (OQS). While these extensions are capable of improving performance, they also have clear limitations.

Overall, we propose a new, high-performing offline MTRL method and provide valuable insights into the fundamental capabilities and limitations of PD in capturing common structure across tasks, specifically within the offline MTRL setting.

Reinforcement learning techniques have demonstrated great promise in tackling sequential decision-making problems. However, the inherent complexity of real-world scenarios presents significant challenges for its application. This thesis takes a fresh approach that explores the untapped potential of factored state representations as a means to enhance the efficiency of reinforcement learning.

Factored representations involve variables describing various features of the environment. These variables, along with their possible values, define the agent’s states. Unlike standard representations, factored representations provide a unique perspective that enables us to gain deeper insights into the underlying structure of the environment and refine our understanding of the problem at hand.

By analyzing variable dependencies, we can abstract simplified representations of the environment states and construct computationally lightweight models. To do so, we will explore potential factorizations of key functions governing the reinforcement learning problem, such as transitions, rewards, policies, or value functions. These factorizations can be achieved by exploiting variable redundancies and leveraging relations of conditional independence.

This thesis proposes a set of methods that are shown to improve the efficiency and scalability of reinforcement learning in complex scenarios. We hope that the findings of this research contribute to showcasing the potential of factored representations and serve as inspiration for future research in this direction.

Bayesian Optimization (BO) has demonstrated significant utility across numerous applications. However, due to it being designed as a universal optimizer, its performance can often be suboptimal in specialized environments. To overcome this issue, research has been conducted into the application of transfer learning for enhancing BO performance in these specialized contexts. This paper describes the research done into evaluating the MetaBO algorithm in some specific environments. MetaBO innovates by substituting the acquisition function component in BO with a neural network that serves as an acquisition function, trained via a reinforcement learning framework. Although the results indicate that the algorithm's performance is not optimal in the environments tested, these limitations are ascribed to elements of the implementation rather than the concept of the algorithm itself. Consequently, further research is necessary to refine the implementation process and fully exploit the potential of the MetaBO algorithm.

Autonomous driving is a rapidly evolving field that aims to enhance road safety and reduce accidents through the use of advanced software and hardware technologies. Reinforcement learning (RL) combined with deep neural networks has emerged as a promising approach for training autonomous agents. This research paper investigates three exploration algorithms —Epsilon-Greedy, Random Network Distillation (RND), and Bootstrapped Deep Q-Network (DQN)— within the context of autonomous driving. Performance is assessed based on episodic returns in training and testing environments, as well as the time required to train the networks. The results show significant improvement in learning capability using Bootstrapped DQN without critical differences in training time. There also exists a potential to increase episodic returns further given an increase in the number of steps to train the models.

Reinforcement Learning (RL) has gained atten-tion as a way of creating autonomous agents for self-driving cars. This paper explores the adap- tation of the Deep Q Network (DQN), a popular deep RL algorithm, in the Carla traffic simulator for autonomous driving. It investigates the influ- ence of action space discretization and DQN ex-
tensions on training performance and robustness. Results show that action space discretization en- hances behaviour consistency but negatively af- fects Q-values, training performance, and robust- ness. Double Q-Learning decreases training per- formance and leads to suboptimal convergence, re- ducing robustness. Prioritized Experience Replay
also performs worse during training, but consis-tently outperforms in robustness testing, reward es-timation and generalization.

Autonomous driving is a complex problem that can potentially be solved using artificial intelligence. The complexity stems from the system's need to understand the surroundings and make appropriate decisions. However, there are various challenges in constructing such a sophisticated system. One of the main challenges is to make the agent learn from the environmental input and since the environment is not fully observable and constantly changing, the agent should be flexible enough to extract important information from the input. To create such an agent this paper focused on the 3 different methods, specifically the application of frame stacking, long short-term memory (LSTM), and a combination of both methods. To analyze these methods 3 Deep Q Network(DQN) based agents are created. These 3 agents have frame stacking, LSTM, and both combined methods to solve the partially observable problem of autonomous driving. Their training performances are analyzed, and the results revealed significantly different trends in the training and evaluation phase.
Especially the experiment resulted in LSTM being a more robust method but had lower performance than DQN with frame stack, which showed a trade-off between these 2 qualities of an agent. The agent with LSTM and frame stack was able to learn faster at the beginning, but as it was unstable, it got a lower return value from the training run at the end. This instability can be a problem in the real world, especially in autonomous driving, where it is very important to have robust implementation.