Inverse Reinforcement Learning (IRL) is a subfield of Reinforcement Learning (RL) that focuses on recovering the reward function using expert demonstrations. In the field of IRL, Adversarial IRL (AIRL) is a promising algorithm that is postulated to recover non-linear rewards in environments with unknown dynamics. This study investigates the potential benefits of applying the Curriculum Learning (CL) strategy to the AIRL algorithm. For our experiments, we use a randomized partially observable Markov decision process in the form of a grid-world-like environment. Using only expert demonstrations obtained with an RL algorithm under the true reward function, we train AIRL in a variety of configurations and identify an effective curriculum. Our results show, that a well-constructed curriculum can enhance the performance of AIRL twofold in both key aspects: the speed of convergence and the efficiency of using expert demonstrations. We thus conclude that CL can be a useful addition to an AIRL-based solution. Full code is available online in the supplementary material https://github.com/mikhail-vlasenko/curriculum-learning-IRL.

Inverse Reinforcement Learning (IRL) aims to recover a reward function from expert demonstrations in a Markov Decision Process (MDP). The objective is to understand the underlying intentions and behaviors of experts and derive a reward function based on their reasoning, rather than their exact actions. However, expert demonstrations can be influenced by various types of noise (e.g., from random behavior) which can affect their accuracy and effectiveness in solving the MDP. This research investigates the capability of IRL to recover reward functions from noisy demonstrations. Three types of noises, namely Random Action Noise, Random Bias Noise, and Sparse Noise, are introduced and modeled. Demonstrations are generated with these noises, and the corresponding reward functions are recovered. Comparisons are made between the noisy and optimal recovered rewards using various metrics. The results indicate that IRL exhibits certain tolerance level against Random Events and Sparse Noise, while being more vulnerable to Random Bias Noise.

This paper aims to investigate the effect of conflicting demonstrations on Inverse Reinforcement Learning (IRL). IRL is a method to understand the intent of an expert, by only feeding it demonstrations of that expert, which may be a promising approach for areas such as self driving vehicles, where there are a lot of demonstrations from experts. This paper aims to investigate the effect of conflicting demonstrations on IRL. Demonstrations may not always come from the same expert or the expert may prioritize different goals at times. For example, a driver may not always do grocery shopping at the same store or they may take a slightly different route on different occasions. The results showcase a negative effect from severely conflicting demonstrations on the ability of Max Entropy IRL to recover rewards, but do show some slightly optimistic results on more than two goals.

A key issue in Reinforcement Learning (RL) research is the difficulty of defining rewards. Inverse Reinforcement Learning (IRL) is a technique that addresses this challenge by learning the rewards from expert demonstrations. In a realistic setting, expert demonstrations are collected from humans, and it is important to acknowledge that these demonstrations can deviate from rationality due to systematic biases known as cognitive biases. One group of cognitive biases, known as risk-sensitive cognitive biases, pertains to individuals' attitudes and behaviors towards risk and uncertainty. This paper investigates the extent to which IRL can learn from demonstrations that contain risk-sensitive cognitive biases such as loss aversion and risk aversion. Modelling biases using concepts from Prospect Theory and System 1 and 2 model and using Maximum Entropy IRL algorithm, this paper concludes that IRL can recreate similar solutions to experts but inferring the underlying motivations and the interactions between them is an intricate problem that requires novel approaches.

Purpose: This paper explores the potential of machine learning (ML) algorithms to mitigate uncertainty in early environmental assessments (ex-ante LCA), which are hindered by prospective nature and limited quantitative data availability. Methods: A systematic literature review with keyword searches on Scopus identified three ML categorization groups in ex-ante LCA: streamlined LCA, ex-ante LCA parameter projection, and ancillary models and data. Two following case studies addressed literature gaps in price forecasting for economic allocation and recycling rate projections. Results: In streamlined LCA, 16 studies linked molecular and technical parameters to project production-related emissions of organic chemicals, applied product clustering of product groups, and generated spatially explicit impact category results. The application of ex-ante LCA parameter projection, as evidenced by 14 publications, involves the use of ML to project life cycle inventory (LCI) data, project characterization factors, and integrate natural parameters with LCI data in a comprehensive modeling approach. In nine other papers the applications to ex-ante LCA remained undefined but potentially applicable. For both case studies, best results were obtained with a Recurrent Neural Networks (RNN) algorithm with long-short-term-memory (LSTM). Commodity price forecasting in the first case study achieved a projection accuracy of 0.96 (MSE), 0.98 (RSME), and 10.17% (MAPE) for copper and 88.86 (MSE), 9.43 (RMSE), and 21.23% (MAPE) for molybdenum. Probability modelling is identified as a modeling approach which incorporates uncertainty. The recycling rate forecast case study identified plastic recycling and glass recycling rates as the best suiting covariates and demonstrated multivariate modeling possibilities with 0.22 (MSE), 0.48 (RSME), and 0.38% (MAPE) in a model with 68 covariates. Discussion: A limited yet growing body of literature indicates that ML applications in ex-ante LCA represent an emerging field of science. While streamlined LCA shows promise, it faces constraints related to data precision and a static nature. In the ex-ante LCA parameter projection categorization, the sub-group of similarity clustering of LCI processes suffers from data uncertainty in LCI databases, making the approach more suitable for updates of existing technologies than for emerging ones. On the other hand, LCI generation through ex-ternal parameters represents a highly technology-specific case, showing significant promise. The projection of characterization factors and the sub-group of integrated modeling are identified as promising, but the limited number of scientific studies hinders the generalizability of these findings. Case studies on price forecasting and recycling rate projection demonstrate ML’s applicability in economic allocation and waste treatment projections. Overall, the results suggest that ML holds potential for reducing uncertainty in ex-ante LCA, laying the groundwork for focused research and contributing to a nuanced understanding of uncertainty reduction in this domain. Recommendations: The paper emphasizes the need for targeted research in the goal and scope phase and in End-of-Life (EoL) treatment forecasts, e.g. via the use of time-series multivariate modeling. Furthermore, it encourages further exploration of streamlined LCA into applications with a high degree of technical predictors, along with the extended projection of characterization factors and integrated modeling. Additionally, the use of probabilistic modeling as a tool to incorporate uncertainty into the modeling is recommended, aiming to enhance the applicability and transparency of ML applications for reducing uncertainties in ex-ante LCA.