This thesis concerns the fundamental problem of learning the behavior of decisionmaking agents using only observations of how they act in different situations. As humans, we do it all the time, and have been doing it since birth: think about how a child learns to speak and walk. However, this thesis does not only focus on imitating. We go a step further and try to learn why an agent does what it does. In other words, what is the agent’s objective, which led them to act in a certain way? This is a much harder question, but also much more rewarding when answered correctly: knowing the agent’s motivation allows us not only to imitate but also to understand or even influence the agent’s behavior. For this purpose, we ask the question: what is the agent optimizing for when making decisions? For instance, imagine a consumer agent that buys a certain set of products given their budget. To model the consumer’s behavior, we interpret their action as buying the products with maximum utility, given a limited budget. Thus, learning how the consumer evaluates each product (that is, the utility of each product for the consumer) would allow us to understand, replicate, and possibly influence their behavior. Mathematically, we model the decision process of the agent as an optimization program, and we use Inverse Optimization (IO) as a tool to “reverse engineer” the agent’s optimization program from observed behavior. This thesis can be divided into two parts. First, we dive into the mathematical formalization of the IO problem. We look at the geometry of thema thematical objects emerging from IO problems, and we discuss what it means to solve the IO problem and different ways to do it, proposing tractable reformulations and efficient algorithms. In the second part of this thesis, we develop a tailored IO methodology to solve IO problems emerging from routing problems. We test the potential of our methodology for modeling human driving behavior on real-world problems using data from the Amazon Last Mile Routing Research Challenge. We achieve excellent results, showcasing the potential of IO to solve real-world problems. Additionally, we also developed Inv Opt, an open-source Python package to solve general IO problems. @en

In the past few years, online convex optimization (OCO) has received notable attention in the control literature thanks to its flexible real-time nature and powerful performance guarantees. In this article, we propose new step-size rules and OCO algorithms that simultaneously exploit gradient predictions, function predictions and dynamics, features particularly pertinent to control applications. The proposed algorithms enjoy static and dynamic regret bounds in terms of the dynamics of the reference action sequence, gradient prediction error, and function prediction error, which are generalizations of known regularity measures from the literature. We present results for both convex and strongly convex costs. We validate the performance of the proposed algorithms in a trajectory tracking case study, as well as portfolio optimization using real-world datasets.

@en

In the Netherlands, online groceries are becoming increasingly popular, as are the challenges grocery companies face in meeting customers' rising demand for smaller and cheaper time slots while maintaining thin profit margins due to a highly competitive market. Customer choice modelling will be used to identify customers' behaviour and control the trade-offs between customer attractiveness and profitability with demand management. As there are parametric and non-parametric models, including Machine Learning, to identify the behaviour, they will be compared to define which model represents customer behaviour best. Based on the F1-score, it is evaluated that the Multinomial logit (MNL) and Neural Network (NN) models outperform the created Benchmark model, derived from assumptions from the data, to ensure added value in predicting performance. Since it is found that the advanced predictive models can provide a better understanding of customer behaviour and identify critical customer characteristics, it will be tested whether these models enhance the simulation closer to reality or produce similar results to the Benchmark model, and if so, whether they can be used to optimise the offer strategy. In essence, does the learned behaviour impact the routes and number of offered slots, and can it be utilised to optimise the provided set of offers? A simulation tool is conducted to determine this, which uses real-selected customer time slots to assess any differences from reality and collect key performance indicators (KPIs) for evaluation. The results of the simulations show that incorporating advanced choice models in the simulations adds value and brings the outcomes closer to reality. As a result, a first attempt is made with two test scenarios to determine whether the advanced choice models can be used to optimise the offer strategy. Promising outcomes are found by analysing the results. However, further research is needed to assess the exact impact of the strategies. In future research, it is recommended to determine the influence of using more historical data and data from other companies in combination with other choice models. Also, it is recommended to consider different strategies for optimising the offer set based on incentives or other features.

Optimizing delivery routes is a well-researched topic, however, most of the classical approaches do not incorporate preferences of drivers, as those approaches focus on minimizing the time or distance of the routes. As a result, the actual driven route of an experienced driver often deviates from the proposed route since the drivers have tacit knowledge about the real-life conditions of the road network. Amazon proposed a challenge to learn a delivery route planning strategy from historically driven routes and thus incorporate this tacit knowledge.
In this thesis, we will tackle the challenge using data-driven inverse optimization to learn the zone sequencing patterns of drivers. The zone sequences of expert drivers are assumed to be the solutions to a traveling salesman problem (TSP) in which the weights represent the preference of a driver to use a certain edge. The values of the weights will be learned through inverse optimization. Our final approach achieves a score that ranks 4th out of the 48 models that qualified for the final round of the challenge.

Logistics and mobility services play a major role in our society, and efficient routing is a crucial part of this. However, even though routing problems have been widely researched, the solutions provided by algorithms do not always match drivers' expectations. Routing costs used by these algorithms are often based on one or a few parameters, but in real-world operations, many factors and sometimes hard-to-define aspects are responsible for this. Drivers can consider these different aspects and some studies found that experienced drivers often plan better delivery routes than the optimization tools. In this research, we focus on using expert decision data as examples for learning the costs of routing and train a policy that can make decisions more in line with the expectations of the expert. We formulate state-action representations for the TSP and CVRP, which we use to interpret these routing problems as inverse optimization and multiclass classification problems. Additionally, we propose multiple policy training approaches as well as state feature vector transformations that can be used based on the characteristics of the routing problems. These different training configurations are utilized to train different existing algorithms with training data sets consisting of example state-action pairs. The performance of the trained models is compared to each other and the optimal solution. The optimal solution acted as expert example and was used to create the training data. We demonstrate that both inverse optimization and multiclass classification algorithms are able to imitate expert decision-making for new problem instances from example data. However, we also show a large variation in performance depending on the problem, state features, algorithm formulations and training configuration.

Inverse Learning is implemented in order to learn a control/decision policy (in the integer space) from an Expert Agent. The Learner Agent assumes that the Expert is acting minimizing an unknown cost function and tries to approximate it, through its own parametrized version of it. Learning can be performed in two different ways: offline (exploiting a training set containing Expert data) and online, in which the Learner Agent is directly controlling the system while learning the policy, exploiting corrective advice from the Expert. We propose three different learning algorithms and draw a comparison between them, as well as assess their performance against the Expert. We use our Agent for two different applications: control of a dynamical system (1) and classic ML classification/regression tasks (2). For application (1), our case study is the Heavy Shell Oil Fractionator system, with an MPC Expert Agent. For application (2), we train and test our Agent on several real data-sets available online, with tasks such as medical diagnosis, social media comment volume prediction, fault detection/isolation and multi-class classification.