Learning terminal value functions for Model Predictive Control can provide long-term information without the computational cost of long prediction horizons. This thesis primarily investigates efficient learning methods for approximating infinite horizon terminal value functions due to their desirable theoretical properties, with a focus on model-free Temporal-Difference (TD) learning. Fitted Value Iteration (FVI) is used as a model-based baseline, while TD learning is investigated both for fine-tuning FVI-initialised value functions through the proposed two-stage learning (2SL) approach and for learning from scratch.

The proposed learning methods are evaluated on a pendulum, a two-joint manipulator, and a KUKA iiwa14 in simulation. The proposed 2SL approach achieves similar performance as TD learning from scratch, with both requiring substantially fewer interactions than FVI, including up to a 70× reduction for the iiwa14. These results demonstrate that model-free TD learning can efficiently learn useful terminal value functions without the computational cost of FVI. Additionally, the comparable performance of TD learning from scratch is attributed to the guided exploration provided by the hybrid MPC controller during data collection.

As a secondary investigation, the thesis examines the interplay between the learned terminal value function and constrained MPC. First, we investigate how the long-term information provided by the learned value function enables significantly shorter MPC prediction horizons while maintaining long-horizon performance, reducing computational cost and improving settling times. Second, we examine the respective roles of the learned value function and MPC in constraint-aware decision making. The learned value function provides long-term guidance but does not explicitly encode hard constraints, while MPC retains responsibility for enforcing these constraints during online optimisation. This demonstrates complementary and non-overlapping roles between the learned value function and MPC. The resulting hybrid controller outperformed MPC with prediction horizons up to 10 times longer, while requiring lower computational cost, and achieved up to 99.9% goal-reaching success in randomized environments.

Finally, the approach was demonstrated on a physical KUKA iiwa14, showing its applicability to real-time robotic control for reaching and tracking tasks while maintaining safety under conditions outside the training distribution. To support reproducibility and future research, the developed software framework and implementation is released as open-source. ... Read more

To safely and efficiently solve motion planning problems in multi-agent settings, most approaches attempt to solve a joint optimization that explicitly accounts for the responses triggered in other agents. This often results in solutions with an exponential computational complexity, making these methods intractable for complex scenarios with many agents. While sequential predict-and-plan approaches are more scalable, they tend to perform poorly in highly interactive environments. This paper proposes a method to improve the interactive capabilities of sequential predict-and-plan methods in multi-agent navigation problems by introducing predictability as an optimization objective. We interpret predictability through the use of general prediction models, by allowing agents to predict themselves and estimate how they align with these external predictions. We formally introduce this behavior through the free-energy of the system, which reduces (under appropriate bounds) to the Kullback-Leibler divergence between plan and prediction, and use this as a penalty for unpredictable trajectories. The proposed interpretation of predictability allows agents to more robustly leverage prediction models, and fosters a ‘soft social convention' that accelerates agreement on coordination strategies without the need of explicit high level control or communication. We show how this predictability-aware planning leads to lower-cost trajectories and reduces planning effort in a set of multi-robot problems, including autonomous driving experiments with human driver data, where we show that the benefits of considering predictability apply even when only the ego-agent uses this strategy. ... Read more