Y. Li | TU Delft Repository

Uncertainty-Driven Distributional Reinforcement Learning for Flight Control

Conference paper (2025) - M. Homola, Y. Li, E. van Kampen

In the rapidly evolving aviation sector, the quest for safer and more efficient flight operations has historically relied on traditional Automatic Flight Control Systems (AFCS) based on high-fidelity models. However, such models not only incur high development costs but also struggle to adapt to new, complex aircraft designs and unexpected operational conditions. As an alternative, deep Reinforcement Learning (RL) has emerged as a promising solution for model-free, adaptive flight control. Yet, RL-based approaches pose significant challenges in terms of sample efficiency and safety assurance. Addressing these gaps, this paper introduces Returns Uncertainty-Navigated Distributional Soft Actor-Critic (RUN-DSAC). Designed to enhance the learning efficiency, adaptability, and safety of flight control systems, RUN-DSAC leverages the rich uncertainty information inherent in the returns distribution to refine the decision-making process. When applied to the attitude tracking task on a high-fidelity, non-linear fixed-wing aircraft model, RUN-DSAC demonstrates superior performance in learning efficiency, adaptability to varied and unforeseen flight scenarios, and robustness in fault tolerance that outperforms the current state-of-the-art SAC and DSAC algorithms. ...

Stability Analysis for Incremental Adaptive Dynamic Programming with Approximation Errors

Journal article (2024) - Yifei Li, Erik Jan Van Kampen

This paper provides a convergence and stability analysis of the incremental value iteration algorithm under the influence of various errors. Incremental control is firstly used to linearize the continuous-time nonlinear system, recursive least squares (RLS) identification is then introduced to identify the incremental model online. Based on the incremental model, the value iteration algorithm is used to design an optimal adaptive controller, with an analytical optimal control law. Moreover, the convergence of the developed incremental value iteration algorithm is proved. The stability of the controller is analyzed using Lyapunov stability theory. Finally, a flight control simulation verifies the robustness of the controller to various initial conditions, as well as adaptation to actuator faults. ...

Reinforcement Learning-based Intelligent Flight Control for a Fixed-wing Aircraft to Cross an Obstacle Wall

Conference paper (2024) - Yifei Li, Erik Jan Van Kampen

This paper develops an intelligent flight controller for a fixed-wing aircraft model in the longitudinal plane, using a Reinforcement Learning (RL)-based control method, namely Deep Deterministic Policy Gradient (DDPG). The neural net-work controller is fed the values of aircraft position, velocity, pitch angle and pitch rate, and outputs the elevator deflection. Artificial Neural Network (ANN)s are used to approximate the nonlinear state-action value function and the policy function. Simulation results show that the flight controller learns from the experienced data to fly over an obstacle wall with constrained pitch angle. ...

Incremental Generalized Policy Iteration for Adaptive Attitude Tracking Control of a Spacecraft

Conference paper (2023) - Yifei Li, Erik Jan van Kampen

This paper proposes a novel dynamic programming algorithm for nonlinear system optimal control problem, namely Incremental Generalized Policy Iteration (IGPI). The proposed IGPI algorithm combines the advantages of Incremental Control(IC) and Generalized Policy Iteration(GPI). Incremental control can handle the nonlinearity and uncertainty in nonlinear systems without knowing the nonlinear system information, GPI can learn an optimal control law for dynamical systems. Based on the proposed IGPI algorithm, a data-driven adaptive attitude controller is designed for a spacecraft with sloshing liquid fuel. Simulation results demonstrate the effectiveness of the spacecraft attitude controller. ...

Twin-Delayed Deep Deterministic Policy Gradient for altitude control of a flying-wing aircraft with an uncertain aerodynamic model

Conference paper (2023) - W.J.E. Völker, Y. Li, E. van Kampen

Recent research on the Flying V - a flying-wing long-range passenger aircraft - shows that its airframe design is 25% more aerodynamically efficient than a conventional tube-and-wing airframe. The Flying V is therefore a promising contribution towards reduction in climate impact of long-haul flights. However, some design aspects of the Flying V still remain to be investigated, one of which is automatic flight control. Due to the unconventional airframe shape of the Flying V, aerodynamic modelling cannot rely on validated aerodynamic-modelling tools and the accuracy of the aerodynamic model is uncertain. Therefore, this contribution investigates how an automatic flight controller that is robust to aerodynamic-model uncertainty can be developed, by utilising Twin-Delayed Deep Deterministic Policy Gradient (TD3) - a recent deep-reinforcement-learning algorithm. The results show that an offline-trained single-loop altitude controller that is fully based on TD3 can track a given altitude-reference signal and is robust to aerodynamic-model uncertainty of more than 25%. ...

Adaptive Optimal Flight Control for a Fixed-wing Unmanned Aerial Vehicle using Incremental Value Iteration

Conference paper (2023) - Y. Li, E. van Kampen

This paper deals with the design of an adaptive optimal controller for a fixed-wing Unmanned Aerial Vehicle(UAV) using an incremental value iteration algorithm. The incremental model is firstly introduced to linearize a nonlinear system. The recursive least squares(RLS) identification algorithm is then used to identify the incremental model. Based on incremental control, the incremental value iteration algorithm is developed for a nonlinear optimal control problem. Moreover, this algorithm is applied to longitudinal attitude tracking of a fixed-wing unmanned aerial vehicle. Simulation results show that the designed adaptive flight controller is robust to variations in initial value of the angle of attack. ...