Hybrid Soft Actor-Critic and Incremental Dual Heuristic Programming Reinforcement Learning for Fault-Tolerant Flight Control

Hybrid Soft Actor-Critic and Incremental Dual Heuristic Programming Reinforcement Learning for Fault-Tolerant Flight Control

Teirlinck, C. (Student TU Delft)
van Kampen, E. (TU Delft Control & Simulation)

2024

Recent advancements in fault-tolerant flight control have involved model-free offline and online Reinforcement Learning (RL) algorithms in order to provide robust and adaptive control to autonomous systems. Inspired by recent work on Incremental Dual Heuristic Programming (IDHP) and Soft Actor-Critic (SAC), this research proposes a hybrid SAC-IDHP framework aiming to combine adaptive online learning from IDHP with the high complexity generalization power of SAC in controlling a fully coupled system. The hybrid framework is implemented into the inner loop of a cascaded altitude controller for a high-fidelity, six-degree-of-freedom model of the Cessna Citation II PH-LAB research aircraft. Compared to SAC-only, the SAC-IDHP hybrid demonstrates an improvement in tracking performance of 0.74%, 5.46% and 0.82% in nMAE for nominal case, longitudinal and lateral failure cases respectively. Random online policy initialization is eliminated due to identity initialization of the hybrid policy, resulting in an argument for increased safety. Additionally, robustness to biased sensor noise, initial flight condition and random critic initialization is demonstrated.

http://resolver.tudelft.nl/uuid:02616c7b-ffaa-45f1-9b32-875a6a9e3061

https://doi.org/10.2514/6.2024-2406

American Institute of Aeronautics and Astronautics Inc. (AIAA)

978-1-62410-711-5

Proceedings of the AIAA SCITECH 2024 Forum

AIAA SCITECH 2024 Forum, 2024-01-08 → 2024-01-12, Orlando, United States

Institutional Repository

conference paper

© 2024 C. Teirlinck, E. van Kampen