The critical challenge for employing autonomous control systems in aircraft is ensuring robustness and safety. This study introduces an intelligent and fault-tolerant controller that merges two Reinforcement Learning (RL) algorithms in a hybrid approach: the Distributional Soft Actor-Critic (DSAC) and the Incremental Dual Heuristic Programming (IDHP). The integration combines the strengths of DSAC in learning a robust control strategy and IDHP in allowing real-time control adaption. Compared to earlier controllers, such as a hybrid using the Soft Actor-Critic (SAC) algorithm and strictly offline DSAC and SAC, our hybrid demonstrates enhanced robustness against changing flight conditions and in the face of sensor noise and bias. During fault tolerance tests, it maintains superior control even when the effectiveness of the aircraft’s ailerons and elevators is compromised. By demonstrating the potential of RL-based controllers to provide robustness and fault tolerance, this research advances the feasibility of safe and autonomous flight control operations.@en

The Flying-V emerges as a unique flying wing type commercial aircraft design, distinguished by its V-shaped configuration. For such unconventional airframes, flight control systems are vital for ensuring safety and enhancing flight performance. Current Flying-V control systems primarily use incremental nonlinear dynamic inversion (INDI), a sensor-based feedback linearization method requiring an onboard control effectiveness model. Although INDI handles model uncertainties, significant mismatches between actual and onboard models caused by damages or faults degrade performance and compromise flight safety. This study proposes an adaptive strategy for incremental nonlinear dynamic inversion, employing an online two-step method to estimate changes in the aircraft’s control effectiveness. Estimates are used to update the onboard control effectiveness model to minimize the mismatch between the actual and onboard representation of control effectiveness.@en

Reinforcement Learning applied to flight control has shown to have several benefits over classical, linear flight controllers, as it eliminates the need for gain scheduling and it could provide fault-tolerance. The application to civil aviation in practice, however, is non-existent as there are multiple safety concerns. This research demonstrates the evaluation of longitudinal Handling Qualities of the Soft Actor-Critic Deep Reinforcement Learning framework with the aim to translate the unpredictable black box of Reinforcement Learning into classical flight control terminology. The framework is applied to a pitch rate command system of a jet aircraft and shows robustness to off-nominal flight conditions, center of gravity shifts and biased sensor noise. Accurate tracking performance is achieved, while adhering to Level 1 longitudinal Handling Qualities for all conditions.@en

In the rapidly evolving aviation sector, the quest for safer and more efficient flight operations has historically relied on traditional Automatic Flight Control Systems (AFCS) based on high-fidelity models. However, such models not only incur high development costs but also struggle to adapt to new, complex aircraft designs and unexpected operational conditions. As an alternative, deep Reinforcement Learning (RL) has emerged as a promising solution for model-free, adaptive flight control. Yet, RL-based approaches pose significant challenges in terms of sample efficiency and safety assurance. Addressing these gaps, this paper introduces Returns Uncertainty-Navigated Distributional Soft Actor-Critic (RUN-DSAC). Designed to enhance the learning efficiency, adaptability, and safety of flight control systems, RUN-DSAC leverages the rich uncertainty information inherent in the returns distribution to refine the decision-making process. When applied to the attitude tracking task on a high-fidelity, non-linear fixed-wing aircraft model, RUN-DSAC demonstrates superior performance in learning efficiency, adaptability to varied and unforeseen flight scenarios, and robustness in fault tolerance that outperforms the current state-of-the-art SAC and DSAC algorithms. @en

The Flying-V is a novel flying wing aircraft design that aims to improve aerodynamic efficiency and fuel consumption. In recent years, there has been an increased effort to evaluate the handling qualities (HQs) of the Flying-V, and to develop new flight control strategies. One of the latest flight control designs for the Flying-V uses Incremental Nonlinear Dynamic Inversion (INDI) with a Flight Envelope Protection system. The purpose of this research is to validate this FCS design and its conclusions on handling qualities by conducting piloted experiments. To facilitate a sound comparison, the aerodynamic model, flight control structure, and airframe characteristics are taken as is from the existing study. For the experiment, numerous maneuvers are designed and carried out. The experiment is performed on the SIMONA Research Simulator of the Delft University of Technology, with three real-life test pilots as participants. The results show a significant correlation with offline findings and demonstrate the improvement in HQs provided by the flight control system.@en

Commercial applications of flying wing aircraft, as the Flying-V here considered, can contribute to reducing carbon and nitrogen emissions produced by the aviation sector. However, because of the lack of a tail, all flying wing aircraft have reduced controllability. For this reason, the placement and sizing of the control surfaces along the wing is a non-trivial problem. The paper focuses on solving this problem using offline handling quality simulations based on certification requirements. In different flight conditions, the aircraft must be able to perform a certain set of maneuvers as defined by the certifying authorities. First, offline simulations calculate the minimum control authority required from the elevator, aileron, and rudder to perform each maneuver. Then, based on the global minimum for all maneuvers, the control surfaces are sized and placed along the wings. The aerodynamic model employed uses a combination of Reynolds-averaged Navier-Stokes (RANS) and vortex lattice method (VLM) simulations. The control authority of the control surfaces is estimated with VLM and VLM calibrated with RANS simulations, showing significant differences between the two.@en

MegAWES is a reference design and simulation framework for ground-generation, fixed-wing airborne wind energy systems with a nominal power output of 3 MW. The winch size of MegAWES is based on a smaller system and needs to be scaled up because the current size leads to unrealistically fast dynamics, which require saturation. However, there is no available method to select an appropriate size for the winch. Additionally, while it has been hypothesized that the size of the winch has a significant effect on the dynamics of the overall system for ground-generation concepts, this effect has not been quantified. In this work, we first analyze the effects of the winch size on the system dynamics using a linearized model. Second, we present a method to find the upper bound for the size of the winch based on a selected maximum tether force overshoot during nominal operation. Third, we apply this method to find an upper bound for the winch size for the MegAWES reference design. Using the nonlinear MegAWES simulation framework, we validated this upper bound. At the upper bound, the system accurately tracked the reference tether force without overshoot and when exceeding our upper bound, the tether force response was oscillatory and overshot its ideal value.

@en

Incremental Nonlinear Dynamic Inversion (INDI) has received substantial interest in the recent years as a nonlinear flight control law design methodology that features inherent robustness against bare airframe aerodynamic variations. However, systematic studies into the robust design benefits of INDI-based control over the classical divide-and-conquer philosophy have been scarce. To bridge this gap, this paper compares the setup of hybrid INDI with a standard industry benchmark that is based on two-degree-of-freedom gain-scheduled proportional-integral-derivative control. This is done on an architectural basis and in terms of achievable robust stability and performance levels with respect to a common set of design requirements. To this end, a non-smooth, multi-objective H_∞-synthesis algorithm is used that incorporates mixed parametric and dynamic uncertainties in the design objective and constraints. It is shown that close similarities exist between hybrid INDI design and gain-scheduled PID control, which leads to virtually equivalent robustness and performance outcomes in both linear time-invariant and linear time-varying contexts. It is therefore concluded that the main benefit of the hybrid INDI does not lie in improved robustness properties per se, but in the opportunity to perform modular robust design in an implicit model-following context. Specifically, this implies that the areas of flying qualities, robustness, and nonlinear implementation are directly visible and accessible in the control law structure.

@en

Recent advancements in fault-tolerant flight control have involved model-free offline and online Reinforcement Learning (RL) algorithms in order to provide robust and adaptive control to autonomous systems. Inspired by recent work on Incremental Dual Heuristic Programming (IDHP) and Soft Actor-Critic (SAC), this research proposes a hybrid SAC-IDHP framework aiming to combine adaptive online learning from IDHP with the high complexity generalization power of SAC in controlling a fully coupled system. The hybrid framework is implemented into the inner loop of a cascaded altitude controller for a high-fidelity, six-degree-of-freedom model of the Cessna Citation II PH-LAB research aircraft. Compared to SAC-only, the SAC-IDHP hybrid demonstrates an improvement in tracking performance of 0.74%, 5.46% and 0.82% in nMAE for nominal case, longitudinal and lateral failure cases respectively. Random online policy initialization is eliminated due to identity initialization of the hybrid policy, resulting in an argument for increased safety. Additionally, robustness to biased sensor noise, initial flight condition and random critic initialization is demonstrated.@en

This paper develops an intelligent flight controller for a fixed-wing aircraft model in the longitudinal plane, using a Reinforcement Learning (RL)-based control method, namely Deep Deterministic Policy Gradient (DDPG). The neural net-work controller is fed the values of aircraft position, velocity, pitch angle and pitch rate, and outputs the elevator deflection. Artificial Neural Network (ANN)s are used to approximate the nonlinear state-action value function and the policy function. Simulation results show that the flight controller learns from the experienced data to fly over an obstacle wall with constrained pitch angle.

@en

To reduce the impact of aviation on the environment, technological innovations, such as the Flying-V are required. The Flying-V is a proposed commercial flying wing, which uses the Airbus A350-900 as reference aircraft. In this work, a Flight Control system for the Flying-V is proposed with a longitudinal C ^* control law, and a Rate Control Attitude Hold roll control law. This Flight Control System also includes a Flight Envelope Protection law to prevent reaching angles of attack higher than 30 degrees, where the Flying-V becomes statically unstable. The FEP also prevents the Flying-V from reaching load factors above 2.5 and limits the roll angle. The control laws are tuned to be within level 1 handling qualities in the selected approach and cruise conditions, with the presence of sensor dynamics, and a digital control system. Robustness for aerodynamic uncertainties is also shown. Finally, it is shown that the FEP is able to prevent the angle of attack from becoming too large.

@en

Recent research in artificial intelligence potentially provides solutions to the challenging problem of fault-tolerant and robust flight control. This paper proposes a novel Safety-Informed Evolutionary Reinforcement Learning algorithm (SERL), which combines Deep Reinforcement Learning (DRL) and neuroevolution to optimize a population of nonlinear control policies. Using SERL, the work has trained agents to provide attitude tracking on a high-fidelity nonlinear fixed-wing aircraft model. Compared to a state-of-the-art DRL solution, SERL achieves better tracking performance in nine out of ten cases, remaining robust against faults and changes in flight conditions, while providing smoother action signals.

@en

This paper provides a convergence and stability analysis of the incremental value iteration algorithm under the influence of various errors. Incremental control is firstly used to linearize the continuous-time nonlinear system, recursive least squares (RLS) identification is then introduced to identify the incremental model online. Based on the incremental model, the value iteration algorithm is used to design an optimal adaptive controller, with an analytical optimal control law. Moreover, the convergence of the developed incremental value iteration algorithm is proved. The stability of the controller is analyzed using Lyapunov stability theory. Finally, a flight control simulation verifies the robustness of the controller to various initial conditions, as well as adaptation to actuator faults.

@en

This paper investigates the performance of an autonomous navigation system to navigate a spacecraft in the proximity of a binary asteroid system using optical and laser ranging measurements. The knowledge about the binary asteroid is limited to its orbital parameters and ellipsoid shape models. The accelerometer bias random walk is included in the estimation process. Over a four-hour landing maneuver starting from 6770 m altitude and ending at 550 m, the mean position estimation uncertainty is 41.6 m (3). It is shown that the navigation accuracy is sensitive to the Sun phase angle, the irregularity of the asteroid shape, and the goodness of fit of the ellipsoid shape model. The paper demonstrates that the navigation system is robust to large errors in the initialization of the extended Kalman filter state. The impact of image distortion and two types of image noise on the navigation performance are investigated.

@en

Recent research in artificial intelligence potentially provides solutions to the challenging problem of fault-tolerant and robust flight control. The current work proposes a novel Safety-informed Evolutionary Reinforcement Learning (SERL) algorithm, which combines Deep Reinforcement Learning (DRL) and neuro-evolution to optimize a population of non-linear control policies. Using SERL, the work has trained agents to provide attitude tracking on a high-fidelity non-linear fixed-wing aircraft model. Compared to a state-of-the-art DRL solution, SERL achieves better tracking performance in nine out of ten cases, remaining robust against faults and changes in flight conditions, while providing smoother actions. @en

Unforeseen failures during flight can lead to Loss of Control In-Flight, a significant cause of fatal aircraft accidents worldwide. Current offline synthesized flight control methods have limited capability to recover from failures, due to their limited adaptability. Incremental Approximate Dynamic Programming (iADP) control is a model-agnostic online adaptive control method, which integrates an online identified locally linearized incremental model, with a Reinforcement Learning (RL) based optimization technique to minimize an infinite horizon quadratic cost-to-go. A key challenge for adopting these self-learning flight control methods is validation through flight testing. This paper presents the iADP flight control law design for CS-25 class aircraft to achieve rate control. It outlines the controller evaluation strategy, controller integration, verification & validation procedures, and a discussion on flight test results. To the author’s understanding, this flight test marks the world’s first demonstration of an online RL based automatic flight control system for this aircraft category, demonstrating real-time learning and adaptation capabilities to aircraft configurations.@en

Airborne wind energy is an emerging technology that uses tethered flying devices to capture stronger and more steady winds at higher altitudes. Compared to smaller systems, megawatt-scale systems are substantially affected by gravity during flight operation, resulting in power fluctuations. MegAWES, a 3 MW reference model, experiences power fluctuations between -5.8 MW and +20.5 MW every 12.5 seconds during the traction phase when using its baseline controller at a wind speed of 22 m/s. The baseline controller does not have a power limit, leading to high peak power, and aims to keep the tether force constant, causing it to consume power when the kite is flying upwards. In this paper, we implement an optimal torque controller in the MegAWES framework and show that this eliminates the power consumption during the traction phase. Furthermore, we propose a kite tether force controller that allows setting a power limit when combined with the 2-phase reeling strategy, which decreases the peak power. Our new architecture reduces the power output range by 75% to between +3.7 MW and +9.4 MW in strong wind conditions.
@en

Enabling the capability of assessing risk and making risk-aware decisions is essential to applying reinforcement learning to safety-critical robots like drones. In this paper, we investigate a specific case where a nano quadcopter robot learns to navigate an apriori-unknown cluttered environment under partial observability. We present a distributional reinforcement learning framework to generate adaptive risk-tendency policies. Specifically, we propose to use lower tail conditional variance of the learnt return distribution as intrinsic uncertainty estimation, and use exponentially weighted average forecasting (EWAF) to adapt the risk-tendency in accordance with the estimated uncertainty. In simulation and real-world empirical results, we show that (1) the most effective risk-tendency varies across states, (2) the agent with adaptive risk-tendency achieves superior performance compared to risk-neutral policy or risk-averse policy baselines. Code and video can be found in this repository: https://github.com/tudelft/risk-sensitive-rl.git

@en

The functional architecture of a flight control system (FCS) is driven by multiple objectives related to the aircraft’s operational mission and in-service performance targets. Control allocation (CA) is a common method to ensure adequate use of the available effector architecture once the number of control effectors linked to the FCS increases and the control problem becomes overdetermined. A primary CA design objective is to ensure that high-level motion control demands are met. However, the additional degrees of freedom offered by the control effector suite can also be exploited to perform secondary control tasks. In this light, this article focuses on the Incremental Control Allocation (INCA) framework in the context of in-flight optimization of arbitrary secondary flight control design objectives. An extension to the existing INCA concept is formulated that isolates this secondary control task from generating primary control demands. Moreover, an alternative, but closely related design method based on optimal control principles is proposed that extents the role of the control allocator to active control of the system dynamics. Design examples are demonstrated for least-squares minimization of total drag and control activity. These are analyzed in linear and nonlinear simulation scenarios based on an open-source General Dynamics F-16 simulation model.

@en