From Data to Prediction: Vision-Based UAV Fault Detection and Diagnosis

Abstract

Despite the camera being ubiquitous to unmanned aerial vehicles (UAVs), it has not been used for fault detection and diagnosis (FDD) due to the nonexistence of UAV multi-sensor datasets that include actuator failure scenarios. This thesis proposes a knowledge-based FDD framework based on a lightweight Long-Short Term Memory network that fuses camera and Inertial Measurement Unit (IMU) information. Camera data is pre-processed by extracting features from its optical flow. Short-Time Fourier Transform is applied on the IMU data for obtaining its time-frequency information. For training and assessing the proposed framework, UUFOSim was developed: an Unreal Engine-based simulator built on AirSim that allows the collection of high-fidelity photo-realistic camera and sensor information with the possibility of injecting in-flight actuator failures. To simulate blade damage, a Blade Element Theory (BET) model is introduced as plug-in which enables any level of blade damage simulation without costly experimental data. The BET model was validated with static test stand wrench measurements obtained at 3 levels of blade damage (0%, 10%, 25%) from a dedicated wind tunnel experimental campaign in the Open Jet Facility of TU Delft with velocities up to 12 m/s. In the presence of blade damage, at high propeller rotational speeds the BET model shows a relative error between 5% and 24%. At low propeller rotational speeds, the relative error oscillates between 15% and 75%. Results of the FDD framework trained on 5,000 simulated flights demonstrate the added value of the camera and the complementary nature of both sensors with failure detection and diagnosis accuracies of 99.98% and 98.86%, respectively.