Aircraft inspections after unexpected incidents, like lightning strikes, currently require a timeconsuming and costly inspection process, due to the small size of the lightning strike damages. Mainblades Inspections is working on an automated, drone-based solution, that scans the aircraft hull with a high-resolution camera. The objective of this project is to assess the feasibility of using a deep Convolutional Neural Network (CNN) for (semi-)automated damage detection, with the goal of achieving a high recall (low False Negative Rate (FNR)) on the small damages. The problem is framed as a classification problem on limited and imbalanced data. However, it is not pre-defined if single-label or multi-label classification should be used, and both approaches are investigated. The main contribution of this work is to show experimentally how common deep CNN architectures and Deep Learning practices can be used to train classifiers that recognize damages in a specialized domain, with application-specific metrics. We present methods for synthesizing, pre-processing and re-sampling of the necessary dataset. It is shown that pre-trained, parameter-efficient CNN architectures that implement skip-connections, complemented by global max-pooling before the final layer, are well suited for that dataset. The Xception architecture has been chosen as backbone for the classifier due to its high recall and fast convergence. To mitigate the detrimental influence of imbalanced training data, training data re-sampling that equalizes the class distribution is implemented. It has a positive effect recall, especially when applied to multi-label classification. When using re-sampling and data augmentation, the performance of multi-label and single-label classification can be brought to the same level. However, the best achieved FNR is 5.4%, with a softmax classifier, combining all regularization methods. Finally, we investigated how regularization can be used to increase generalization capability with limited training data. Data augmentation is the most effective regularization method, even though its full potential has not been explored yet. Dropout benefits single-label classification but not multi-label classification. L2-regularization has a moderate positive effect on both. Naively combining the regularization techniques without an exhaustive grid search or automated search on average does not yield any additional gains and shows the limit of manual hyper-parameter tuning.

Online Reinforcement Learning is a possible solution for adaptive nonlinear flight control. In this research an Adaptive Critic Design (ACD) based on Dual Heuristic Dynamic Programming (DHP) is developed and implemented on a simulated Cessna Citation 550 aircraft. Using an online identified system model approximation, the method is independent of prior model knowledge. The agent consists of two Artificial Neural Networks (ANNs) which form the Adaptive Critic Design and is supplemented with a Recursive Least Squares (RLS) online model estimation. The implemented agent is demonstrated to learn a near optimal control policy for different operating points, which is capable of tracking pitch and roll rate while actively minimizing the sideslip angle in a faster than real-time simulation. Providing limited model knowledge is shown to increase the learning, performance and robustness of the controller.

To operate in open world environments a symbolic Artificial Intelligence (AI) to be able to adapt and incorporate new objects and relations in its Knowledge Base (KB). Symbolic AI use the objects and relations in their KB to navigate the world and create plans. These KB are filled with knowledge in advance, so they can not add new knowledge when encountering novel situations. This thesis presents Heuristics-Based Causal Discovery (HBCD), a method that identifies and labels causal relations and transforms those causal relations into logic statements, which can be inserted into a KB autonomously. HBCD can operate in a partially-observable environment by performing actions and observing the effects of its actions. The actions are chosen by heuristics, which are modelled after human strategies for causal discovery. The discovered causal relations are tested for the properties necessity and sufficiency. These properties provide information on the completeness of the result, whether there are any missing causal relations. HBCD uses this information to adjust its search space by moving variables in and out of it during the search. If there are no more missing causal relations HBCD stops the search. The method was tested in two simulated environments and the results are promising.

Interactive imitation learning refers to learning methods where a human teacher interacts with an agent during the learning process providing feedback to improve its behaviour. This type of learning may be preferable with respect to reinforcement learning techniques when dealing with real-world problems. This fact is especially true in the case of robotic applications where reinforcement learning may be unfeasible as there are long training times and reward functions can be hard to shape/compute. The present thesis focuses on interactive learning with corrective feedback and, in particular, in the framework Deep Corrective Advice Communicated by Humans (D-COACH), which has successfully shown to be advantageous in terms of training time and data efficiency. D-COACH, a supervised learning method whose policy is represented by an artificial neural network, incorporates a replay buffer where samples of states and corresponding labels gathered by the agent's policy from human feedback are stored and replayed. However, this causes conflicts between the data in the buffer because samples collected by older versions of the policy may be contradictory and could deteriorate the performance of the current policy. In order to reduce this issue, the current implementation of D-COACH uses a first-in-first-out buffer with limited size, as the older the sample is, the more likely it is to deteriorate the performance of the learner. Nonetheless, this limitation propitiates catastrophic forgetting, an inherent tendency of neural networks to forget what they have already learnt, and that can be mitigated by replaying information gathered during all the stages of the problem. Therefore, D-COACH suffers from a trade-off between reducing conflicting data and avoiding catastrophic forgetting. The fact that D-COACH limits the size of its buffer automatically restricts the types of problems that it can solve, given that, if the problem is too complex (i.e. it requires large amounts of data), it simply will not be able to remember everything. If we want to utilise a buffer to train data intensive tasks with corrective feedback, a new method is needed to solve the problem of using information gathered by older versions of the policy. We propose an improved version of D-COACH, which we call Batch Deep COACH (BD-COACH, pronounced “be the coach”). BD-COACH incorporates a human model module that learns the feedback from the teacher and that can be employed to make corrections gathered by older versions of the policy still useful for batch updating the current version of the policy. To compare the performance of BD-COACH with respect to D-COACH, three simulated experiments were done using the open-source Meta-World benchmark, which is based on MuJoCo and OpenAI gym. Moreover, to validate the proposed method in a real setup, two planar manipulation tasks were solved using a seven degrees of freedom KUKA robot arm. Furthermore, we present an analysis between on-policy and off-policy methods both in the fields of reinforcement learning and in imitation learning. We believe there is an interesting simile between this classification and the problem of correctly implementing a replay buffer when learning from corrective feedback.

Introduction - Grasping unknown objects is an important ability for robots in logistic environments. While humans have an excellent understanding of how to grasp objects because of their visual perception and understanding of the 3D world, robotic grasping is still a challenge. Due to the fast-growing development of deep learning methods, it is now possible to train deep neural networks on this grasp task. Objective - This thesis proposes a bin-picking pipeline that uses deep learning to take care of the perception and estimation task. The pipeline can predict grasps for known and unknown objects with a two-fingered pinch-gripper in real-world environments in a single object and multi-object scenes. Method - A grasp annotation tool has been developed to generate a wide variety of grasps in the training data that are antipodal and collision-free. Together with annotated objects, the generated grasps are used to train Grasp-RCNN. The developed Grasp-RCNN combines an object- and a grasp-detection network to predict objects masks and grasps, and a decision algorithm that picks the best-estimated grasp based on a grasp score. Results - Robotic experiments demonstrate that the proposed method allows a robot gripper to grasp both known and unknown objects in single-object and multi-object scenes with a total success-rate of 89.7% and 81.0% with average process-times of 616 ms and 739 ms per scene respectively. In a bin-picking scene a success-rate of 87.5% with a process-time of 1235 ms is achieved. Conclusion - These results indicate that the proposed Grasp-RCNN is able to grasp known and unknown objects with an accuracy that is comparable to the state-of-the-art. For production purposes, the speed of the network still can be improved.

Automated vehicles are conventional vehicles equipped with advanced sensors, controller and actuators. They achieve intelligent information exchange with the environment through the onboard sensing and cooperative system. vehicles are possible to have situation awareness and automatically analyze the safety and dangerous state of journeys. Finally vehicles can reach destinations following drivers' willing. The ongoing research on intelligent vehicles is mainly about improving the safety, comfort, efficiency and provide an excellent human-car interface. As a self-organizing system, the traffic system is quite complicated. There are many disturbance factors to lead to various traffic problems. One of the daily occurring problems is congestion on the motorway. In order to reduce congestion, Rijkswaterstaat applies various dynamic traffic management (DTM) measures to guide the traffic. It works well nowadays in conventional traffic. However, automated vehicles entered the market recently and will start to play an essential role in future traffic. The automated vehicles' reaction to DTM measures may be different from conventional vehicles while the traffic problems still exist. Therefore, it is necessary to research the effectiveness of current Dutch traffic management in automated vehicles. This thesis aims to investigate the effectiveness of current Dutch DTM measures with driver assistant and partially automated vehicles. Due to the time limitation, only the ramp metering measure will be researched through a simulation study. Therefore the main research question is 'How partial automated driving influences the performance of current Dutch dynamic traffic management system and how can this be evaluated via simulation?'. Three methods are applied, including literature review, simulation and statistical analysis. The literature part reviews levels of automation, various longitudinal and lateral vehicle motion models, which are chosen and modified in the simulation. Many ramp metering algorithms are also introduced in the literature review. The ramp metering controller in the simulation follows RWS algorithm. Besides, the motorway demand and the penetration rate of level 1 and 2 vehicles are two input of the simulation. From the simulation results, it is concluded that the level 2 automation consisting of Adaptive Cruise Control (ACC) and Lane Change Assistance (LCA) system brings a negative impact on the motorway capacity. The ramp metering measure remains efficient if the penetration rate of level 2 vehicles is low. However, when the capacity reduces to the critical flow set up in the ramp metering controller, Ramp metering loses its efficiency. The parameters in the ramp metering controller therefore, require an update. For further research, it is recommended to simulate the same scenarios with different ramp metering algorithms. Since the functions of the algorithms are different, there might be other robust control algorithms for automated vehicles. Besides, another limitation of this thesis is that the automation system in level 2 vehicles is defined as Adaptive Cruise Control (ACC) plus Lane Change Assistance (LCA) system. Other partial automation systems may have a different effect on the performance of ramp metering. This thesis can be expanded by research the ramp metering performance under various types of partial automation systems.

Machine learning can be effectively applied in control loops to robustly make optimal control decisions. There is increasing interest in using spiking neural networks (SNNs) as the apparatus for machine learning in control engineering, because SNNs can potentially offer high energy efficiency and new SNN-enabling neuromorphic hardwares are being rapidly developed. A defining character of control problems is that environmental reactions and delayed rewards must be considered. While reinforcement learning (RL) provides the fundamental mechanisms to address such problems, realizing these mechanisms in SNN learning has been underexplored. Previously, schemes of spike timing dependent plasticity (STDP) learning modulated by factors of temporal difference (TD-STDP) or reward (R-STDP) have been proposed for RL with SNN. Here we designed and implemented an SNN controller to explore and compare these two schemes by considering Cart-Pole balancing as a representative example. While the TD-based learning rules are very general ones, the resulted model exhibits rather slow convergence, producing noisy and imperfect results even after prolonged training. We show that by integrating the understanding of the dynamics of the environment into the reward function of R-STDP, a robust SNN-based controller can be learnt much more efficiently than by TD-STDP. The work of this master thesis project has also been published as a paper in Electronics, Vol. 12.

Learning capabilities are a key requisite for an autonomous agent operating in dynamically changing and complex environments, where pre-programming is not anymore possible. Furthermore, it is essential to guarantee that the learning agent will act safely by considering its stability properties. In this thesis, novel conditions are proposed, aiming to examine stability of the learned dynamics for two important model classes; namely Rectified Linear Unit (ReLU) Deep Neural Networks (DNNs) and Locally Weighted Learning (LWL). For the former method, a theoretical and computational framework is developed by establishing an equivalence between ReLU DNN models and Piecewise Affine (PWA) systems. This allows to leverage well-known tools of PWA system analysis, and consequently compute, characterize equilibria and determine their region of attraction for ReLU DNNs. Due to their increased complexity, a structured search for appropriate stability conditions was performed for LWL methods until the optimal trade-off between conservativeness and computational efficiency was obtained. These stability conditions are given as Linear Matrix Inequality (LMI) problems and they consist the first stability results in literature for these two model classes. Their efficacy is assessed in numerical and real-world dynamical systems and it is shown that the proposed LMIs are not unreasonably conservative, as they can evaluate accurately the stability properties of these two representations. Finally, this work demonstrates how to formulate appropriate stability conditions for learning methods in a principled manner.

Reinforcement Learning (RL) is a learning paradigm where an agent learns a task by trial and error. The agent needs to explore its environment and by simultaneously receiving rewards it learns what is appropriate behaviour. Even though it has roots in machine learning, RL is essentially different from other machine learning methods. In contrast to others, RL agent has to generate its own data to learn from. In this thesis, we aim to train an RL agent to fly a quadcopter to track any target position (way-point) in three dimensional space. Where conventional control strategies for quadcopters involve a separate attitude and position controller and most RL solutions focus on one of the two controllers, our goal is to design a low level RL controller capable of computing motor commands directly from sensor input, therefore replacing both attitude and position controller with one RL policy. The policies we develop utilize the algorithm ’Twin Delayed Deep Deterministic Policy Gradient’ (TD3) for learning. TD3 is a variant of the Deep Deterministic Policy Gradient (DDPG) algorithm. The policy for attitude control trained for 3500 episodes 3, around 6.1e5 time steps. The learned policy is able to stabilize the attitude of the quadcopter (in simulation) with a success rate of 94 %. For position control, two policies are generated with two different types of dense reward. The resulting type 1 policy has high fluctuations in motor commands and therefore oscillating attitude and position values. In none of the evaluation trajectories a steady state value is reached. The type 2 produces a working policy after a shorter training time of 1200 episodes, 1.1푒6 time steps. For all tested trajectories, this policy achieves steady state for almost each way-point. This thesis proves that TD3 can be used for low-level quadcopter control, replacing both inner and outer loops of the quadcopter control. Using the dense reward function and applying negative reward on position control only results in a stable policy that can track way-points all throughout the 3D space. Future work requires the twostep parameter estimation to be tested on a real life quadcopter, as well as enrolling the policy onto a real life quadcopter.

One of the most promising ideas in autonomous vehicle control systems is letting the vehicle drive autonomously outside the normal, linear, operating region and letting it "drift". By doing so, the maneuverability of the vehicle could be enhanced. To enable systems that can control this behaviour, estimation of certain vehicle states is needed with high accuracy and high frequency.In this project, a new solution to this problem is proposed by combining a mixed dynamic-kinematic observer with a single camera that produces velocity measurements based on tracking the ground plane. To improve filtering of the camera velocity measurements, the measurement error covariance matrix is updated online based on a model of the camera measurement error. Evaluation of the new methodology was done on data recorded from a 1:10 scale test vehicle and performance was assessed based on ground truth data obtained using a Motion Capture System.In normal driving conditions with correctly identified vehicle parameters, an observer without camera performs better by 25% in terms of RMSE on lateral velocity and sideslip angle estimation. However, the online adaptation of the covariance matrix results in an estimate that is at least 45% more accurate in terms of RMSE than the same observer without online covariance adaptation. Next to that, experiments show that the proposed observer with camera has better robustness to uncertainty in model parameters by almost a factor five in terms of RMSE than the observer without camera.When the grip of the tires is physically lowered and the vehicle is drifting, the proposed observer can track large sideslip angles (>30°), where the state-of-the-art observer without camera is not able. The state-of-the-art observer has an increase in RMSE of 75% on all estimated quantities in comparison to the proposed methodology. These results show that adding a camera to an existing sideslip angle observer greatly enhances robustness of the observer to uncertainty in model parameters and violation of model assumptions. This comes dat the cost of losing some accuracy in normal driving conditions.

Neural networks have achieved great success in many difficult learning tasks like image classification, speech recognition and natural language processing. However, neural architectures are hard to design, which requires lots of knowledge and time of human experts. Therefore, there has been a growing interest in automating the process of designing neural architectures. Though these searched architectures have achieved competitive performance on various tasks, the efficiency of NAS still needs to be improved. Moreover, current neural architecture search approach disregards the dependency between a node and its predecessors and successors. This thesis builds upon BayesNAS which employs the classic Bayesian learning method to search for CNN architectures, and extends it to the problem of neural architecture search for recurrent architectures. Hierarchical sparse priors are used to model the architecture parameters to alleviate the dependency issue. Since the update of posterior variance is based on Laplace approximation, an efficient method to compute the Hessian of recurrent layer is proposed. We can find candidated architectures after training the over-parameterized network for only one epoch. Our experiments on Penn Treebank and WikiText-2 show that competitive architectures can be found in 0.3 GPU days using a single GPU for language modeling task. We find that our algorithm is more efficient than state-of-the-art.

The use of artificial neural networks is becoming ever more ubiquitous as the computational power available to use grows. The widespread implementation of neural networks as controllers in the field of systems and control is however being hindered by the lack of verifiability of these controllers. One type of controller that does not lack verifiability is the correct-by-design controller. The main drawback of correct-by-design controllers is that they inherently produce large data structures in order to store their control rules. In this work, a novel methodology to synthesize correct-by-design neural network controllers is presented in order to alleviate these issues. This methodology combines reinforcement learning techniques and abstraction based correct-by-design control verification techniques in order to synthesize neural network controllers that are correct-by-design. The procedure does so by alternating between a controller training routine and a controller verification routine in a system abstraction framework. This framework ensures that numerical training and verification results in a controller with formal guarantees applicable to real systems. Using the proposed methodology, neural network controllers are synthesized and verified in order to prove that the methodology works. In addition to this, the resulting correct-by-design neural network controllers are compared to conventional correct-by-design controllers in order to judge their performance and data requirements. This comparison also includes alternative structures used to store these different controllers. Based on this comparison, a conclusion is drawn regarding when to use which type of controller. The proposed methodology is implemented into a correct-by-design neural network synthesis framework called COSYNNC. This framework is intended as a basis for further research into correct-by-design neural network control and is publicly available.