Insects have — over millions of years of evolution — perfected many of the systems that roboticists aim to achieve; they can swiftly and robustly navigate through different environments under various conditions while at the same time being highly energy efficient. To reach this level of performance and efficiency one might want to look at and take inspiration from how these insects achieve their feats. Currently, no dataset exists that allows bio-inspired navigation models to be evaluated over long real- life routes. We present a novel dataset containing omnidirectional event vision, frame-based vision, depth frames, inertial measurement (IMU) readings, and centimeter-accurate GNSS positioning over kilometer long stretches in and around the TUDelft campus. The dataset is used to evaluate familiarity-based insect-inspired neural navigation models on their performance over longer sequences. It demonstrates that current scene familiarity models are not suited for long-ranged navigation, at least not in their current form.

Event cameras and spiking neural networks (SNNs) allow for a highly bio-inspired, low-latency and power efficient implementation of optic flow estimation. Just recently, a hierarchical SNN was proposed in which motion selectivity is learned from raw event data in an unsupervised manner using spike-timing-dependent plasticity (STDP). However, real-life applications of this SNN are currently still limited by the fact that the exact choice of neuron parameters depends on the spatiotemporal properties of the input. Furthermore, tuning the network is a challenging task due to the high degree of coupling between the various parameters. Inspired by neurons in biological brains that modify their intrinsic parameters through a process called intrinsic plasticity, this research proposes update rules which adapt the voltage threshold and maximum synaptic delay during inference. This allows applying the already trained network to a wider range of operating conditions and simplifies the tuning process. Starting with a detailed parameter analysis, primary functions and undesired side effects are assigned to each parameter. The update rules are then designed in such a way as to eliminate these side effects. Unlike existing update rules for the voltage threshold, this work does not attempt to keep the firing activity of output neurons within a specific range, but instead aims to adjust the threshold such that only the correct output maps spike. In particular, the voltage threshold is adapted such that output spikes occur in no more than two maps per retinotopic location. The maximum synaptic delay is adapted such that the resulting apparent pixel velocities of the input match those of the data used during training. A sensitivity analysis is presented which illustrates the effects of newly introduced parameters on the network performance. Furthermore, the adapted network is tested on real event data recorded onboard a drone avoiding obstacles. Due to the difficulties in matching the output of the adapted SNN to the ground truth data, quantitative results are inconclusive. However, qualitative results show a clear improvement in both the density and correctness of optic flow estimates.

Micro Air Vehicles (MAVs) are able to support humans in dangerous operations, such as search and rescue operations at night on unknown terrain. These scenes require a great amount of autonomy from the MAV, as they are often radio and GPS-denied. As MAVs have limited computational resources and energy storage, onboard navigation tasks have to be performed efficient and fast. To address this challenge, this research proposes an approach to visual obstacle detection and avoidance onboard an MAV. The algorithmic approach is based on event-based optic flow, using a monocular event-based camera. This camera captures the apparent motion in the scene, has microsecond latency and very low power consumption, therefore a good fit for onboard navigation tasks. Firstly, a literature study is performed to provide theoretical concepts and foundation for the obstacle avoidance approach. A processing pipeline is designed, based on the use of event-based normal optic flow. This pipeline consists of three sections: course estimation, obstacle detection and obstacle avoidance. A novel course estimation method 'FAITH' is proposed which uses optic flow half-planes along with a fast RANSAC scheme. The object detection method is based on DBSCAN clustering of optic flow vectors, using the time-to-contact and vector location as clustering variables. The performance of these methods is experimentally demonstrated by three experiments: in a simulated environment, offline on real sensor data and online onboard an MAV. As currently no event-based obstacle avoidance datasets are publicly available, a dataset is recorded as supplement to this and future research. Approximately 1350 runs of event-based camera, RADAR, IMU and OptiTrack data are recorded, manually avoiding either a single or two poles using an MAV in the flying arena of the TU Delft. This dataset is used in this research to determine the performance of the course estimation method using real sensor data. The course estimation method is shown to have state-of-the-art accuracy and beyond state-of-the-art computation time on both simulated data and the recorded dataset. The final experiment shows the obstacle detection and avoidance approach integrated onboard an MAV in a real-time obstacle avoidance task. The approach is shown to have a success rate of 80% in a frontal obstacle avoidance task on a low-textured 50-cm wide pole. The contribution of this research is an obstacle detection and avoidance approach using a monocular event-based camera onboard an MAV, along with the novel course estimation algorithm 'FAITH'.

Micro robotic airships offer significant advantages in terms of safety, mobility, and extended flight times. However, their highly restrictive weight constraints pose a major challenge regarding the available computational power to perform the required control tasks. Thus, spiking neural networks (SNNs) are a promising research direction. By mimicking the biological process for transferring information between neurons using spikes or impulses, they allow for low power consumption and asynchronous event-driven processing. In this work, we propose an evolved altitude controller based on a SNN for an airship which relies solely on the sensory feedback provided by an airborne radar sensor. Starting from the design of a a lightweight, low-cost, open-source airship, we also present a low-control-effort SNN architecture, an evolutionary framework for training the network in a simulated environment, and a control scheme for ameliorating the performance of the system in real-world scenarios. The system's performance is evaluated through real-world experiments, demonstrating the advantages of our approach by comparing it with an artificial neural network (ANN) and a linear controller (PID). The results show an accurate tracking of the altitude command while ensuring efficient management of the control effort. The main contributions of this work are presented in the scientific paper, corresponding to Part I of the document. Besides the research on altitude control based on SNNs and their comparison with an ANN and a PID, this thesis includes an in-depth review of the relevant literate on the main topics covered, in Part II. Finally, a detailed explanation of the methodologies used, the conclusions and recommendations for future work are proposed in Part III.

Micro air vehicles (MAVs) are increasingly being considered for aerial tasks such as delivery of goods and surveillance due to their lightweight, compact design and manoeuvrability. To safely and reliably carry out these tasks and navigate to its objective, especially in complex and cluttered environments, the MAV is also required to sense and avoid (S&A) obstacles. Due to the MAVs limitations in weight, power and processing power, vision systems usually prove ideal for sensing the environment, being a cheap, lightweight, power efficient and a rich source of information. They do however require adequate computational resources and most importantly, good visibility. When the environment does not host these conditions, for instance when flying though dust, smoke or fog, other sensors need to be utilised that can provide more robust sensing to ensure safe and reliable operation. Radar sensors are mostly unaffected by atmospheric conditions and have been used extensively in the aerospace industry for this purpose. These sensors were traditionally heavy and power hungry, only applicable on ground or in large craft. However other radar sensors have since come about that are more suited for use in small MAVs. Specifically, lightweight, power efficient and compact frequency modulated continuous wave (FMCW) radars have increasingly been used in advanced driver assistance systems as auxiliary sensors, however there has been little work to integrate them on MAVs. This sensor provides the range, horizontal bearing and radial velocity (Doppler shift) of any objects in the field of view, which can then be used for multi-target tracking (MTT) [38]. The major disadvantage of the sensor is the limited field of view (approximately 80 degrees horizontal) and noisy nature of the sensor, especially in cluttered environments. The challenge is to explore filtering, tracking and avoidance algorithm pipelines to extract meaningful information from the raw data and investigate the sensor’s effectiveness with respect to obstacle avoidance on MAVs. This will include algorithms such as data association, estimation and avoidance, as well as an investigation of neural networks to aid in processing the raw data and provide some filtering. This will be accomplished by integrating the sensor on a MAV and testing and tuning the algorithms both in real life (in the cyberzoo flying arena of the aerospace faculty), and using data gathered as part of an obstacle detection and avoidance dataset that was generated during this project. This will hopefully allow MAVs to operate safer, either using a standalone radar or integrated with other sensors.

In an effort to develop a new relative sensing method for drone swarms, the suitability of event cameras is assessed for propeller detection. Benchmark tests were conducted for different propellers under different lighting and background conditions, varying the observation distance and spinning frequency. The different tests were evaluated on event count, frequency, and clustering, as these are the most characteristic properties of the propeller-generated signal. A propeller detection metric was derived as a fuzzy classifier to assess detectability. It was observed that the sensor employed is limiting the detection range due to low resolution, with a maximum detection range of 75 cm. While at low spinning frequencies it is possible to detect the propeller at such distance, for higher frequences (6000 to 8000 RPMs) the range decreases to 45 cm for the tests with highest blade to background contrast and two-blade propellers. It was observed that lower contrasts reduce the successful detections only to low frequencies, and three-blade propellers become completely indetectable due to the static overlap between the blades. Therefore, it is concluded that, at this stage of the technology, the use case of event cameras for relative sensing is constrained to close distances with high contrast.