JH

J.J. Hagenaars

info

Please Note

8 records found

Safety solutions for autonomous vehicle integration into mixed-mode mobility

Journal article (2026) - G. C.H.E. de Croon, J. Hagenaars, J. Kober, Y. H. Liu, A. Palffy, T. Sun, N. van Nes, W. P. Medendorp, S. M. Bohte, More Authors
Uncertainty fundamentally shapes human perception and decision-making, a factor that is becoming increasingly critical as autonomous vehicles (AVs) begin to share physical and social spaces with humans. This perspective paper synthesizes insights from neuroscience, robotics, and behavioral science represent uncertainty and act under it. We identify a significant disconnect between low-level mathematical concepts of uncertainty and risk, the uncertainty-aware prediction and planning methods used in robotics, and the high-level psychological uncertainty experienced by humans. Here, we conceptualize an integration pathway for mixed-mode mobility: we show that all three treat control under partial information as the same two-stage problem – probabilistic inference of the current and future state, followed by a risk -weighted choice of action – and we make this shared structure explicit. We thus argue that AVs should quantify and calibrate their own uncertainty, select risk metrics that reflect both individual and collective safety, model how their actions shape human uncertainty and behavior, and communicate intent and confidence in ways that support predictable interaction. This synthesis positions psychological uncertainty as a behaviorally relevant variable for AV design and outlines a research agenda for safer, more understandable transport systems. ...
Doctoral thesis (2026) - Jesse J. Hagenaars, G.C.H.E. de Croon, Sander M. Bohté
Vision is central to autonomous flight. From crop monitoring and infrastructure inspection to search-and-rescue and defense, the ability to perceive and navigate a complex environment in real time determines what a drone can do and where it can go. Current computer vision achieves impressive results, but at considerable cost: deep neural networks on graphics processors consume tens of watts and add hundreds of grams. For larger drones, this limits flight time and on-board intelligence. For the smallest platforms, i.e., those light enough to safely operate near people, navigate collapsed buildings, or monitor ecosystems in massive swarms, it is prohibitive. Biology suggests this trade-off is not inevitable: insects navigate complex environments with brains consuming microwatts, relying on sparse, event-driven neural computation rather than dense, synchronous processing.
This dissertation investigates how event-based vision can enable effective autonomous behavior on flying robots. Event cameras, inspired by the biological retina, respond asynchronously to brightness changes rather than capturing full frames, offering microsecond temporal resolution and high dynamic range at milliwatts of power. Neuromorphic processors extend this paradigm to computation through spiking neural networks that communicate via discrete spikes. Together, they promise a fully event-driven vision pipeline whose efficiency gains would benefit robots of any size. This dissertation targets the hardest case, asking: how can a flying robot learn to navigate autonomously when it only has event-based vision? Four research questions decompose this problem into a systematic progression from learning to deployment, adaptation, and sensor minimalism.
Chapter 2 establishes that spiking neural networks can learn complex visual tasks, presenting the first deep spiking networks to solve dense optical flow estimation from events in a self-supervised manner, moving beyond simple classification to real-world vision problems relevant for robotics. Chapter 3 takes the next step by deploying a fully neuromorphic vision-to-control pipeline on Intel’s Loihi neuromorphic processor on board a flying quadrotor, achieving the first autonomous drone flight with a fully neuromorphic system, running inference at 200 Hz consuming only 7–12 mW. Chapter 4 brings learning on board by optimizing the self-supervised contrast maximization framework for computational efficiency, achieving a 100-fold runtime reduction that enables online depth learning during flight; just two minutes of on-device adaptation improved obstacle avoidance by about 30% over pre-training. Finally, Chapter 5 pushes toward ultimate sensor minimalism by demonstrating that vision can replace inertial measurement units entirely: a recurrent neural network learns to estimate attitude and rotation rates from events alone, enabling the first stable vision-only and IMU-free quadrotor flight.
Together, these contributions chart a path from theoretical promise to practical deployment for event-based robot vision, demonstrating that it is not merely a compelling concept but a viable foundation for autonomous flight. As the technology matures, it will enable a new generation of autonomous drones that perceive and navigate the world through efficient, brain-inspired vision. ...

Learning flight attitude from vision alone

Journal article (2026) - Jesse J. Hagenaars, Stein Stroobants, Sander M. Bohté, Guido C.H.E. de Croon
Vision is an essential part of attitude control for many flying animals, some of which have no dedicated sense of gravity. Flying robots, on the other hand, typically depend heavily on accelerometers and gyroscopes for attitude stabilization. In this work, we present the first vision-only approach to flight control for use in generic environments. We show that a quadrotor drone equipped with a downward-facing event camera can estimate its attitude and rotation rate from just the event stream, enabling flight control without inertial sensors. Our approach uses a small recurrent convolutional neural network trained through supervised learning. Real-world flight tests demonstrate that our combination of event camera and low-latency neural network is capable of replacing the inertial measurement unit in a traditional flight control loop. Furthermore, we investigate the network’s generalization across different environments, and the impact of memory and different fields of view. While networks with memory and access to horizon-like visual cues achieve best performance, variants with a narrower field of view achieve better relative generalization. Our work showcases vision-only flight control as a promising candidate for enabling autonomous, insect-scale flying robots. ...
Journal article (2026) - Dequan Ou, Jesse J. Hagenaars, Maciej R. Jankowski, Michiel V.M. Firlefyn, Christophe De Wagter, Florian T. Muijres, Jacqueline Degen, Guido C.H.E. de Croon
Navigation is a crucial capability for both animals and robots. Although tiny flying insects can robustly navigate over long distances1, state-of-the-art robot navigation methods are computationally expensive and therefore restricted to large robots2,3. Here we propose ‘Bee-Nav’, a highly efficient navigation strategy inspired by the visual learning flights of honeybees4, 5–6. In equivalent robotic learning flights, a tiny neural network is trained to map omnidirectional images to a home vector based on path integration. After learning, the robot can fly far away from home, come straight back using path integration and cancel integration drift using the visual homing network. Simulations showed that, for realistic path integration accuracies, the neural network requires training on only approximately 0.25–10.00% of the total flight area. In real-world indoor and outdoor experiments, a small drone successfully returned to within 0.5 m of home for 100% of 30–110-m flights and 70% of 200–600-m flights in windy conditions, using 3.4-kB and 42-kB neural networks, respectively. The proposed navigation strategy will be vital for resource-constrained robots that perform tasks while travelling from and to a home location. Furthermore, it provides new perspectives on the neuroethology of insect navigation, from how visual learning shapes homing trajectories to the nature of cognitive maps. ...
Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions because of the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present a fully neuromorphic vision-to- control pipeline for controlling a flying drone. Specifically, we trained a spiking neural network that accepts raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28,800 neurons, maps incoming raw events to ego-motion estimates and was trained with self-supervised learning on real event data. The control part consists of a single decoding layer and was learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to- real transfer of the fully learned neuromorphic pipeline. The drone could accurately control its ego-motion, allowing for hovering, landing, and maneuvering sideways—even while yawing at the same time. The neuromorphic pipeline runs on board on Intel’s Loihi neuromorphic processor with an execution frequency of 200 hertz, consuming 0.94 watt of idle power and a mere additional 7 to 12 milliwatts when running the network. These results illustrate the potential of neuromorphic sensing and processing for enabling insect-sized intelligent robots. ...
Insects have long been recognized for their ability to navigate and return home using visual cues from their nest's environment. However, the precise mechanism underlying this remarkable homing skill remains a subject of ongoing investigation. Drawing inspiration from the learning flights of honey bees and wasps, we propose a robot navigation method that directly learns the home vector direction from visual percepts during a learning flight in the vicinity of the nest. After learning, the robot will travel away from the nest, come back by means of odometry, and eliminate the resultant drift by inferring the home vector orientation from the currently experienced view. Using a compact convolutional neural network, we demonstrate successful learning in both simulated and real forest environments, as well as successful homing control of a simulated quadrotor. The average errors of the inferred home vectors in general stay well below the 90° required for successful homing, and below 24° if all images contain sufficient texture and illumination. Moreover, we show that the trajectory followed during the initial learning flight has a pronounced impact on the network's performance. A higher density of sample points in proximity to the nest results in a more consistent return. Code and data are available at https://mavlab.tudelft.nl/learning_to_home. ...
Neuromorphic processors like Loihi offer a promising alternative to conventional computing modules for endowing constrained systems like micro air vehicles (MAVs) with robust, efficient and autonomous skills such as take-off and landing, obstacle avoidance, and pursuit. However, a major challenge for using such processors on robotic platforms is the reality gap between simulation and the real world. In this study, we present for the very first time a fully embedded application of the Loihi neuromorphic chip prototype in a flying robot. A spiking neural network (SNN) was evolved to compute the thrust command based on the divergence of the ventral optic flow field to perform autonomous landing. Evolution was performed in a Python-based simulator using the PySNN library. The resulting network architecture consists of only 35 neurons distributed among 3 layers. Quantitative analysis between simulation and Loihi reveals a root-mean-square error of the thrust setpoint as low as 0.005 g, along with a 99.8% matching of the spike sequences in the hidden layer, and 99.7% in the output layer. The proposed approach successfully bridges the reality gap, offering important insights for future neuromorphic applications in robotics. Supplementary material is available at https://mavlab.tudelft.nl/loihi/. ...
Journal article (2020) - Jesse J. Hagenaars, Federico Paredes-Vallés, Sander M. Bohté, Guido C.H.E. De Croon
Flying insects are capable of vision-based navigation in cluttered environments, reliably avoiding obstacles through fast and agile maneuvers, while being very efficient in the processing of visual stimuli. Meanwhile, autonomous micro air vehicles still lag far behind their biological counterparts, displaying inferior performance at a much higher energy consumption. In light of this, we want to mimic flying insects in terms of their processing capabilities, and consequently show the efficiency of this approach in the real world. This letter does so through evolving spiking neural networks for controlling landings of micro air vehicles using optical flow divergence from a downward-looking camera. We demonstrate that the resulting neuromorphic controllers transfer robustly from a highly abstracted simulation to the real world, performing fast and safe landings while keeping network spike rate minimal. Furthermore, we provide insight into the resources required for successfully solving the problem of divergence-based landing, showing that high-resolution control can be learned with only a single spiking neuron. To the best of our knowledge, this work is the first to integrate spiking neural networks in the control loop of a real-world flying robot. Videos of the experiments can be found at https://bit.ly/neuro-controller. ...