J.J. Hagenaars
Please Note
8 records found
1
Navigating uncertainty
Safety solutions for autonomous vehicle integration into mixed-mode mobility
Uncertainty fundamentally shapes human perception and decision-making, a factor that is becoming increasingly critical as autonomous vehicles (AVs) begin to share physical and social spaces with humans. This perspective paper synthesizes insights from neuroscience, robotics, and behavioral science represent uncertainty and act under it. We identify a significant disconnect between low-level mathematical concepts of uncertainty and risk, the uncertainty-aware prediction and planning methods used in robotics, and the high-level psychological uncertainty experienced by humans. Here, we conceptualize an integration pathway for mixed-mode mobility: we show that all three treat control under partial information as the same two-stage problem – probabilistic inference of the current and future state, followed by a risk -weighted choice of action – and we make this shared structure explicit. We thus argue that AVs should quantify and calibrate their own uncertainty, select risk metrics that reflect both individual and collective safety, model how their actions shape human uncertainty and behavior, and communicate intent and confidence in ways that support predictable interaction. This synthesis positions psychological uncertainty as a behaviorally relevant variable for AV design and outlines a research agenda for safer, more understandable transport systems.
This dissertation investigates how event-based vision can enable effective autonomous behavior on flying robots. Event cameras, inspired by the biological retina, respond asynchronously to brightness changes rather than capturing full frames, offering microsecond temporal resolution and high dynamic range at milliwatts of power. Neuromorphic processors extend this paradigm to computation through spiking neural networks that communicate via discrete spikes. Together, they promise a fully event-driven vision pipeline whose efficiency gains would benefit robots of any size. This dissertation targets the hardest case, asking: how can a flying robot learn to navigate autonomously when it only has event-based vision? Four research questions decompose this problem into a systematic progression from learning to deployment, adaptation, and sensor minimalism.
Chapter 2 establishes that spiking neural networks can learn complex visual tasks, presenting the first deep spiking networks to solve dense optical flow estimation from events in a self-supervised manner, moving beyond simple classification to real-world vision problems relevant for robotics. Chapter 3 takes the next step by deploying a fully neuromorphic vision-to-control pipeline on Intel’s Loihi neuromorphic processor on board a flying quadrotor, achieving the first autonomous drone flight with a fully neuromorphic system, running inference at 200 Hz consuming only 7–12 mW. Chapter 4 brings learning on board by optimizing the self-supervised contrast maximization framework for computational efficiency, achieving a 100-fold runtime reduction that enables online depth learning during flight; just two minutes of on-device adaptation improved obstacle avoidance by about 30% over pre-training. Finally, Chapter 5 pushes toward ultimate sensor minimalism by demonstrating that vision can replace inertial measurement units entirely: a recurrent neural network learns to estimate attitude and rotation rates from events alone, enabling the first stable vision-only and IMU-free quadrotor flight.
Together, these contributions chart a path from theoretical promise to practical deployment for event-based robot vision, demonstrating that it is not merely a compelling concept but a viable foundation for autonomous flight. As the technology matures, it will enable a new generation of autonomous drones that perceive and navigate the world through efficient, brain-inspired vision. ...
This dissertation investigates how event-based vision can enable effective autonomous behavior on flying robots. Event cameras, inspired by the biological retina, respond asynchronously to brightness changes rather than capturing full frames, offering microsecond temporal resolution and high dynamic range at milliwatts of power. Neuromorphic processors extend this paradigm to computation through spiking neural networks that communicate via discrete spikes. Together, they promise a fully event-driven vision pipeline whose efficiency gains would benefit robots of any size. This dissertation targets the hardest case, asking: how can a flying robot learn to navigate autonomously when it only has event-based vision? Four research questions decompose this problem into a systematic progression from learning to deployment, adaptation, and sensor minimalism.
Chapter 2 establishes that spiking neural networks can learn complex visual tasks, presenting the first deep spiking networks to solve dense optical flow estimation from events in a self-supervised manner, moving beyond simple classification to real-world vision problems relevant for robotics. Chapter 3 takes the next step by deploying a fully neuromorphic vision-to-control pipeline on Intel’s Loihi neuromorphic processor on board a flying quadrotor, achieving the first autonomous drone flight with a fully neuromorphic system, running inference at 200 Hz consuming only 7–12 mW. Chapter 4 brings learning on board by optimizing the self-supervised contrast maximization framework for computational efficiency, achieving a 100-fold runtime reduction that enables online depth learning during flight; just two minutes of on-device adaptation improved obstacle avoidance by about 30% over pre-training. Finally, Chapter 5 pushes toward ultimate sensor minimalism by demonstrating that vision can replace inertial measurement units entirely: a recurrent neural network learns to estimate attitude and rotation rates from events alone, enabling the first stable vision-only and IMU-free quadrotor flight.
Together, these contributions chart a path from theoretical promise to practical deployment for event-based robot vision, demonstrating that it is not merely a compelling concept but a viable foundation for autonomous flight. As the technology matures, it will enable a new generation of autonomous drones that perceive and navigate the world through efficient, brain-inspired vision.
All eyes, no IMU
Learning flight attitude from vision alone
Vision is an essential part of attitude control for many flying animals, some of which have no dedicated sense of gravity. Flying robots, on the other hand, typically depend heavily on accelerometers and gyroscopes for attitude stabilization. In this work, we present the first vision-only approach to flight control for use in generic environments. We show that a quadrotor drone equipped with a downward-facing event camera can estimate its attitude and rotation rate from just the event stream, enabling flight control without inertial sensors. Our approach uses a small recurrent convolutional neural network trained through supervised learning. Real-world flight tests demonstrate that our combination of event camera and low-latency neural network is capable of replacing the inertial measurement unit in a traditional flight control loop. Furthermore, we investigate the network’s generalization across different environments, and the impact of memory and different fields of view. While networks with memory and access to horizon-like visual cues achieve best performance, variants with a narrower field of view achieve better relative generalization. Our work showcases vision-only flight control as a promising candidate for enabling autonomous, insect-scale flying robots.
Insects have long been recognized for their ability to navigate and return home using visual cues from their nest's environment. However, the precise mechanism underlying this remarkable homing skill remains a subject of ongoing investigation. Drawing inspiration from the learning flights of honey bees and wasps, we propose a robot navigation method that directly learns the home vector direction from visual percepts during a learning flight in the vicinity of the nest. After learning, the robot will travel away from the nest, come back by means of odometry, and eliminate the resultant drift by inferring the home vector orientation from the currently experienced view. Using a compact convolutional neural network, we demonstrate successful learning in both simulated and real forest environments, as well as successful homing control of a simulated quadrotor. The average errors of the inferred home vectors in general stay well below the 90° required for successful homing, and below 24° if all images contain sufficient texture and illumination. Moreover, we show that the trajectory followed during the initial learning flight has a pronounced impact on the network's performance. A higher density of sample points in proximity to the nest results in a more consistent return. Code and data are available at https://mavlab.tudelft.nl/learning_to_home.
Flying insects are capable of vision-based navigation in cluttered environments, reliably avoiding obstacles through fast and agile maneuvers, while being very efficient in the processing of visual stimuli. Meanwhile, autonomous micro air vehicles still lag far behind their biological counterparts, displaying inferior performance at a much higher energy consumption. In light of this, we want to mimic flying insects in terms of their processing capabilities, and consequently show the efficiency of this approach in the real world. This letter does so through evolving spiking neural networks for controlling landings of micro air vehicles using optical flow divergence from a downward-looking camera. We demonstrate that the resulting neuromorphic controllers transfer robustly from a highly abstracted simulation to the real world, performing fast and safe landings while keeping network spike rate minimal. Furthermore, we provide insight into the resources required for successfully solving the problem of divergence-based landing, showing that high-resolution control can be learned with only a single spiking neuron. To the best of our knowledge, this work is the first to integrate spiking neural networks in the control loop of a real-world flying robot. Videos of the experiments can be found at https://bit.ly/neuro-controller.