YW
Y. Wu
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
5 records found
1
Continuous-Time Modelling of Pixel Attributes
The Development of a Continuous Frame Interpolation Method and a Fast, Real-Time Event Camera Simulator
There is an increasing interest in Event-based cameras for computer vision purposes. These cameras provide benefits such as high temporal resolution and high dynamic range by capturing the change in brightness instead of the absolute brightness signal. In order to train learning-based behaviours with event-based cameras, it is essential to be able to simulate the output of these cameras in a virtual world. Unlike their traditional frame-based counterparts, there are no real-time event-based camera rendering engines yet. All existing methods of generating events from virtual scenes impose a significant computational burden, simulating events by generating frames at a very high temporal resolution and converting these frames to events. This thesis proposes a novel method of event generation, by continuously interpolating between frames at a low temporal resolution, using auxiliary data from the rendering engine, such as depth and optical flow. In this method, the backwards pixel sampling problem is solved using a triangle grid created between all pixels, that is used to model the continuous response of each pixel between rendered frames. Other than previous methods that used VFI to temporally upsample LFR frames for event generation, this method does not create intermediate frames, instead simulating events directly from the continuous model. This method is implemented on the CPU, where it is shown to correctly interpolate the signal between frames. It is also implemented in Unreal Engine, where it is used to generate events in near real-time.
...
There is an increasing interest in Event-based cameras for computer vision purposes. These cameras provide benefits such as high temporal resolution and high dynamic range by capturing the change in brightness instead of the absolute brightness signal. In order to train learning-based behaviours with event-based cameras, it is essential to be able to simulate the output of these cameras in a virtual world. Unlike their traditional frame-based counterparts, there are no real-time event-based camera rendering engines yet. All existing methods of generating events from virtual scenes impose a significant computational burden, simulating events by generating frames at a very high temporal resolution and converting these frames to events. This thesis proposes a novel method of event generation, by continuously interpolating between frames at a low temporal resolution, using auxiliary data from the rendering engine, such as depth and optical flow. In this method, the backwards pixel sampling problem is solved using a triangle grid created between all pixels, that is used to model the continuous response of each pixel between rendered frames. Other than previous methods that used VFI to temporally upsample LFR frames for event generation, this method does not create intermediate frames, instead simulating events directly from the continuous model. This method is implemented on the CPU, where it is shown to correctly interpolate the signal between frames. It is also implemented in Unreal Engine, where it is used to generate events in near real-time.
Master thesis
(2025)
-
T.N.A. den Blanken, Y. Wu, G.C.H.E. de Croon, Holger Caesar, C. de Wagter, L. Ferranti
Event-based cameras provide high temporal resolution, robustness to lighting conditions and low power consumption, but their sparse, temporal data require models that reason over time. In supervised settings, this is increasingly handled with recurrent architectures. In contrast, most self-supervised learning (SSL) methods still adapt non-recurrent RGB techniques, with masking-based objectives that favor spatial reconstruction over temporal understanding. We introduce SPICE: Self-supervised Predictive Coding on Events, an SSL framework tailored to event data that processes longer sequences recurrently and learns by predicting future latent representations rather than reconstructing masked inputs, promoting a more natural objective focyused on anticipating what comes next. SPICE further incorporates an event-specific contrastive loss only operating on active regions. SPICE pre-training improves downstream performance on semantic segmentation, depth estimation and optical flow estimation. Low-dimensional projections confirm that the learned representations are meaningful and avoid collapse, while also revealing limitations in temporal stability and semantic organization, indicating clear directions for future event-specific SSL research. Code is available upon request.
...
Event-based cameras provide high temporal resolution, robustness to lighting conditions and low power consumption, but their sparse, temporal data require models that reason over time. In supervised settings, this is increasingly handled with recurrent architectures. In contrast, most self-supervised learning (SSL) methods still adapt non-recurrent RGB techniques, with masking-based objectives that favor spatial reconstruction over temporal understanding. We introduce SPICE: Self-supervised Predictive Coding on Events, an SSL framework tailored to event data that processes longer sequences recurrently and learns by predicting future latent representations rather than reconstructing masked inputs, promoting a more natural objective focyused on anticipating what comes next. SPICE further incorporates an event-specific contrastive loss only operating on active regions. SPICE pre-training improves downstream performance on semantic segmentation, depth estimation and optical flow estimation. Low-dimensional projections confirm that the learned representations are meaningful and avoid collapse, while also revealing limitations in temporal stability and semantic organization, indicating clear directions for future event-specific SSL research. Code is available upon request.
The estimation of optical flow, which determines the movement of objects in a visual scene, is a crucial problem in computer vision. It is essential for applications such as autonomous navigation, where precise motion estimation is critical for performance and safety.
Frame-based cameras capture sequences of still images at regular intervals, from which optical flow is traditionally extracted using optimization-based or learning-based methods. Recently, event-based cameras, which detect changes in pixel brightness asynchronously, have gained traction due to their high temporal resolution and robustness to motion blur, and many algorithms have been developed to estimate optical flow from this data. IDNet is a learning-based approach that achieves state-of-the-art performance. However, IDNet and similar models face two major challenges: they require labeled ground-truth data for training, which is scarce and difficult to collect, and they rely on recurrent neural networks (RNNs) with a fixed number of refinement iterations. This fixed iteration scheme does not adapt to scene complexity, limiting accuracy for complex flows and increasing computational effort for simpler patterns.
The aim of this project is to explore, implement, and evaluate potential methods to address these two mentioned limitations and enhance the capabilities of models like IDNet.
To remove the need for ground-truth data, a self-supervised learning paradigm was implemented by introducing a novel contrast maximization loss that assesses the blur present when accumulating raw events for a certain time interval and compensating it with the predicted flow. To assess the effectiveness of this method, models were trained on the benchmark MVSEC dataset, showing improved results over previous methods with up to 15% on some sequences and an 8% improvement on average. Based on these experiments and results, further research directions were proposed.
As for the problem of the current fixed iteration scheme, Deep Equilibrium Models were found to provide a promising pathway to solving it. These novel models reformulate their iterative structure into a root-finding problem and utilize traditional solvers to find a solution based on some tolerance, providing a trade-off between speed and accuracy. Moreover, they allow for direct differentiation through the network using only their final estimate, compared to previous methods that keep track of their state through all iterations, leading to an O(1) memory consumption. Implementing these and some additional ideas, the trained DEQ IDNet model reached competitive performance on the DSEC dataset while consuming 15% less memory. Yet, further work is needed to close the gap and achieve state-of-the-art performance. ...
Frame-based cameras capture sequences of still images at regular intervals, from which optical flow is traditionally extracted using optimization-based or learning-based methods. Recently, event-based cameras, which detect changes in pixel brightness asynchronously, have gained traction due to their high temporal resolution and robustness to motion blur, and many algorithms have been developed to estimate optical flow from this data. IDNet is a learning-based approach that achieves state-of-the-art performance. However, IDNet and similar models face two major challenges: they require labeled ground-truth data for training, which is scarce and difficult to collect, and they rely on recurrent neural networks (RNNs) with a fixed number of refinement iterations. This fixed iteration scheme does not adapt to scene complexity, limiting accuracy for complex flows and increasing computational effort for simpler patterns.
The aim of this project is to explore, implement, and evaluate potential methods to address these two mentioned limitations and enhance the capabilities of models like IDNet.
To remove the need for ground-truth data, a self-supervised learning paradigm was implemented by introducing a novel contrast maximization loss that assesses the blur present when accumulating raw events for a certain time interval and compensating it with the predicted flow. To assess the effectiveness of this method, models were trained on the benchmark MVSEC dataset, showing improved results over previous methods with up to 15% on some sequences and an 8% improvement on average. Based on these experiments and results, further research directions were proposed.
As for the problem of the current fixed iteration scheme, Deep Equilibrium Models were found to provide a promising pathway to solving it. These novel models reformulate their iterative structure into a root-finding problem and utilize traditional solvers to find a solution based on some tolerance, providing a trade-off between speed and accuracy. Moreover, they allow for direct differentiation through the network using only their final estimate, compared to previous methods that keep track of their state through all iterations, leading to an O(1) memory consumption. Implementing these and some additional ideas, the trained DEQ IDNet model reached competitive performance on the DSEC dataset while consuming 15% less memory. Yet, further work is needed to close the gap and achieve state-of-the-art performance. ...
The estimation of optical flow, which determines the movement of objects in a visual scene, is a crucial problem in computer vision. It is essential for applications such as autonomous navigation, where precise motion estimation is critical for performance and safety.
Frame-based cameras capture sequences of still images at regular intervals, from which optical flow is traditionally extracted using optimization-based or learning-based methods. Recently, event-based cameras, which detect changes in pixel brightness asynchronously, have gained traction due to their high temporal resolution and robustness to motion blur, and many algorithms have been developed to estimate optical flow from this data. IDNet is a learning-based approach that achieves state-of-the-art performance. However, IDNet and similar models face two major challenges: they require labeled ground-truth data for training, which is scarce and difficult to collect, and they rely on recurrent neural networks (RNNs) with a fixed number of refinement iterations. This fixed iteration scheme does not adapt to scene complexity, limiting accuracy for complex flows and increasing computational effort for simpler patterns.
The aim of this project is to explore, implement, and evaluate potential methods to address these two mentioned limitations and enhance the capabilities of models like IDNet.
To remove the need for ground-truth data, a self-supervised learning paradigm was implemented by introducing a novel contrast maximization loss that assesses the blur present when accumulating raw events for a certain time interval and compensating it with the predicted flow. To assess the effectiveness of this method, models were trained on the benchmark MVSEC dataset, showing improved results over previous methods with up to 15% on some sequences and an 8% improvement on average. Based on these experiments and results, further research directions were proposed.
As for the problem of the current fixed iteration scheme, Deep Equilibrium Models were found to provide a promising pathway to solving it. These novel models reformulate their iterative structure into a root-finding problem and utilize traditional solvers to find a solution based on some tolerance, providing a trade-off between speed and accuracy. Moreover, they allow for direct differentiation through the network using only their final estimate, compared to previous methods that keep track of their state through all iterations, leading to an O(1) memory consumption. Implementing these and some additional ideas, the trained DEQ IDNet model reached competitive performance on the DSEC dataset while consuming 15% less memory. Yet, further work is needed to close the gap and achieve state-of-the-art performance.
Frame-based cameras capture sequences of still images at regular intervals, from which optical flow is traditionally extracted using optimization-based or learning-based methods. Recently, event-based cameras, which detect changes in pixel brightness asynchronously, have gained traction due to their high temporal resolution and robustness to motion blur, and many algorithms have been developed to estimate optical flow from this data. IDNet is a learning-based approach that achieves state-of-the-art performance. However, IDNet and similar models face two major challenges: they require labeled ground-truth data for training, which is scarce and difficult to collect, and they rely on recurrent neural networks (RNNs) with a fixed number of refinement iterations. This fixed iteration scheme does not adapt to scene complexity, limiting accuracy for complex flows and increasing computational effort for simpler patterns.
The aim of this project is to explore, implement, and evaluate potential methods to address these two mentioned limitations and enhance the capabilities of models like IDNet.
To remove the need for ground-truth data, a self-supervised learning paradigm was implemented by introducing a novel contrast maximization loss that assesses the blur present when accumulating raw events for a certain time interval and compensating it with the predicted flow. To assess the effectiveness of this method, models were trained on the benchmark MVSEC dataset, showing improved results over previous methods with up to 15% on some sequences and an 8% improvement on average. Based on these experiments and results, further research directions were proposed.
As for the problem of the current fixed iteration scheme, Deep Equilibrium Models were found to provide a promising pathway to solving it. These novel models reformulate their iterative structure into a root-finding problem and utilize traditional solvers to find a solution based on some tolerance, providing a trade-off between speed and accuracy. Moreover, they allow for direct differentiation through the network using only their final estimate, compared to previous methods that keep track of their state through all iterations, leading to an O(1) memory consumption. Implementing these and some additional ideas, the trained DEQ IDNet model reached competitive performance on the DSEC dataset while consuming 15% less memory. Yet, further work is needed to close the gap and achieve state-of-the-art performance.
Although deep reinforcement learning (DRL) is a highly promising approach to learning robotic vision-based control, it is plagued by long training times. This report introduces a DRL setup that relies on self-supervised learning for extracting depth information valuable for navigation. Specifically, a literature study is conducted to investigate the effects of learning how to synthesize one view from the other in a stereo-vision setup without relying on any preliminary knowledge of the camera extrinisics and how it can be integrated for its downstream use for an obstacle avoidance task. As such, the literature study concludes that competitive geometry-free monocular-to-stereo image view synthesis is feasible due to recent developments in computer vision. The scientific paper further develops concepts proposed in the literature study and benchmarks the proposed architectures on depth estimation benchmarks for KITTI. Competitive results are achieved for view synthesis and despite sub-optimal performance compared to state-of-the-art monocular depth estimation, an ability to encode depth and detect shapes is present and, therefore, satisfactory for the application to DRL. Additionally, the research examines the benefits of using the latent space of a view synthesis architecture compared to other feature extractor methods as an input to the PPO agent implemented as auxiliary tasks. This method achieves quicker convergence and better performance for an obstacle avoidance task in a simulated indoor environment than the autoencoding feature extractor and end-to-end DRL methods. It is only outperformed by the monocular depth estimation feature extractor method. Overall, this research provides valuable insights for developing more efficient and effective DRL methods for monocular camera-based drones. Finally, the complementary code for this research can be found: \url{https://github.com/ldenridder/drl-obstacle-avoidance-view-synthesis}.
...
Although deep reinforcement learning (DRL) is a highly promising approach to learning robotic vision-based control, it is plagued by long training times. This report introduces a DRL setup that relies on self-supervised learning for extracting depth information valuable for navigation. Specifically, a literature study is conducted to investigate the effects of learning how to synthesize one view from the other in a stereo-vision setup without relying on any preliminary knowledge of the camera extrinisics and how it can be integrated for its downstream use for an obstacle avoidance task. As such, the literature study concludes that competitive geometry-free monocular-to-stereo image view synthesis is feasible due to recent developments in computer vision. The scientific paper further develops concepts proposed in the literature study and benchmarks the proposed architectures on depth estimation benchmarks for KITTI. Competitive results are achieved for view synthesis and despite sub-optimal performance compared to state-of-the-art monocular depth estimation, an ability to encode depth and detect shapes is present and, therefore, satisfactory for the application to DRL. Additionally, the research examines the benefits of using the latent space of a view synthesis architecture compared to other feature extractor methods as an input to the PPO agent implemented as auxiliary tasks. This method achieves quicker convergence and better performance for an obstacle avoidance task in a simulated indoor environment than the autoencoding feature extractor and end-to-end DRL methods. It is only outperformed by the monocular depth estimation feature extractor method. Overall, this research provides valuable insights for developing more efficient and effective DRL methods for monocular camera-based drones. Finally, the complementary code for this research can be found: \url{https://github.com/ldenridder/drl-obstacle-avoidance-view-synthesis}.
Bachelor thesis
(2022)
-
A. Singh, A.P. Tripathi, T. Aantjes, S. Blaga, A. Pavlov, V. Gupta, G. Basile, D.F. Bolkestein, M.L. Mazurkiewicz, I.A. Quintanilla Rojas, I. Uriol Balbin, C.J. van Calcar, Y. Wu, Stijn Mast
Maintaining a high speed, secure and robust connection to the world is of paramount importance
in modern society. This proved especially true during a recent volcanic eruption in the small island nation of Tonga, where the only optical fibre line to the country was severed due to the cataclysm, completely disconnecting the island from the rest of the world. The lack of communication made the disaster relief to the eruption more complex. The Emergency Communications Node (ECN) can be deployed within 48 hours to enable alternative means of communication that replace damaged communication lines... ...
in modern society. This proved especially true during a recent volcanic eruption in the small island nation of Tonga, where the only optical fibre line to the country was severed due to the cataclysm, completely disconnecting the island from the rest of the world. The lack of communication made the disaster relief to the eruption more complex. The Emergency Communications Node (ECN) can be deployed within 48 hours to enable alternative means of communication that replace damaged communication lines... ...
Maintaining a high speed, secure and robust connection to the world is of paramount importance
in modern society. This proved especially true during a recent volcanic eruption in the small island nation of Tonga, where the only optical fibre line to the country was severed due to the cataclysm, completely disconnecting the island from the rest of the world. The lack of communication made the disaster relief to the eruption more complex. The Emergency Communications Node (ECN) can be deployed within 48 hours to enable alternative means of communication that replace damaged communication lines...
in modern society. This proved especially true during a recent volcanic eruption in the small island nation of Tonga, where the only optical fibre line to the country was severed due to the cataclysm, completely disconnecting the island from the rest of the world. The lack of communication made the disaster relief to the eruption more complex. The Emergency Communications Node (ECN) can be deployed within 48 hours to enable alternative means of communication that replace damaged communication lines...