Optimizing Event-Based Vision by Realizing Super-Resolution in Event-Space: an Experimental Approach
M.M. Şabanoğlu (TU Delft - Mechanical Engineering)
Nergis Tömen – Mentor (TU Delft - Pattern Recognition and Bioinformatics)
Joost C F Winter – Mentor (TU Delft - Human-Robot Interaction)
Martijn Souman – Mentor
Jan van Gemert – Graduation committee member (TU Delft - Pattern Recognition and Bioinformatics)
Julian F.P. Kooij – Graduation committee member (TU Delft - Intelligent Vehicles)
Hesam Araghi – Graduation committee member (TU Delft - Pattern Recognition and Bioinformatics)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
An event-based camera enables capturing a video at a high temporal resolution, high dynamical range, reduced power consumption and minimal data bandwidth while the camera has minimal physical dimensions compared to a frame-based camera with the same vision properties. The limiting factor, however, of an event-based camera is the spatial resolution which ranges between 40 × 40 and 1280 × 960. To counter this deficiency, a method is researched to super resolve event-based vision in order to enhance spatial resolution. A selection of different neural network types and configurations are researched in a step-by-step fashion. Subsequent experiments tested the selected networks on their ability to process event-based data and extract features from it. Followed by experiments that exploited the limitations of the networks to super resolve at different ratios, lengths of eventstreams and more complex event-based data. Results of various experiments showed that a network configuration that utilizes a transformer architecture was best able to super resolve event-based vision. This type of network leverages the ability to extract features based on dependencies between events which aligns with the characteristics of event- based vision. Based on the obtained results from the exper- iments, a pipeline is proposed to super resolve event-based vision and consists of a combination of a transformer network, multilayer perceptrons and a k-nearest-neighbor algorithm. Using this pipeline, eventstreams can be super resolved in the spatial resolution at a scaling ratio of 4. Visually, these super resolved eventstreams resemble more detailed and enhanced version to the low-resolution input. This proposed pipeline can be considered as a starting point in further research toward the super-resolution of event-based data and thereby contributes to the extension of application possibilities of event-based vision.