Continuous-Time Modeling of Pixel Attributes

Abstract

There is an increasing interest in Event-based cameras for computer vision purposes. These cameras provide benefits such as high temporal resolution and high dynamic range by capturing the change in brightness instead of the absolute brightness signal. In order to train learning based behaviours with event-based camera, it is essential to be able to simulate the output of these cameras in a virtual world. Unlike their traditional frame-based counterparts, there are no real-time event-based camera rendering engines yet. All existing methods of generating events from virtual scenes impose a significant computational burden, simulating events by generating frames at a very high temporal resolution and converting these frames to events. This thesis proposes a novel method of event generation, by continuously interpolating between frames at a low temporal resolution, using auxiliary data from the rendering engine, such as depth and optical flow. In this method, the backwards pixel sampling problem is solved using a triangle grid created between all pixels, that is used to model the continuous response of each pixel between rendered frames. Other than previous methods that used VFI to temporally upsample LFR frames for event generation, this method does not create intermediate frames, instead simulating events directly from the continuous model. This method is implemented on the CPU, where it is shown to correctly interpolate the signal between frames. It is also implemented in Unreal Engine, where it is used to generate events in near real-time.