Real-time Building of Multiplane Images from RGBD Data
F.J. van Onna (TU Delft - Electrical Engineering, Mathematics and Computer Science)
P. Kellnhofer – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Multiplane Images (MPIs) are among the simpler scene representations used for novel view synthesis. They consist of stacks of partially transparent RGB images that create parallax when viewed from different viewpoints. As MPIs are a method of adding depth to images, one might expect them to have been extensively studied in conjunction with images obtained from depth cameras (RGB-D images), but the contrary is true.
We propose a method for generating MPIs from RGB-D images using only classical algorithms, as opposed to machine learning-based methods. This makes the approach fast to run and therefore suitable for real-time use, such as building MPIs from live video streams captured by depth cameras. Our method is evaluated in terms of performance using scenes adapted from the VKITTI 2 dataset. Results show that, on a modern laptop, our method far exceeds the speed required to ingest real-time data. The process can still run at interactive frame rates when generating MPIs with more than 400 planes.
Since visual artifacts remain clearly visible in the generated MPIs, more sophisticated algorithms should be investigated to alleviate these artifacts while maintaining real-time performance.