Learning Scale-Aware Optical Flow
X. Wen (TU Delft - Electrical Engineering, Mathematics and Computer Science)
MJT Reinders – Mentor (TU Delft - Pattern Recognition and Bioinformatics)
J.C. van Gemert – Mentor (TU Delft - Pattern Recognition and Bioinformatics)
Marco Loog – Graduation committee member (TU Delft - Pattern Recognition and Bioinformatics)
A Hanjalic – Graduation committee member (TU Delft - Intelligent Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Optical flow is a representation of projected real-world motion of the object between two consecutive images. The optical flow measures the pixel displacement on the image coordinate plane. However, it does not reveal the motion in depth explicitly, which could be useful as input in some tasks such as vehicle tracking. To extend the original optical flow approach, we model the depth change of the object as the scale change of object in the image and present an approach to estimate the scale change and optical flow jointly. Considering the scenario that obvious scale change occurs between two images, the traditional convolution network fails because it lacks the scale invariance. According to the Scale-space theory and the idea of learning a combination of Gaussian derivative basis to approximate arbitrary filters, we build a Basis Convolution layer that allows the network to see the scale change between two images and make use of it to better capture the same feature with various scales on two images.
We test our models on our own optical flow datasets which involve obvious scale change. According to the experimental results, our method is capable of estimating the scale change between images, and it significantly improves the optical flow estimation by modelling the scale change explicitly.