Towards Understanding RGB-Depth Pre-Training in ViT-based Models
An Exploration of a Novel Training Regime
P.M. Skullerud (TU Delft - Electrical Engineering, Mathematics and Computer Science)
J.C. van Gemert – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
P. Kellnhofer – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
P.J.W. Reijalt – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Adding depth to RGB inputs (RGB-D) is known to improve model accuracy. State-of-the-art RGB-D models routinely adapt the Vision Transformer (ViT), but training ViTs purely on RGB-D is infeasible given the scarcity of depth data. A solution is using large RGB datasets to pre-train before fine-tuning on RGB-D, leveraging depth estimators to add complementary pseudo-depth to RGB datasets. We investigate the characteristics of models trained in this setup. We find that models, regardless of RGB-D fusion architecture, consistently learn simple patterns of depth utilization in the attention mechanism and across encoder layers. Our conclusions motivate the need to justify proposed depth fusion architectures against simple baselines, and to use depth fusion modules suited to the value of depth at each layer. We also show that after pre-training on pseudo-depth, fine-tuning favors pseudo- as opposed to real depth, highlighting the importance of minimizing their differences.