Addressing Illumination-Based Domain Shifts in Deep Learning
A Physics-Based Approach
A. Lengyel (TU Delft - Electrical Engineering, Mathematics and Computer Science)
J.C. Gemert – Mentor (TU Delft - Pattern Recognition and Bioinformatics)
Marcel J. T. Reinders – Graduation committee member (TU Delft - Pattern Recognition and Bioinformatics)
K.A. Hildebrandt – Graduation committee member (TU Delft - Computer Graphics and Visualisation)
Michael J. Milford – Mentor (Queensland University of Technology)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
This work investigates how prior knowledge from physics-based reflection models can be used to improve the performance of semantic segmentation models under an illumination-based domain shift. We implement various color invariants as a preprocessing step and find that CNNs trained on these color invariants get stuck in worse local minima compared to RGB inputs, but can achieve comparable or even superior performance when applying knowledge transfer from RGB. We also find Batch Normalization to severely affect the performance of neural networks under an illumination-based domain shift and demonstrate that Instance Normalization offers a simple remedy to this issue. Additionally, we investigate different fusion models for combining color invariants with RGB. Using a combination of these methods we achieve a 14.5% performance increase on nighttime semantic segmentation without any additional training data.