PS
P.M. Skullerud
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Towards Understanding RGB-Depth Pre-Training in ViT-based Models
An Exploration of a Novel Training Regime
Adding depth to RGB inputs (RGB-D) is known to improve model accuracy. State-of-the-art RGB-D models routinely adapt the Vision Transformer (ViT), but training ViTs purely on RGB-D is infeasible given the scarcity of depth data. A solution is using large RGB datasets to pre-train before fine-tuning on RGB-D, leveraging depth estimators to add complementary pseudo-depth to RGB datasets. We investigate the characteristics of models trained in this setup. We find that models, regardless of RGB-D fusion architecture, consistently learn simple patterns of depth utilization in the attention mechanism and across encoder layers. Our conclusions motivate the need to justify proposed depth fusion architectures against simple baselines, and to use depth fusion modules suited to the value of depth at each layer. We also show that after pre-training on pseudo-depth, fine-tuning favors pseudo- as opposed to real depth, highlighting the importance of minimizing their differences.
...
Adding depth to RGB inputs (RGB-D) is known to improve model accuracy. State-of-the-art RGB-D models routinely adapt the Vision Transformer (ViT), but training ViTs purely on RGB-D is infeasible given the scarcity of depth data. A solution is using large RGB datasets to pre-train before fine-tuning on RGB-D, leveraging depth estimators to add complementary pseudo-depth to RGB datasets. We investigate the characteristics of models trained in this setup. We find that models, regardless of RGB-D fusion architecture, consistently learn simple patterns of depth utilization in the attention mechanism and across encoder layers. Our conclusions motivate the need to justify proposed depth fusion architectures against simple baselines, and to use depth fusion modules suited to the value of depth at each layer. We also show that after pre-training on pseudo-depth, fine-tuning favors pseudo- as opposed to real depth, highlighting the importance of minimizing their differences.
Optical Flow Estimation Using Event-Based Cameras
Improving Optical Flow Estimation Accuracy Using Space-Aware De-Flickering
Event cameras are novel sensors whose high temporal resolution and bandwidth motivate their use for the optical flow estimation problem. However, the properties of event cameras also introduce a vulnerability to flickering. Flickering hurts the perceptibility of motion by overwhelming event data with unrelated information. The single existing event de-flicker method (EFR) is built for scenarios where the relative position of the camera and the flickering object is constant, which is uncommon in motion-heavy optical flow estimation scenarios. Our contribution is a new de-flickering method that incorporates spatial awareness of nearby pixels. We hypothesize this feature to increase robustness to movement, and thus to better improve optical flow accuracy. Compared to EFR our method falters at filtering intensely flickering surfaces, but better preserves the spatial coherence of edges. However, we observe that both de-flickering methods remove much geometric information, especially given slow motion or weak ambient illumination. Our benchmarking shows that neither our method nor EFR significantly affects optical flow estimation accuracy, despite reducing event counts by 50-65%. Overall, we conclude that the niche benefits of spatial filtering are nullified by the result that filtering hardly affects optical flow estimation.
...
Event cameras are novel sensors whose high temporal resolution and bandwidth motivate their use for the optical flow estimation problem. However, the properties of event cameras also introduce a vulnerability to flickering. Flickering hurts the perceptibility of motion by overwhelming event data with unrelated information. The single existing event de-flicker method (EFR) is built for scenarios where the relative position of the camera and the flickering object is constant, which is uncommon in motion-heavy optical flow estimation scenarios. Our contribution is a new de-flickering method that incorporates spatial awareness of nearby pixels. We hypothesize this feature to increase robustness to movement, and thus to better improve optical flow accuracy. Compared to EFR our method falters at filtering intensely flickering surfaces, but better preserves the spatial coherence of edges. However, we observe that both de-flickering methods remove much geometric information, especially given slow motion or weak ambient illumination. Our benchmarking shows that neither our method nor EFR significantly affects optical flow estimation accuracy, despite reducing event counts by 50-65%. Overall, we conclude that the niche benefits of spatial filtering are nullified by the result that filtering hardly affects optical flow estimation.