M. Zaffar
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
4 records found
1
Deep neural networks are increasingly used for ego-motion estimation. Often, in self-supervised ego-motion networks, it is decoded from a depth network and until now has not been decoded from an optical flow network. This is surprising given the tight relationship between optical flow and ego-motion. While both representations are widely used in learning-based approaches, the extent to which their latent space encodes motion information remains poorly understood. This paper presents a controlled analysis of how well depth-based and optical flow-based neural networks encode ego-motion. Using supervised depth, flow and pose networks trained on TartanAirV2, we probe motion information by attaching identical minimal pose decoders to frozen encoders. Representational space analysis through Centered Kernel Analysis (CKA) and feature space analysis through Principal Component Analysis (PCA) are used to examine how motion information is structured across network hierarchies. This work shows that optical flow representations encode ego-motion information more explicitly, in a lower dimensional, linearly accessible structure as opposed to depth representations, which exhibit weak alignment with pose. These findings suggest that in a self-supervised setting, ego-motion estimation can best be decoded from an optical flow network as opposed to a depth network.
...
Deep neural networks are increasingly used for ego-motion estimation. Often, in self-supervised ego-motion networks, it is decoded from a depth network and until now has not been decoded from an optical flow network. This is surprising given the tight relationship between optical flow and ego-motion. While both representations are widely used in learning-based approaches, the extent to which their latent space encodes motion information remains poorly understood. This paper presents a controlled analysis of how well depth-based and optical flow-based neural networks encode ego-motion. Using supervised depth, flow and pose networks trained on TartanAirV2, we probe motion information by attaching identical minimal pose decoders to frozen encoders. Representational space analysis through Centered Kernel Analysis (CKA) and feature space analysis through Principal Component Analysis (PCA) are used to examine how motion information is structured across network hierarchies. This work shows that optical flow representations encode ego-motion information more explicitly, in a lower dimensional, linearly accessible structure as opposed to depth representations, which exhibit weak alignment with pose. These findings suggest that in a self-supervised setting, ego-motion estimation can best be decoded from an optical flow network as opposed to a depth network.
Visual Place Recognition (VPR) remains a challenging problem, particularly under difficult conditions such as night-time or winter weather, which are often underrepresented in existing training datasets. Although transformer-based models have recently advanced the state-of-the-art, their high computational demands can hinder deployment in real-world robotic systems. This thesis proposes a new data augmentation strategy for VPR using image-to-image Vision Foundation Model InstructPix2Pix to generate realistic visual variations such as night and snow scenes from the original training data. These synthetic augmentations are added to the original training dataset to extend dataset diversity without requiring additional data collection. To further improve performance, the method is combined with more advanced augmentations using the Kornia library, which already improves robustness over traditional augmentation techniques. Experiments on multiple benchmark datasets show that lightweight, ResNet-based models trained with our VFM augmentations achieve significantly improved performance under challenging visual conditions. Additional ablations demonstrate the importance of careful prompt design and hyperparameter tuning. Overall, this work shows that VFMs can serve as practical tools for targeted dataset augmentation, improving the robustness of existing VPR methods in difficult scenarios.
...
...
Visual Place Recognition (VPR) remains a challenging problem, particularly under difficult conditions such as night-time or winter weather, which are often underrepresented in existing training datasets. Although transformer-based models have recently advanced the state-of-the-art, their high computational demands can hinder deployment in real-world robotic systems. This thesis proposes a new data augmentation strategy for VPR using image-to-image Vision Foundation Model InstructPix2Pix to generate realistic visual variations such as night and snow scenes from the original training data. These synthetic augmentations are added to the original training dataset to extend dataset diversity without requiring additional data collection. To further improve performance, the method is combined with more advanced augmentations using the Kornia library, which already improves robustness over traditional augmentation techniques. Experiments on multiple benchmark datasets show that lightweight, ResNet-based models trained with our VFM augmentations achieve significantly improved performance under challenging visual conditions. Additional ablations demonstrate the importance of careful prompt design and hyperparameter tuning. Overall, this work shows that VFMs can serve as practical tools for targeted dataset augmentation, improving the robustness of existing VPR methods in difficult scenarios.
Visual place recognition (VPR) is a form of visual localization. Current approaches are designed to handle common VPR challenges, such as appearance and viewpoint variations. With the introduction of DINOv2, vision foundation models have been used as feature extractors to improve performance for VPR techniques, as they show great generalizing capabilities for image representations. By fine-tuning these large models on VPR-specific datasets, performance increases even more. A problem with these big VPR datasets is the bias towards urban environments. To solve this problem, we propose to use a simple pipeline to fine-tune existing techniques on the reference databases of test datasets. Our experiments show that performance improves by reference database fine-tuning for multiple techniques on different datasets. To handle appearance and viewpoint variations as well, image augmentations can be used during training. With this complete pipeline, techniques improve performance. The experiments show improvement even if a large query-reference domain gap exists for that dataset given that a part of the test queries are know during fine-tuning.
...
Visual place recognition (VPR) is a form of visual localization. Current approaches are designed to handle common VPR challenges, such as appearance and viewpoint variations. With the introduction of DINOv2, vision foundation models have been used as feature extractors to improve performance for VPR techniques, as they show great generalizing capabilities for image representations. By fine-tuning these large models on VPR-specific datasets, performance increases even more. A problem with these big VPR datasets is the bias towards urban environments. To solve this problem, we propose to use a simple pipeline to fine-tune existing techniques on the reference databases of test datasets. Our experiments show that performance improves by reference database fine-tuning for multiple techniques on different datasets. To handle appearance and viewpoint variations as well, image augmentations can be used during training. With this complete pipeline, techniques improve performance. The experiments show improvement even if a large query-reference domain gap exists for that dataset given that a part of the test queries are know during fine-tuning.
VPR describes a task where an agent (e.g., a robot) attempts to recognize its current location by comparing the incoming visual data from its sensor(s) (query images), usually a camera, to geotagged reference images. Both query and reference images are described using a feature extractor, and the query descriptor is matched to its closest reference descriptor in the feature space. Within VPR there are many different VPR techniques that have been proposed throughout the years with many different types of architecture and trained on different datasets. With the many test datasets available, there exists no VPR technique that is able to reach state-of-the-art performance on all these datasets. For this reason, existing work has argued it can be beneficial to utilize an ensemble-based method to combine multiple VPR techniques and achieve better VPR performance. Some of these Ensemble-based methods have already been proposed. These ensemble-methods combine individual VPR techniques and weigh their predictions using these same predictions to give an indication of their confidence. This calculation, however, is strictly based on predictions obtained from applying the VPR techniques on test data at inference time. Generally within VPR research, the dataset that was used to train the VPR technique is often different from the dataset it is tested on. This means there is a domain shift between the training and test data. This domain shift is not taken into account when weighting the predictions of VPR techniques in an ensemble using these existing methods. In this work, we analyze how this degree of domain shift between train and test data, which can be observed by looking at the relative location of descriptors in the feature space, impacts downstream VPR performance. Intuitively, one would expect better VPR performance in a situation where the degree of train-test domain shift is minimal. Our analysis shows that this is indeed the case. We propose two different methods that utilize this degree of domain shift to calculate the weights given to the VPR techniques in an ensemble. First, we propose a generative method. Here weights are given to the VPR techniques based on the likelihood that the query sample originated from the same distribution as the training dataset of the technique and is in distribution. This way each individual technique is given a weight. Secondly, we propose a discriminative method. Here weights are given to the training datasets used to train the techniques in the ensemble. These training datasets are given weights based on relative proximity to a query sample in the feature space, an indicator for the degree of domain-shift between the training dataset and the query sample. all VPR techniques are given the weight corresponding to their training dataset. We compare these proposed approaches to other ensemble-based baselines and individual VPR techniques. The quantitative results show that our proposed methods generally outperform the ensemblebased baselines and the individual VPR techniques. We also propose further future work. One of the generative methods still delivers lower performance than could be possible, caused by applying this method to high-dimensionality descriptors. A solution for this issue should lead to higher VPR performance using this method. Additionally, we suggest future work to expand on the datasets used in this research, to strengthen the claims made, and verify that results and trends found to hold up when testing and training using other datasets.
...
VPR describes a task where an agent (e.g., a robot) attempts to recognize its current location by comparing the incoming visual data from its sensor(s) (query images), usually a camera, to geotagged reference images. Both query and reference images are described using a feature extractor, and the query descriptor is matched to its closest reference descriptor in the feature space. Within VPR there are many different VPR techniques that have been proposed throughout the years with many different types of architecture and trained on different datasets. With the many test datasets available, there exists no VPR technique that is able to reach state-of-the-art performance on all these datasets. For this reason, existing work has argued it can be beneficial to utilize an ensemble-based method to combine multiple VPR techniques and achieve better VPR performance. Some of these Ensemble-based methods have already been proposed. These ensemble-methods combine individual VPR techniques and weigh their predictions using these same predictions to give an indication of their confidence. This calculation, however, is strictly based on predictions obtained from applying the VPR techniques on test data at inference time. Generally within VPR research, the dataset that was used to train the VPR technique is often different from the dataset it is tested on. This means there is a domain shift between the training and test data. This domain shift is not taken into account when weighting the predictions of VPR techniques in an ensemble using these existing methods. In this work, we analyze how this degree of domain shift between train and test data, which can be observed by looking at the relative location of descriptors in the feature space, impacts downstream VPR performance. Intuitively, one would expect better VPR performance in a situation where the degree of train-test domain shift is minimal. Our analysis shows that this is indeed the case. We propose two different methods that utilize this degree of domain shift to calculate the weights given to the VPR techniques in an ensemble. First, we propose a generative method. Here weights are given to the VPR techniques based on the likelihood that the query sample originated from the same distribution as the training dataset of the technique and is in distribution. This way each individual technique is given a weight. Secondly, we propose a discriminative method. Here weights are given to the training datasets used to train the techniques in the ensemble. These training datasets are given weights based on relative proximity to a query sample in the feature space, an indicator for the degree of domain-shift between the training dataset and the query sample. all VPR techniques are given the weight corresponding to their training dataset. We compare these proposed approaches to other ensemble-based baselines and individual VPR techniques. The quantitative results show that our proposed methods generally outperform the ensemblebased baselines and the individual VPR techniques. We also propose further future work. One of the generative methods still delivers lower performance than could be possible, caused by applying this method to high-dimensionality descriptors. A solution for this issue should lead to higher VPR performance using this method. Additionally, we suggest future work to expand on the datasets used in this research, to strengthen the claims made, and verify that results and trends found to hold up when testing and training using other datasets.