Computer vision systems, such as image classifiers, object detectors and video analysis tools, serve diverse applications, ranging from autonomous vehicles and drone navigation to medical image analysis and anomaly inspection in the manufacturing industry. The development of these systems relies heavily on well established practices, which include the adoption of conventional training and evaluation metrics and benchmark datasets. However, we argue that standard approaches are sub-optimal with respect to the ultimate objectives of the computer vision systems. In this thesis, we question whether the training and evaluation of computer vision systems for object detection and long-term action recognition are typically aligned with human-defined end goals.

Object detectors are deployed for object tracking in autonomous vehicles and drones, but also as user assistive tools in medical image analysis and anomaly inspection in industry. Regardless of the end use, object detectors are trained with standard optimization and evaluation strategies. By investigating whether the optimization and evaluation methods of object detectors correlate with human quality judgments, we discover a discrepancy between established metrics and human preferences. To address this, we propose an alternative training loss that better aligns object detectors with human preference.

Subsequently, we ask whether object detections can be used to improve longterm human action recognition in videos. We find that explicitly focusing on the region containing the detected human is beneficial to long-term action recognition models. Unexpectedly, we also find that including a temporal attention module does not help recognizing the videos. Motivated by this result, we investigate how much temporal information is needed to solve long-term action recognition in three popular video datasets. Our results show that most of these videos can be recognized without any long-term temporal information. This suggests that models trained on these videos might exploit short-term shortcuts, instead of learning long-term temporal dependencies. Importantly, these models would not perform successfully on new videos where long-term reasoning is necessary.

As a follow-up, we investigate the impact of the temporal receptive field in longterm action recognition models. The size of the temporal receptive field determines the capability to encode long-term information in videos, like the actions order and duration. We experimentally verify that large temporal receptive fields are sensitive to order and can overfit on the exact action orders seen at training time. Contrarily, short temporal receptive fields are more robust to order permutations and perform better on a current long-term video dataset. This result further demonstrates the irrelevance of long-term information in current long-term action recognition datasets. Our research findings highlight the importance of using training and evaluation metrics that match the intended use of the computer vision systems and choosing training and evaluation datasets that carefully represent the problem at hand. ... Read more

In temporal action localization, given an input video, the goal is to predict which actions it contains, where they begin, and where they end. Training and testing current state-of- the-art deep learning models requires access to large amounts of data and computational power. However, gathering such data is challenging and computational resources might be limited. This work explores and measures how current deep temporal action localization models perform in settings constrained by the amount of data or computational power. We measure data efficiency by training each model on a subset of the training set. We find that TemporalMaxer outperforms other models in data-limited settings. Furthermore, we recommend TriDet when training time is limited. To test the efficiency of the models during inference, we pass videos of different lengths through each model. We find that TemporalMaxer requires the least computational resources, likely due to its simple architecture. ... Read more

Previous work on long-term video action recognition relies on deep 3D-convolutional models that have a large temporal receptive field (RF). We argue that these models are not always the best choice for temporal modeling in videos. A large temporal receptive field allows the model to encode the exact sub-action order of a video, which causes a performance decrease when testing videos have a different sub-action order. In this work, we investigate whether we can improve the model robustness to the sub-action order by shrinking the temporal receptive field of action recognition models. For this, we design Video BagNet, a variant of the 3D ResNet-50 model with the temporal receptive field size limited to 1, 9, 17 or 33 frames. We analyze Video Bag-Net on synthetic and real-world video datasets and experimentally compare models with varying temporal receptive fields. We find that short receptive fields are robust to sub-action order changes, while larger temporal receptive fields are sensitive to the sub-action order. ... Read more

Color is a crucial visual cue readily exploited by Convolutional Neural Networks (CNNs) for object recognition. However, CNNs struggle if there is data imbalance between color variations introduced by accidental recording conditions. Color invariance addresses this issue but does so at the cost of removing all color information, which sacrifices discriminative power. In this paper, we propose Color Equivariant Convolutions (CEConvs), a novel deep learning building block that enables shape feature sharing across the color spectrum while retaining important color information. We extend the notion of equivariance from geometric to photometric transformations by incorporating parameter sharing over hue-shifts in a neural network. We demonstrate the benefits of CEConvs in terms of downstream performance to various tasks and improved robustness to color changes, including train-test distribution shifts. Our approach can be seamlessly integrated into existing architectures, such as ResNets, and offers a promising solution for addressing color-based domain shifts in CNNs. ... Read more

Many real-world applications, from sport analysis to surveillance, benefit from automatic long-term action recognition. In the current deep learning paradigm for automatic action recognition, it is imperative that models are trained and tested on datasets and tasks that evaluate if such models actually learn and reason over long-term information. In this work, we propose a method to evaluate how suitable a video dataset is to evaluate models for long-term action recognition. To this end, we define a long-term action as excluding all the videos that can be correctly recognized using solely short-term information. We test this definition on existing long-term classification tasks on three popular real-world datasets, namely Breakfast, CrossTask and LVU, to determine if these datasets are truly evaluating long-term recognition. Our study reveals that these datasets can be effectively solved using shortcuts based on short-term information. Following this finding, we encourage long-term action recognition researchers to make use of datasets that need long-term information to be solved. ... Read more

The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient. ... Read more

Long-Term activities involve humans performing complex, minutes-long actions. Differently than in traditional action recognition, complex activities are normally composed of a set of sub-actions, that can appear in different order, duration, and quantity. These aspects introduce a large intra-class variability, that can be hard to model. Our approach aims to adaptively capture and learn the importance of spatial and temporal video regions for minutes-long activity classification. Inspired by previous work on Region Attention, our architecture embeds the spatio-temporal features from multiple video regions into a compact fixed-length representation. These features are extracted with a 3D convolutional backbone specially fine-tuned. Additionally, driven by the prior assumption that the most discriminative locations in the videos are centered around the human that is carrying out the activity, we introduce an Actor Focus mechanism to enhance the feature extraction both in training and inference phase. Our experiments show that the Multi-Regional fine-tuned 3D-CNN, topped with Actor Focus and Region Attention, largely improves the performance of baseline 3D architectures, achieving state-of-the-art results on Breakfast, a well known long-term activity recognition benchmark. ... Read more