In temporal action localization, given an input video, the goal is to predict which actions it contains, where they begin and where they end. Training and testing current state-of-the-art, deep learning models is done assuming access to large amounts of data and computational power. Gathering such data is however a challenging task and access to computational resources might be limited. This work thus explores and measures how well one of such deep learning models, ActionFormer, performs in settings constrained by the amount of data or computational power. Data efficiency was measured by training the model on a subset of the training set and testing on the test set. Although ActionFormer showed promising results on both THUMOS'14 and ActivityNet datasets, TriDet and TemporalMaxer models should likely be chosen in favor of ActionFormer in limited data settings as they exhibit better data efficiency. Similarly, the TriDet model should be chosen in favor of ActionFormer in cases where the training time is limited, as it showed better computational efficiency during training. To test the efficiency of the model during inference, videos of different lengths were passed through the model. Most importantly, we find that both the inference time and the memory usage of the model scale linearly with input video length, as predicted by the authors of the ActionFormer.

In temporal action localization, given an input video, the goal is to predict the action that is present in the video, along with its temporal boundaries. Several powerful models have been proposed throughout the years, with transformer-based models achieving state-of-the-art performance in the recent months. Although novel models are becoming more and more accurate, authors rarely study how limited training data or computation power environments affect the performance of their model. This study is carried out on TriDet, a transformer-based temporal action localization model that achieves state-of-the-art performance on two different benchmarks. It evaluates the model’s behavior in a limited training data and computation power environment. It is found that TriDet achieves close to state-of-the-art performance when only 60% of the training data or approximately 90 action instances per class are used. It is also notable that inference time, memory usage, multiply-accumulate operations and GPU utilization scale linearly along with the length of the tensor that is passed to the model. These findings, combined with TriDet’s mean training time of 11 minutes on the THUMOS’14 dataset can be used to determine the model’s hypothetical behavior when run in lower computation power environments.

The aim of this paper is to find out which Machine Learning (ML) model predicts the concentration of Chlorophyll-a, in the Palmar lake in Uruguay best. Currently there are no such models to predict the growth in this lake. The algorithms which will be compared in this paper are a Linear Regression model and the U-Net model. We will compare the losses of the two models to determine which algorithm performs best. The less loss a model has, the more accurate it is, and thus the better it is. The loss of the U-Net model failed to converge to a value, therefore it was impossible to compare the two models.

Training Convolutional Neural Network (CNN) models is difficult when there is a lack of labeled training data and no unlabeled data is available. A popular method for this is domain adaptation where the weights of a pre-trained CNN model are transferred to the problem setup. The model is pre-trained on the same task but in a different domain that has plenty of labeled data samples available. In a CNN model, we can rearrange the weights of a convolutional layer by permuting them along the input channel dimension. This work shows that certain weights that are learned in the pre-trained model work well in the problem setup when the weights are rearranged in this manner. Computing the set of all possible rearrangements of the weights is computationally intractable. This work proposes two algorithms to find a good rearrangement of the weights in reasonable computation time. The solutions from the algorithms perform equally well or better than fine-tuning in the domain adaptation between SVHN and MNIST data.

An algal bloom is defined as a rapid increase in common algae (phytoplankton) abundance in water bodies and it can occur when a group of certain environmental factors is combined. If the algae populations grow out of control, such algal blooms become problematic and cause damage to the ecosystem, such phenomena are called harmful algal blooms. For this reason, it is important to detect and forecast these phenomena to be able to take action beforehand. Remote sensing is measuring and monitoring the characteristics of an area at a distance and it is typically done by satellites. Remote sensed data containing various environmental measurements can be used as the input for a machine learning system to estimate chlorophyll-a concentrations which is the main indicator used for detecting algal blooms. The main question this research aims to answer is: Which input modality is the most predictive for estimating chlorophyll-a concentrations for water bodies in Uruguay? This research presents the step-by-step construction of a system to pre-process the environmental data collected through remote sensing and use this data to train and test a machine learning system to assess and compare 11 different environmental factors or so-called data modalities individually against each other to find out the most predictive one. Carrying out the machine learning experiments brings the results into the open that radiation mean and turbidity of water are the two most predictive data modalities for algal bloom forecasting with accuracy scores of approximately 34%, while radiation mean is performing slightly better.

Temporal Action Localization (TAL) aims to localize the start and end times of actions in untrimmed videos and classify the corresponding action types. TAL plays an important role in understanding video. Existing TAL approaches heavily rely on deep learning and require large-scale data and expensive training processes. Recent advances in Contrastive Language-Image Pre-Training (CLIP) have brought vision-language modeling into the field of TAL. While current CLIP-based TAL methods have been proven to be effective, their capabilities under data and compute-limited settings are not explored. In this paper, we have investigated the data and compute efficiencies of the CLIP-based STALE model. We evaluate the model performances under data-limited open/close-set scenarios. We find that STALE can demonstrate adequate generalizability using limited data. We experimented with the training time, inference time, GPU utilization, MACs, and memory consumption of STALE by inputting with varying video lengths. We discover an optimal input length for STALE to inference. Using model quantization, we find a significant forward time reduction for STALE on a single CPU. Our findings shed light on the capabilities and limitations of CLIP-based TAL methods under constrained data and compute resources. The insights gained from this research contribute to enhancing the efficiency and applicability of CLIP-based TAL techniques in real-world scenarios. The results provide valuable guidance for future advancements in CLIP-based TAL models and their potential for broader adoption in resource-constrained environments.

This paper presents an analysis of the data and compute efficiency of the TemporalMaxer deep learning model in the context of temporal action localization (TAL), which involves accurately detecting the start and end times of specific video actions. The study explores the performance and scalability of the TemporalMaxer model under limited resources and data availability, focusing on factors such as hardware requirements, training time, and data utilization, thus contributing to the advancement of efficient deep learning models for real-world video tasks. Through a literature review of temporal action recognition models, evaluation of learning curves for data efficiency, and development of metrics to assess the compute efficiency, the study provides insights into the performance trade-offs of the TemporalMaxer model. Experiments conducted on the widely used THUMOS dataset further demonstrate the model's generalizability with limited data, achieving significant accuracy performance with only 50% of the training data. Notably, TemporalMaxer exhibits superior compute efficiency by significantly reducing the number of Multiply-Accumulate operations (MACs) compared to other state-of-the-art models. However, alternative models like TriDet and TadTR outperform TemporalMaxer in training time-constrained scenarios. These findings shed light on the model's practical applicability in resource-constrained environments, offering insights for further optimization and study.

Temporal Action Localization (TAL) is an important problem in computer vision with uses in video surveillance and recommendation, healthcare, entertainment, and human-computer interaction. Being an inherently data-heavy process, TAL has been bound by the availability of computing power, resulting in its slow pace of innovation. This work aims to accelerate the development of TAL models by conducting a short review of TAL's state-of-the-art, and providing extensive data about the latest models' data and compute efficiency. By researching how TAL models perform in limited data and compute settings, we find that using less data than available is often beneficial to iterating a model quickly, while in some cases, TAL is constrained by the limited amount of data. Finally, we provide general guidelines that create a simple framework for efficient TAL model development.

Forecasting algal blooms using remote sensing data is less labour-intensive and has better cover- age in time and space than direct water sampling. The paper implements a deep learning technique, the UNet Architecture, to predict the chlorophyll concentration, which is a good indicator for al- gal bloom in the Rio Negro water reservoirs of Uruguay. The research question focuses on the dif- ferences between classification and regression in algal bloom forecasting. The experiments show that the regression implementation achieves bet- ter accuracy and lower mean squared error than the classification implementation that uses cross- entropy loss and four pre-fixed bins. Different loss functions that account for the class imbalance in the data do not improve the model’s performance. Fi- nally, a quantile-based binning strategy that consid- ers the data’s underlying distribution achieves the highest accuracy in both settings.

The term ”Algal Bloom” refers to the accumulation of algae in a confined geological space. They may harm human health and negatively affect ecological systems around the area. Thus, forecasting algal blooms could mitigate the environmental and socio-economical damages. Particularly, the use of deep learning methods could distinguish underlying patterns such as spatio-temporal dependencies in the available remote sensing data of environmental factors that might cause algal blooms, such as change in water temperature. This research paper will aim to answer the following research question: Does the inclusion of explicit spatio-temporal embedding methods display a significant improvement for predicting algal blooms? The paper will use the UNet architecture and further encodes spatial and temporal information to be explicitly included as features in the training and validation process of deep learning models. The results of the experiment show that the inclusion of explicit spatio-temporal information separately into the feature space exhibits a small increase in performance. However, the combination of spatio-temporal information does not display a significant improvement for the predictions.

This research presents a method for forecasting algal blooms using remote sensing with spatially and temporally sparse satellite data. The method involves the use of multiple interpolation methods to interpolate the sparse input data. The approach is shown to be effective in predicting algal blooms in areas where data is sparse, and the results demonstrate the potential for using this method to improve the forecasting and management of harmful algal blooms.

In real-life scenarios, there are many variations in sizes of objects of the same category and the objects are not always placed at a fixed distance from the camera. This results in objects taking up an arbitrary size of pixels in the image. Vanilla CNNs are by design only translation equivariant and thus have to learn separate filters for scaled variants of the same objects. Recently, scale-equivariant approaches have been developed that share features across a set of pre-determined fixed scales. We further refer to this set of scales as the internal scales. Existing work gives little information about how to best choose the internal scales when the underlying distribution of sizes, or scale distribution, in the dataset, is known. In this work, we develop a model of how the features at different internal scales are used for samples containing differently-sized objects. The proposed model return comparable internal scales to the best-performing internal scales for different data scale distribution of various width. However, in most cases, the scale distribution is not known. Compared to previous scale-equivariant methods, we do not treat the internal scales as a fixed set but directly optimise them with regard to the loss, removing the need for prior knowledge about the data scale distribution. We parameterise the internal scales by the smallest scale which we refer to as σbasis, and the Internal Scale Range (ISR) that models the ratio between the smallest and largest scale. By varying the ISR, we learn the range of the scales the model is equivariant to. We show that our method can learn the internal scales on various data scale distributions and can better adapt the internal scales than other parameterisations. Finally, we compare our scale learning approach and other parameterisations to current State-of-the-art scale-equivariant approaches on the MNIST-Scale dataset.

Convolutional Neural Networks (CNNs) benefit from fine-grained details in high-resolution images, but these images are not always easily available as data collection can be expensive or time-consuming. Transfer learning pre-trains models on data from a related domain before fine-tuning on the main domain, and is a common strategy to deal with limited data. However, transfer learning requires a similar domain with enough available data to exist, and transferability varies from task to task. To deal with limited high-resolution data we propose resolution transfer: using low-resolution data to improve high-resolution accuracy. For resolution transfer, we use Continuous kernel CNNs (CKCNNs) that can adapt their kernel size to changes in resolution and perform well on unseen resolutions. Training CKCNNs on high-resolution images is currently significantly slower than CNNs. We lower the inference costs of CKCNNs to enable training on high-resolution data. We introduce a CKCNN parameterization that constrains the frequencies of kernels to avoid distortions when the kernel size is changed, improving resolution transfer accuracy. We improve fine-tuning with a High-Frequency Adaptation module that complements our constrained kernels. We demonstrate that CKCNNs with kernel resolution adaptation outperform CNNs for resolution transfer tasks with no fine-tuning or with limited fine-tuning data. We compare to transfer learning, and achieve competitive classification accuracy with an ImageNet pre-trained ResNet-18. Our method provides an alternative to transfer learning that uses low-resolution data to improve classification accuracy when high-resolution data is limited.

The natural world is long-tailed: rare classes are observed orders of magnitudes less frequently than common ones, leading to highly-imbalanced data where rare classes can have only handfuls of examples. Learning from few examples is a known challenge for deep learning based classification algorithms, and is the focus of the field of low-shot learning. One potential approach to increase the training data for these rare classes is to augment the limited real data with synthetic samples. This has been shown to help, but the domain shift between real and synthetic hinders the approaches' efficacy when tested on real data. We explore the use of image-to-image translation methods to close the domain gap between synthetic and real imagery for animal species classification in data collected from camera traps: motion-activated static cameras used to monitor wildlife. We use low-level feature alignment between source and target domains to make synthetic data for a rare species generated using a graphics engine more "realistic". Compared against a system augmented with unaligned synthetic data, our experiments show a considerable decrease in classification error rates on a rare species.

In the field of ecology, camera traps are important tools to collect information on the wildlife of certain areas. The problem that arises with many camera traps is that they can collect more images than a human can realistically go trough all by themselves. To help classify these images computer vision is proposed as an alternative to manual classification. Many modern computer vision applications use neural networks. A hard part for the neural networks is that to train them well a large data set is needed, and sometimes it is almost impossible to build this dataset. This is where synthetic samples can be used instead of real samples. These samples are created by using computer graphics software to create realistic looking images to enlarge the dataset, or even be the whole dataset. This work evaluates how well a segmentation network was trained on only synthetic samples could perform on the real data. For this multiple segmentation networks were used like: U-Net and SegNet and the networks were trained on different datasets all derived from the synthetic data. The results show that while the networks can work real images that look similar to the synthetic samples, they fail to segment images that are captured in locations that look different from the synthetic samples.

Camera traps are used around the world to provide data on species, population sizes and how species are interacting. However this creates a lot of work in identifying which animal was actually spotted near the camera. Attempts have been made to use deep-learning to identify animals and work correctly for animals which are not rare but the lack of training data of rare species is a hurdle yet to be overcome. This research is focused how well the MIT Data-Efficient Generative Adversarial Network or MITGAN for short can generate realistic samples to be used as training data. For this we use a modified version of the CCT20 data-set. Which has artificially made a single rare class: the deer class. We trained a MITGAN model to generate images of our artificial rare class and used these images to train a classification model. This model was then compared to a baseline model as well as an oversampled model. From this it follows that using the MITGAN model to generate extra samples for our rare class is not worth the effort it takes compared to the precision increase in the rare class.

To alleviate lower classification performance on rare classes in imbalanced datasets, a possible solution is to augment the underrepresented classes with synthetic samples. Domain adaptation can be incorporated in a classifier to decrease the domain discrepancy between real and synthetic samples. While domain adaptation is generally applied on completely synthetic source domains and real target domains, we explore how domain adaptation can be applied when only a single rare class is augmented with simulated samples. As a testbed, we use a camera trap animal dataset with a rare deer class, which is augmented with synthetic deer samples. We adapt existing domain adaptation methods to two new methods for the single rare class setting: DeerDANN, based on the Domain-Adversarial Neural Network (DANN), and DeerCORAL, based on deep correlation alignment (Deep CORAL) architectures. Experiments show that DeerDANN has the highest improvement in deer classification accuracy of 24.0% versus 22.4% improvement of DeerCORAL when compared to the baseline. Further, both methods require fewer than 10k synthetic samples, as used by the baseline, to achieve these higher accuracies. DeerCORAL requires the least number of synthetic samples (2k deer), followed by DeerDANN (8k deer).

Aside from developing methods to embed the equivariant priors into the architectures, one can also study how the networks learn equivariant properties. In this work, we conduct a study on the influence of different factors on learned equivariance. We propose a method to quantify equivariance and argue why using the correlation to compare intermediate representations may be a better choice as opposed to other commonly used metrics. We show that imposing equivariance or invariance into the objective function does not influence learning more equivariant features in the early parts of the network. We also study how different data augmentations influence translation equivariance. Furthermore, we show that models with lower capacity learn more translation equivariant features. Lastly, we quantify translation and rotation equivariance in different state-of-the-art image classification models and analyse the correlation between the amount of equivariance and accuracy.

Location information is essential for the ViT model. Image data has three types of location information: absolute location, relative direction, and relative distance. Various position embeddings methods have been used to introduce location information to the ViT model. Some existing methods are absolute position embeddings, relative position embeddings, fixed sinusoidal position embeddings, and learnable Fourier position embeddings. However, it is unclear what type of location information can be encoded by different position embeddings methods. This paper investigates this question by conducting fully-controlled experiments and feature-level analysis on synthetic datasets. The results suggest that the relative position embeddings cannot encode absolute location information, which leads to inferior performance. All the position embeddings approaches that we test can encode relative location information. However, they have different levels of relative location bias. The learnable absolute position embeddings do not contain any relative location bias and therefore need more data to learn. The fixed sinusoidal and learnable Fourier position embeddings are relatively better, but they also have minor drawbacks. The fixed sinusoidal position embeddings are not trainable, while the Fourier method does not have much bias on relative location information. We propose to make the fixed sinusoidal position embeddings learnable and use pretraining tasks to improve the Fourier method. Our two new approaches show promising results on the testing datasets, and they are competitive compared with a similar approach.