This paper presents an analysis of the data and compute efficiency of the TemporalMaxer deep learning model in the context of temporal action localization (TAL), which involves accurately detecting the start and end times of specific video actions. The study explores the performance and scalability of the TemporalMaxer model under limited resources and data availability, focusing on factors such as hardware requirements, training time, and data utilization, thus contributing to the advancement of efficient deep learning models for real-world video tasks. Through a literature review of temporal action recognition models, evaluation of learning curves for data efficiency, and development of metrics to assess the compute efficiency, the study provides insights into the performance trade-offs of the TemporalMaxer model. Experiments conducted on the widely used THUMOS dataset further demonstrate the model's generalizability with limited data, achieving significant accuracy performance with only 50% of the training data. Notably, TemporalMaxer exhibits superior compute efficiency by significantly reducing the number of Multiply-Accumulate operations (MACs) compared to other state-of-the-art models. However, alternative models like TriDet and TadTR outperform TemporalMaxer in training time-constrained scenarios. These findings shed light on the model's practical applicability in resource-constrained environments, offering insights for further optimization and study.

In temporal action localization, given an input video, the goal is to predict the action that is present in the video, along with its temporal boundaries. Several powerful models have been proposed throughout the years, with transformer-based models achieving state-of-the-art performance in the recent months. Although novel models are becoming more and more accurate, authors rarely study how limited training data or computation power environments affect the performance of their model. This study is carried out on TriDet, a transformer-based temporal action localization model that achieves state-of-the-art performance on two different benchmarks. It evaluates the model’s behavior in a limited training data and computation power environment. It is found that TriDet achieves close to state-of-the-art performance when only 60% of the training data or approximately 90 action instances per class are used. It is also notable that inference time, memory usage, multiply-accumulate operations and GPU utilization scale linearly along with the length of the tensor that is passed to the model. These findings, combined with TriDet’s mean training time of 11 minutes on the THUMOS’14 dataset can be used to determine the model’s hypothetical behavior when run in lower computation power environments.

In temporal action localization, given an input video, the goal is to predict which actions it contains, where they begin and where they end. Training and testing current state-of-the-art, deep learning models is done assuming access to large amounts of data and computational power. Gathering such data is however a challenging task and access to computational resources might be limited. This work thus explores and measures how well one of such deep learning models, ActionFormer, performs in settings constrained by the amount of data or computational power. Data efficiency was measured by training the model on a subset of the training set and testing on the test set. Although ActionFormer showed promising results on both THUMOS'14 and ActivityNet datasets, TriDet and TemporalMaxer models should likely be chosen in favor of ActionFormer in limited data settings as they exhibit better data efficiency. Similarly, the TriDet model should be chosen in favor of ActionFormer in cases where the training time is limited, as it showed better computational efficiency during training. To test the efficiency of the model during inference, videos of different lengths were passed through the model. Most importantly, we find that both the inference time and the memory usage of the model scale linearly with input video length, as predicted by the authors of the ActionFormer.

Temporal Action Localization (TAL) aims to localize the start and end times of actions in untrimmed videos and classify the corresponding action types. TAL plays an important role in understanding video. Existing TAL approaches heavily rely on deep learning and require large-scale data and expensive training processes. Recent advances in Contrastive Language-Image Pre-Training (CLIP) have brought vision-language modeling into the field of TAL. While current CLIP-based TAL methods have been proven to be effective, their capabilities under data and compute-limited settings are not explored. In this paper, we have investigated the data and compute efficiencies of the CLIP-based STALE model. We evaluate the model performances under data-limited open/close-set scenarios. We find that STALE can demonstrate adequate generalizability using limited data. We experimented with the training time, inference time, GPU utilization, MACs, and memory consumption of STALE by inputting with varying video lengths. We discover an optimal input length for STALE to inference. Using model quantization, we find a significant forward time reduction for STALE on a single CPU. Our findings shed light on the capabilities and limitations of CLIP-based TAL methods under constrained data and compute resources. The insights gained from this research contribute to enhancing the efficiency and applicability of CLIP-based TAL techniques in real-world scenarios. The results provide valuable guidance for future advancements in CLIP-based TAL models and their potential for broader adoption in resource-constrained environments.

An algal bloom is defined as a rapid increase in common algae (phytoplankton) abundance in water bodies and it can occur when a group of certain environmental factors is combined. If the algae populations grow out of control, such algal blooms become problematic and cause damage to the ecosystem, such phenomena are called harmful algal blooms. For this reason, it is important to detect and forecast these phenomena to be able to take action beforehand. Remote sensing is measuring and monitoring the characteristics of an area at a distance and it is typically done by satellites. Remote sensed data containing various environmental measurements can be used as the input for a machine learning system to estimate chlorophyll-a concentrations which is the main indicator used for detecting algal blooms. The main question this research aims to answer is: Which input modality is the most predictive for estimating chlorophyll-a concentrations for water bodies in Uruguay? This research presents the step-by-step construction of a system to pre-process the environmental data collected through remote sensing and use this data to train and test a machine learning system to assess and compare 11 different environmental factors or so-called data modalities individually against each other to find out the most predictive one. Carrying out the machine learning experiments brings the results into the open that radiation mean and turbidity of water are the two most predictive data modalities for algal bloom forecasting with accuracy scores of approximately 34%, while radiation mean is performing slightly better.

The term ”Algal Bloom” refers to the accumulation of algae in a confined geological space. They may harm human health and negatively affect ecological systems around the area. Thus, forecasting algal blooms could mitigate the environmental and socio-economical damages. Particularly, the use of deep learning methods could distinguish underlying patterns such as spatio-temporal dependencies in the available remote sensing data of environmental factors that might cause algal blooms, such as change in water temperature. This research paper will aim to answer the following research question: Does the inclusion of explicit spatio-temporal embedding methods display a significant improvement for predicting algal blooms? The paper will use the UNet architecture and further encodes spatial and temporal information to be explicitly included as features in the training and validation process of deep learning models. The results of the experiment show that the inclusion of explicit spatio-temporal information separately into the feature space exhibits a small increase in performance. However, the combination of spatio-temporal information does not display a significant improvement for the predictions.

Forecasting algal blooms using remote sensing data is less labour-intensive and has better cover- age in time and space than direct water sampling. The paper implements a deep learning technique, the UNet Architecture, to predict the chlorophyll concentration, which is a good indicator for al- gal bloom in the Rio Negro water reservoirs of Uruguay. The research question focuses on the dif- ferences between classification and regression in algal bloom forecasting. The experiments show that the regression implementation achieves bet- ter accuracy and lower mean squared error than the classification implementation that uses cross- entropy loss and four pre-fixed bins. Different loss functions that account for the class imbalance in the data do not improve the model’s performance. Fi- nally, a quantile-based binning strategy that consid- ers the data’s underlying distribution achieves the highest accuracy in both settings.

This research presents a method for forecasting algal blooms using remote sensing with spatially and temporally sparse satellite data. The method involves the use of multiple interpolation methods to interpolate the sparse input data. The approach is shown to be effective in predicting algal blooms in areas where data is sparse, and the results demonstrate the potential for using this method to improve the forecasting and management of harmful algal blooms.

Color information has been shown to provide useful information during image classification. Yet current popular deep convolutional neural networks use 2-dimensional convolutional layers. The first 2-dimensional convolutional layer in the network combines the color channels of the input images, which produces feature maps per channel with only spatial dimensions, height and width, getting rid of the color dimension. In this work we introduce Full Color Deep networks which use 3-dimensional convolutions to retain the color dimension beyond the first layer. The 3D kernels convolve over the color and spatial dimensions. The network can extract features from all three dimensions in all layers which are subsequently used by the classifier. We show that the Full Color Deep networks perform at least as well as the current CNNs but outperform them in learning color information and using that information in other downstream tasks.

While deep neural networks show great potential for being part of safety-critical applications such as autonomous driving, covering their sensitivity to illumination shifts by adding training data is of- ten non-trivial. The undesired illumination shift between train and test data can be addressed by domain adaptation methods. Recent work [9] has demonstrated performance improvements with a novel zero-shot domain adaptation setting by in- troducing a physics-based visual inductive prior - a trainable Color Invariant Convolution (CIConv) layer - aiming to transform its input to a more do- main invariant representation.
We compare the performance of image classifica- tion for day-night domain adaptation in the zero- shot and the unsupervised setting, and explore the effectiveness of using CIConv in both settings. We show that unsupervised domain adaptation reduces the day-night distribution shift similarly to CIConv in the zero-shot setting. We demonstrate improved performance when CIConv and unsupervised day- night domain adaptation are combined.

Color Invariant Convolution (CIConv) is a learnable Convolutional Neural Network (CNN) layer that reduces the distribution shift between the source and target set in the CNN under an illumination-based domain shift. We explore the semantic segmentation performance for daynight domain adaptation when using CIConv. We will test this on two settings: one with only labeled train data available and one with access to both labeled training data and unlabeled test data. In both settings, we will cast an invariant edge detector as a trainable CIConv layer in the CNN to transform the daytime dataset to a domain invariant representation. We will execute day-night domain adaptation and evaluate the mean Intersection over Union over the results. We compare this result to the vanilla version of the same code without using the invariant edge detector as a trainable layer. We will discuss the results obtained from our experiments and show that the trainable CIConv layer does not always result in better outcomes for day-night domain adaptation.

CycleGANs [1] and CIConv [2] are both relatively new approaches to their respective applications. For CycleGANs this application is unpaired image-to-image domain adaptation and for CIConv this application is making images more
robust to illumination changes. We investigate whether CycleGANs in combination with CIConv can be used to improve the day-night domain adaptation. The resulting images could then be used during the training of CNNs that can be found in self-driving cars. Attempts were made to get the CycleGANs in combination with CIConv to train in a stable manner. These attempts included a variety of hyperparameter combinations, a number of architecture alterations and training procedure adjustments, and most significantly two different loss functions. Both these loss functions apply a CycleConsistency Loss, one applies an additional Adversarial Loss [1] and the other an additional Wasserstein Distance and Gradient Penalty [3]. In this paper we show that CycleGANs with CIConv as the first layer in either the Discriminators or the Generators resulted in unstable training. We conclude that the root of the instability issues lies in the CIConv layer causing exploding gradients resulting in unsuccessful training of the model. Finally, we propose an adjustment to the CIConv layer which shows promise in resolving these issues for the architecture with CIConv in the Generators. However, no extensive testing has been done.

The possibility to improve an existing method by making (part of) it learnable is explored in this research. The work that this research extends added prior knowledge to a Convolutional Neural Network (CNN) to improve its performance when dealing with an illumination shift. The method used for the preprocessing, is the color invariant. The method was used in a zero-domain adaptation setting, where the network is trained without having access to the target domain. The research demonstrated improved performance, motivating further improvements.

The Color Invariant Convolution (CIConv) layer implements the color invariant edge detectors. The layer converts the RGB input of each pixel to spectral differential quotients, which are used to determine the color invariant representation. This is done through two fixed linear transformations that only approximate these values. This indicates that an even better approximation can be obtained by making this transition learnable.

Two methods are used to make this transition learnable; a linear learning method and a non-linear learning method. The linear learning method uses the original transformation but allows for change and the non-linear method replaces the linear transform with a neural network. Both methods show potential for achieving better results than the fixed transformation, but only the linear learning method actually does perform better in a classification experiment. All experiments are done following the zero-shot day to night domain adaptation on a synthetic dataset.

To alleviate lower classification performance on rare classes in imbalanced datasets, a possible solution is to augment the underrepresented classes with synthetic samples. Domain adaptation can be incorporated in a classifier to decrease the domain discrepancy between real and synthetic samples. While domain adaptation is generally applied on completely synthetic source domains and real target domains, we explore how domain adaptation can be applied when only a single rare class is augmented with simulated samples. As a testbed, we use a camera trap animal dataset with a rare deer class, which is augmented with synthetic deer samples. We adapt existing domain adaptation methods to two new methods for the single rare class setting: DeerDANN, based on the Domain-Adversarial Neural Network (DANN), and DeerCORAL, based on deep correlation alignment (Deep CORAL) architectures. Experiments show that DeerDANN has the highest improvement in deer classification accuracy of 24.0% versus 22.4% improvement of DeerCORAL when compared to the baseline. Further, both methods require fewer than 10k synthetic samples, as used by the baseline, to achieve these higher accuracies. DeerCORAL requires the least number of synthetic samples (2k deer), followed by DeerDANN (8k deer).

In the field of ecology, camera traps are important tools to collect information on the wildlife of certain areas. The problem that arises with many camera traps is that they can collect more images than a human can realistically go trough all by themselves. To help classify these images computer vision is proposed as an alternative to manual classification. Many modern computer vision applications use neural networks. A hard part for the neural networks is that to train them well a large data set is needed, and sometimes it is almost impossible to build this dataset. This is where synthetic samples can be used instead of real samples. These samples are created by using computer graphics software to create realistic looking images to enlarge the dataset, or even be the whole dataset. This work evaluates how well a segmentation network was trained on only synthetic samples could perform on the real data. For this multiple segmentation networks were used like: U-Net and SegNet and the networks were trained on different datasets all derived from the synthetic data. The results show that while the networks can work real images that look similar to the synthetic samples, they fail to segment images that are captured in locations that look different from the synthetic samples.

The natural world is long-tailed: rare classes are observed orders of magnitudes less frequently than common ones, leading to highly-imbalanced data where rare classes can have only handfuls of examples. Learning from few examples is a known challenge for deep learning based classification algorithms, and is the focus of the field of low-shot learning. One potential approach to increase the training data for these rare classes is to augment the limited real data with synthetic samples. This has been shown to help, but the domain shift between real and synthetic hinders the approaches' efficacy when tested on real data. We explore the use of image-to-image translation methods to close the domain gap between synthetic and real imagery for animal species classification in data collected from camera traps: motion-activated static cameras used to monitor wildlife. We use low-level feature alignment between source and target domains to make synthetic data for a rare species generated using a graphics engine more "realistic". Compared against a system augmented with unaligned synthetic data, our experiments show a considerable decrease in classification error rates on a rare species.

Camera traps are used around the world to provide data on species, population sizes and how species are interacting. However this creates a lot of work in identifying which animal was actually spotted near the camera. Attempts have been made to use deep-learning to identify animals and work correctly for animals which are not rare but the lack of training data of rare species is a hurdle yet to be overcome. This research is focused how well the MIT Data-Efficient Generative Adversarial Network or MITGAN for short can generate realistic samples to be used as training data. For this we use a modified version of the CCT20 data-set. Which has artificially made a single rare class: the deer class. We trained a MITGAN model to generate images of our artificial rare class and used these images to train a classification model. This model was then compared to a baseline model as well as an oversampled model. From this it follows that using the MITGAN model to generate extra samples for our rare class is not worth the effort it takes compared to the precision increase in the rare class.

Wheat is a widely used ingredient for food products. To increase the productionand quality of wheat, the density of ’wheat heads’ in a farm can be studied. Accuratelylocating wheat heads in images can be challenging. A lot of work has taken place insupervised semantic segmentation, but these networks typically require large pixel-wisehuman-annotated labeled data. Gathering this data is tedious and labour intensive.This paper proposes to use the novel unsupervised semantic segmentation model W-Net to solve this problem. To improve the accuracy, we investigated the influence ofthe frequency domain, by pre-processing the training data two different times using acustom filter, based on frequencies found in wheat heads, and a high pass filter.The approach is evaluated on the Global Wheat Head Detection (GWHD) dataset[11]. To compare the accuracy the generated segmentations were mapped to boundingboxes based. The proposed method did not show to be able to generate competing de-tection compared to the baseline method associated with the GWHD dataset, but theGWHD dataset has a different measurement of truth, consisting out bounding boxesinstead of segments which is in the disadvantage for the W-Net.Pre-processing the dataset using the high pass filter did increase the intersection overunion with 1,4% and the deviation of the reconstruction loss was smaller when fre-quency filtering was applied.Although the object detection has a low accuracy, this study showed that some ba-sic wheat head detection can be achieved by using the unsupervised segmentationmethod W-Net and the accuracy can be increased if a high pass filter is applied aspre-processing step.

This research paper analyses the effect that using frequency information can have on object detectors. The latter are complex networks that learn information about objects from images and are then able to predict the location of these objects in new, unseen images. There are, however, certain datasets that are hard to learn on, partly because the environment in which images are taken is diverse and complex, and also because the objects to detect can appear in fairly different shapes. The dataset considered in this paper is called the Global Wheat Head Dataset (GWHD, provided by a Kaggle competition). An object detector is run on the original GWHD images and then the performance is compared to running the detector on a frequency filtered version of the images. A mathematical transform called Fourier Transform is used to map images from their spatial (pixel) domain to a new domain called the frequency domain, where certain non-informative frequencies are filtered out and then the images are mapped back to their spatial domain. Two experiments were conducted and results show that with this specific filtering methodology, no improvement is found on the GWHD dataset using an object detector called YoloV5. A pipeline was developed which allows for custom filtering strategy implementations and customs datasets. Similar work has shown that images in their frequency domain can speed up computational time and also increase the accuracy of an object detector, so this paper also gives the opportunity for further experiments with the created pipeline.

In this paper, we run two methods of explanation, namely LIME and Grad-CAM, on a convolutional neural network trained to label images with the LEGO bricks that are visible in them. We evaluate them on two criteria, the improvement of the network's core performance and the trust they are able to generate for users of the system. We nd that in general, Grad-CAM seems to outperform LIME on this specic task: it yields more detailed insight from the point of view of core performance and 80% of respondents asked to choose between them when it comes to the trust they inspire in the model choose Grad-CAM. However, we also posit that it is more useful to employ these two methods together, as the insights they yield are complementary.