In temporal action localization, given an input video, the goal is to predict which actions it contains, where they begin and where they end. Training and testing current state-of-the-art, deep learning models is done assuming access to large amounts of data and computational power. Gathering such data is however a challenging task and access to computational resources might be limited. This work thus explores and measures how well one of such deep learning models, ActionFormer, performs in settings constrained by the amount of data or computational power. Data efficiency was measured by training the model on a subset of the training set and testing on the test set. Although ActionFormer showed promising results on both THUMOS'14 and ActivityNet datasets, TriDet and TemporalMaxer models should likely be chosen in favor of ActionFormer in limited data settings as they exhibit better data efficiency. Similarly, the TriDet model should be chosen in favor of ActionFormer in cases where the training time is limited, as it showed better computational efficiency during training. To test the efficiency of the model during inference, videos of different lengths were passed through the model. Most importantly, we find that both the inference time and the memory usage of the model scale linearly with input video length, as predicted by the authors of the ActionFormer.

In temporal action localization, given an input video, the goal is to predict the action that is present in the video, along with its temporal boundaries. Several powerful models have been proposed throughout the years, with transformer-based models achieving state-of-the-art performance in the recent months. Although novel models are becoming more and more accurate, authors rarely study how limited training data or computation power environments affect the performance of their model. This study is carried out on TriDet, a transformer-based temporal action localization model that achieves state-of-the-art performance on two different benchmarks. It evaluates the model’s behavior in a limited training data and computation power environment. It is found that TriDet achieves close to state-of-the-art performance when only 60% of the training data or approximately 90 action instances per class are used. It is also notable that inference time, memory usage, multiply-accumulate operations and GPU utilization scale linearly along with the length of the tensor that is passed to the model. These findings, combined with TriDet’s mean training time of 11 minutes on the THUMOS’14 dataset can be used to determine the model’s hypothetical behavior when run in lower computation power environments.

An algal bloom is defined as a rapid increase in common algae (phytoplankton) abundance in water bodies and it can occur when a group of certain environmental factors is combined. If the algae populations grow out of control, such algal blooms become problematic and cause damage to the ecosystem, such phenomena are called harmful algal blooms. For this reason, it is important to detect and forecast these phenomena to be able to take action beforehand. Remote sensing is measuring and monitoring the characteristics of an area at a distance and it is typically done by satellites. Remote sensed data containing various environmental measurements can be used as the input for a machine learning system to estimate chlorophyll-a concentrations which is the main indicator used for detecting algal blooms. The main question this research aims to answer is: Which input modality is the most predictive for estimating chlorophyll-a concentrations for water bodies in Uruguay? This research presents the step-by-step construction of a system to pre-process the environmental data collected through remote sensing and use this data to train and test a machine learning system to assess and compare 11 different environmental factors or so-called data modalities individually against each other to find out the most predictive one. Carrying out the machine learning experiments brings the results into the open that radiation mean and turbidity of water are the two most predictive data modalities for algal bloom forecasting with accuracy scores of approximately 34%, while radiation mean is performing slightly better.

Temporal Action Localization (TAL) aims to localize the start and end times of actions in untrimmed videos and classify the corresponding action types. TAL plays an important role in understanding video. Existing TAL approaches heavily rely on deep learning and require large-scale data and expensive training processes. Recent advances in Contrastive Language-Image Pre-Training (CLIP) have brought vision-language modeling into the field of TAL. While current CLIP-based TAL methods have been proven to be effective, their capabilities under data and compute-limited settings are not explored. In this paper, we have investigated the data and compute efficiencies of the CLIP-based STALE model. We evaluate the model performances under data-limited open/close-set scenarios. We find that STALE can demonstrate adequate generalizability using limited data. We experimented with the training time, inference time, GPU utilization, MACs, and memory consumption of STALE by inputting with varying video lengths. We discover an optimal input length for STALE to inference. Using model quantization, we find a significant forward time reduction for STALE on a single CPU. Our findings shed light on the capabilities and limitations of CLIP-based TAL methods under constrained data and compute resources. The insights gained from this research contribute to enhancing the efficiency and applicability of CLIP-based TAL techniques in real-world scenarios. The results provide valuable guidance for future advancements in CLIP-based TAL models and their potential for broader adoption in resource-constrained environments.

This paper presents an analysis of the data and compute efficiency of the TemporalMaxer deep learning model in the context of temporal action localization (TAL), which involves accurately detecting the start and end times of specific video actions. The study explores the performance and scalability of the TemporalMaxer model under limited resources and data availability, focusing on factors such as hardware requirements, training time, and data utilization, thus contributing to the advancement of efficient deep learning models for real-world video tasks. Through a literature review of temporal action recognition models, evaluation of learning curves for data efficiency, and development of metrics to assess the compute efficiency, the study provides insights into the performance trade-offs of the TemporalMaxer model. Experiments conducted on the widely used THUMOS dataset further demonstrate the model's generalizability with limited data, achieving significant accuracy performance with only 50% of the training data. Notably, TemporalMaxer exhibits superior compute efficiency by significantly reducing the number of Multiply-Accumulate operations (MACs) compared to other state-of-the-art models. However, alternative models like TriDet and TadTR outperform TemporalMaxer in training time-constrained scenarios. These findings shed light on the model's practical applicability in resource-constrained environments, offering insights for further optimization and study.

In this work we show how Group Equivariant Convolutional Neural Networks use subsampling to learn to break equivariance to their symmetries. We focus on the 2D roto-translation group and investigate the impact of broken equivariance on network performance. We show that changing the input dimension of a network by as little as a single pixel can be enough for commonly used architectures to become approximately equivariant, rather than exactly. We investigate the impact of networks not being exactly equivariant and find that approximately equivariant networks generalise significantly worse to unseen symmetries compared to their exactly equivariant counterparts. However, when the symmetries in the training data are not identical to the symmetries of the network, we find that approximately equivariant networks are able to relax their own equivariant constraints, causing them to match or outperform exactly equivariant networks on common benchmark datasets.

Forecasting algal blooms using remote sensing data is less labour-intensive and has better cover- age in time and space than direct water sampling. The paper implements a deep learning technique, the UNet Architecture, to predict the chlorophyll concentration, which is a good indicator for al- gal bloom in the Rio Negro water reservoirs of Uruguay. The research question focuses on the dif- ferences between classification and regression in algal bloom forecasting. The experiments show that the regression implementation achieves bet- ter accuracy and lower mean squared error than the classification implementation that uses cross- entropy loss and four pre-fixed bins. Different loss functions that account for the class imbalance in the data do not improve the model’s performance. Fi- nally, a quantile-based binning strategy that consid- ers the data’s underlying distribution achieves the highest accuracy in both settings.

Data collection by means of crowdsourcing can be costly or produce inaccurate results. Methods have been proposed for solving these problems. However, it remains unclear what methods work best in scenarios with multiple similar objects of interest present in the same image, which is important for training computer vision with applications such as automatic quality control in factories. We researched which parameters are important to optimize, which methods are worth considering and what those selected methods score with regard to the parameters cost and quality. This was done through a literature review and substantiated by an experimental crowdsourcing campaign that focused on the annotation of Legos in images. It was found that the parameters to optimize were cost, optimized by reducing the time workers spent on tasks, and quality, optimized by improving the mean intersection over union value of the annotations. We concluded that majority vote, rejecting workers, majority vote adjusted to be resistant to outliers, rejecting workers with the same adjustments and decomposing tasks were the most promising methods. From our experiment we concluded that a clear trade-off exists between cost and quality. The adjusted rejecting workers method, that uses worker credibility, showed to have the highest mean quality. While the method that decomposed the components of the task and distributed them was the cheapest method to use overall and also best when looking at mean quality over cost, it was worse quality wise. These results were similar to the expected performance of the methods. From this we concluded that the best method for crowdsourcing is dependent on the error tolerance of the computer vision model that will be used and the budget available.

Data collection and annotation have proven to be a bottleneck for computer vision applications. When faced with the task of data creation, alternative methods to traditional data collection should be considered, as time and cost may be reduced signif- icantly. We introduce three novel datasets for multi- label classification purposes on LEGO bricks: a traditionally collected dataset, a rendered dataset, and a dataset with pasted cutouts of LEGO bricks. We investigate the accuracy of a ResNet classi- fier tested on real data, but trained on the different datasets. This research seeks to provide both in- sight into future dataset creation of LEGO bricks, as well as act as an advisor for general multi-label dataset creation. Our findings indicate that the tra- ditionally collected dataset is prone to overfitting due to speedups used during collection, with 90% accuracy during training but 19% during testing. Furthermore, synthetic data techniques are appli- cable to multi-label LEGO classification but need improvement, with accuracy ranging from 34% to 45%.

The term ”Algal Bloom” refers to the accumulation of algae in a confined geological space. They may harm human health and negatively affect ecological systems around the area. Thus, forecasting algal blooms could mitigate the environmental and socio-economical damages. Particularly, the use of deep learning methods could distinguish underlying patterns such as spatio-temporal dependencies in the available remote sensing data of environmental factors that might cause algal blooms, such as change in water temperature. This research paper will aim to answer the following research question: Does the inclusion of explicit spatio-temporal embedding methods display a significant improvement for predicting algal blooms? The paper will use the UNet architecture and further encodes spatial and temporal information to be explicitly included as features in the training and validation process of deep learning models. The results of the experiment show that the inclusion of explicit spatio-temporal information separately into the feature space exhibits a small increase in performance. However, the combination of spatio-temporal information does not display a significant improvement for the predictions.

This research presents a method for forecasting algal blooms using remote sensing with spatially and temporally sparse satellite data. The method involves the use of multiple interpolation methods to interpolate the sparse input data. The approach is shown to be effective in predicting algal blooms in areas where data is sparse, and the results demonstrate the potential for using this method to improve the forecasting and management of harmful algal blooms.

The natural world is long-tailed: rare classes are observed orders of magnitudes less frequently than common ones, leading to highly-imbalanced data where rare classes can have only handfuls of examples. Learning from few examples is a known challenge for deep learning based classification algorithms, and is the focus of the field of low-shot learning. One potential approach to increase the training data for these rare classes is to augment the limited real data with synthetic samples. This has been shown to help, but the domain shift between real and synthetic hinders the approaches' efficacy when tested on real data. We explore the use of image-to-image translation methods to close the domain gap between synthetic and real imagery for animal species classification in data collected from camera traps: motion-activated static cameras used to monitor wildlife. We use low-level feature alignment between source and target domains to make synthetic data for a rare species generated using a graphics engine more "realistic". Compared against a system augmented with unaligned synthetic data, our experiments show a considerable decrease in classification error rates on a rare species.

In the field of ecology, camera traps are important tools to collect information on the wildlife of certain areas. The problem that arises with many camera traps is that they can collect more images than a human can realistically go trough all by themselves. To help classify these images computer vision is proposed as an alternative to manual classification. Many modern computer vision applications use neural networks. A hard part for the neural networks is that to train them well a large data set is needed, and sometimes it is almost impossible to build this dataset. This is where synthetic samples can be used instead of real samples. These samples are created by using computer graphics software to create realistic looking images to enlarge the dataset, or even be the whole dataset. This work evaluates how well a segmentation network was trained on only synthetic samples could perform on the real data. For this multiple segmentation networks were used like: U-Net and SegNet and the networks were trained on different datasets all derived from the synthetic data. The results show that while the networks can work real images that look similar to the synthetic samples, they fail to segment images that are captured in locations that look different from the synthetic samples.

The possibility to improve an existing method by making (part of) it learnable is explored in this research. The work that this research extends added prior knowledge to a Convolutional Neural Network (CNN) to improve its performance when dealing with an illumination shift. The method used for the preprocessing, is the color invariant. The method was used in a zero-domain adaptation setting, where the network is trained without having access to the target domain. The research demonstrated improved performance, motivating further improvements. The Color Invariant Convolution (CIConv) layer implements the color invariant edge detectors. The layer converts the RGB input of each pixel to spectral differential quotients, which are used to determine the color invariant representation. This is done through two fixed linear transformations that only approximate these values. This indicates that an even better approximation can be obtained by making this transition learnable. Two methods are used to make this transition learnable; a linear learning method and a non-linear learning method. The linear learning method uses the original transformation but allows for change and the non-linear method replaces the linear transform with a neural network. Both methods show potential for achieving better results than the fixed transformation, but only the linear learning method actually does perform better in a classification experiment. All experiments are done following the zero-shot day to night domain adaptation on a synthetic dataset.

This research paper analyses the effect that using frequency information can have on object detectors. The latter are complex networks that learn information about objects from images and are then able to predict the location of these objects in new, unseen images. There are, however, certain datasets that are hard to learn on, partly because the environment in which images are taken is diverse and complex, and also because the objects to detect can appear in fairly different shapes. The dataset considered in this paper is called the Global Wheat Head Dataset (GWHD, provided by a Kaggle competition). An object detector is run on the original GWHD images and then the performance is compared to running the detector on a frequency filtered version of the images. A mathematical transform called Fourier Transform is used to map images from their spatial (pixel) domain to a new domain called the frequency domain, where certain non-informative frequencies are filtered out and then the images are mapped back to their spatial domain. Two experiments were conducted and results show that with this specific filtering methodology, no improvement is found on the GWHD dataset using an object detector called YoloV5. A pipeline was developed which allows for custom filtering strategy implementations and customs datasets. Similar work has shown that images in their frequency domain can speed up computational time and also increase the accuracy of an object detector, so this paper also gives the opportunity for further experiments with the created pipeline.

Camera traps are used around the world to provide data on species, population sizes and how species are interacting. However this creates a lot of work in identifying which animal was actually spotted near the camera. Attempts have been made to use deep-learning to identify animals and work correctly for animals which are not rare but the lack of training data of rare species is a hurdle yet to be overcome. This research is focused how well the MIT Data-Efficient Generative Adversarial Network or MITGAN for short can generate realistic samples to be used as training data. For this we use a modified version of the CCT20 data-set. Which has artificially made a single rare class: the deer class. We trained a MITGAN model to generate images of our artificial rare class and used these images to train a classification model. This model was then compared to a baseline model as well as an oversampled model. From this it follows that using the MITGAN model to generate extra samples for our rare class is not worth the effort it takes compared to the precision increase in the rare class.

Wheat is a widely used ingredient for food products. To increase the productionand quality of wheat, the density of ’wheat heads’ in a farm can be studied. Accuratelylocating wheat heads in images can be challenging. A lot of work has taken place insupervised semantic segmentation, but these networks typically require large pixel-wisehuman-annotated labeled data. Gathering this data is tedious and labour intensive.This paper proposes to use the novel unsupervised semantic segmentation model W-Net to solve this problem. To improve the accuracy, we investigated the influence ofthe frequency domain, by pre-processing the training data two different times using acustom filter, based on frequencies found in wheat heads, and a high pass filter.The approach is evaluated on the Global Wheat Head Detection (GWHD) dataset[11]. To compare the accuracy the generated segmentations were mapped to boundingboxes based. The proposed method did not show to be able to generate competing de-tection compared to the baseline method associated with the GWHD dataset, but theGWHD dataset has a different measurement of truth, consisting out bounding boxesinstead of segments which is in the disadvantage for the W-Net.Pre-processing the dataset using the high pass filter did increase the intersection overunion with 1,4% and the deviation of the reconstruction loss was smaller when fre-quency filtering was applied.Although the object detection has a low accuracy, this study showed that some ba-sic wheat head detection can be achieved by using the unsupervised segmentationmethod W-Net and the accuracy can be increased if a high pass filter is applied aspre-processing step.

Color Invariant Convolution (CIConv) is a learnable Convolutional Neural Network (CNN) layer that reduces the distribution shift between the source and target set in the CNN under an illumination-based domain shift. We explore the semantic segmentation performance for daynight domain adaptation when using CIConv. We will test this on two settings: one with only labeled train data available and one with access to both labeled training data and unlabeled test data. In both settings, we will cast an invariant edge detector as a trainable CIConv layer in the CNN to transform the daytime dataset to a domain invariant representation. We will execute day-night domain adaptation and evaluate the mean Intersection over Union over the results. We compare this result to the vanilla version of the same code without using the invariant edge detector as a trainable layer. We will discuss the results obtained from our experiments and show that the trainable CIConv layer does not always result in better outcomes for day-night domain adaptation.

The benchmarks for the accuracy of the best performing object detectors to date are usually based on homogeneous datasets, including objects such as vehicles, people, animals and foods. This excludes a whole set of scenarios containing small, cluttered and rotated objects. This paper selects a state-of-the-art object detection model, Faster R-CNN, and investigates its performance on several custom datasets of scattered LEGO pieces. We discover that the model reaches a high F1 score on data with images containing up to 13 bricks and that data manipulation, such as cropping, can further improve this performance. Furthermore, we evaluate how this model can be optimized to perform better on a more complex dataset, showing that tweaking the Faster R-CNN RPN layer results in a higher F1 score for images containing up to 50 bricks. In conclusion, tweaking the RPN layer allows the Faster R-CNN model to reach high performance on datasets containing cluttered images of small LEGO bricks. All data is on the TU Delft server and all code is available at https://gitlab.com/legoproject-group/faster rcnn lego.git.