Coastal zones are dynamic and vulnerable regions, demanding accurate, scalable monitoring tools to inform environmental management and hazard mitigation. While satellite imagery and CNN-based classifiers have improved automated mapping, their reliance on unstructured pixel data limits contextual understanding. This study presents the first fine-tuning of a multi-modal large language model (MLLM), Qwen2.5, on 12-channel satellite input for multilabel coastal classification, demonstrating how architectural adaptation enables integration of spectral, topographic, and derived features beyond RGB. We compare this approach to a ResNet-50 baseline and state-of-the-art prompting methods using GPT-4o and LLaMA-3.2. Our experiments on the CoastBench dataset reveal that MLLMs benefit substantially from few-shot prompting with diverse, balanced sampling and that fine-tuning Qwen2.5 with full 12-channel input outperforms its RGB-only variant. An ablation study quantifies the importance of elevation and water-sensitive indices, while a human benchmark exposes a performance ceiling near F1 ≈ 0.70 due to label ambiguity. Our findings suggest that while MLLMs can rival traditional models and offer interpretability benefits, future gains depend on dataset quality, input diversity, and prompting strategy design.

State-of-the-art models are susceptible to adversarial attacks. These attacks can cause catastrophic misclassification when robustness is required. With the increasing popularity of the retrieval augmentation paradigm in deep learning, we adopt it as a fully differential framework for adversarial robustness. We evaluate our method on three visual classification datasets, including ImageNet and attack our model with two white box attacks and a black box attack under various L₂ and L_∞ norms. The results indicate that a robust classifier emerges if the model fully relies on retrieved examples. We find that we can already obtain a PGD robust ImageNet classifier with 80.1% clean and 64.7% adversarial accuracy, using only one or two examples per class from the training data in the memory set. Contrary to other adversarial defense mechanisms, our method works directly on top of pre-trained models and remains robust when other defenses start to degrade for PGD attacks increasing in strength.

Building registry updates are essential for urban planning but remain a labor-intensive process. This thesis introduces PolyChange, an adaptation of the PolyBuilding model, to automate mutation delineation by integrating aerial imagery with reference maps to produce precise, vectorized building mutations. PolyChange combines raster-based and direct polygonal representation methods, augmented with angle loss, double-thresholding, and label smoothing to enhance delineation accuracy. Experiments on our synthetic dataset demonstrate that raster-based methods achieve superior delineation, with an AR of 89.3\% and AP of 85.2\%. However, real-world datasets reveal challenges in generalization due to overfitting, semantic vagueness, and annotation inconsistencies, resulting in reduced detection and delineation capabilities, indicated by an AR of 30\% and AP of 13.7\% on the test set. Future work should address these limitations by improving dataset quality, refining model architectures, and leveraging reference maps more effectively to achieve fully automated and accurate building registry updates.

In the context of open-world scenarios in autonomous vehicles (AVs), previously unseen classes may arise. To address this, effective extraction of well generalizable features is essential for AV downstream tasks, especially in the context of zero-shot learning. This can be achieved using transformers, and Swin Transformer in particular (vision backbone of most Vision-Language Models). However, to enable on-board applications, knowledge distillation must be utilized to create a lightweight model capable of real-time processing. We explore self-supervised knowledge distillation, given that AV datasets need to generalize to previously unseen classes. Our contributions include adapting existing CNN-to-CNN output-based self-supervised knowledge distillation algorithms to Transformer-to-CNN for benchmarking and enhancing them with a cross-architecture loss function. By leveraging DisCo, the best performing output-based self-supervised knowledge distillation method, and using EfficientNetB0 as the student model, we achieve a 3.9% relative improvement in top-1 accuracy over the supervised Swin-T teacher on our modified ImageNet for open-world classification, up to 5.0% with our loss.

Visual impairment affects over 2.2 billion individuals globally, emphasizing the critical need for effective assistive technologies. This work focuses on developing a video captioning model explicitly tailored for visually impaired users, leveraging advancements in deep learning techniques. Video captioning involves converting video frames into textual descriptions, effectively bridging the domains of computer vision (CV) and natural language processing (NLP). We surveyed young visually impaired individuals from the Visio organization, who provided key insights into the design of our model.
We enhance the existing S2VT model by modifying its temporal attention mechanism to improve the recognition of visual surroundings, addressing the unique challenges visually impaired individuals face.
This research explores critical questions surrounding the model's sensitivity to actions, the readability of generated captions, and methods for latency reduction. To evaluate the model's effectiveness, we implement readability metrics—an approach not previously utilized in video captioning assessments. Our findings contribute to enhancing accessibility and independence for visually impaired individuals through advanced video captioning solutions.

In recent years the emergence of Spiking Neural Net- works (SNNs) has shown that these networks are a promis- ing alternative to traditional Artificial Neural Networks (ANNs) due to their low-power computing capabilities and noise robustness. Nevertheless, in recent approaches, they have either strayed away from spike time backpropaga- tion, used discrete time, single spike neurons, and/or limited themselves to shallow networks. These approaches limit the potential of SNNs by sacrificing significant aspects of what makes them special in order to have a functional network. It is for this reason that we believe that, the implemen- tation of residual connections, allowing a model that has multi-spiking neurons, precise time and spike time back- propagation is the path to allow these networks to truly shine. As it will solve one of this model’s most severe lim- itations which prevent it from being used to build deeper networks, the banishing of spikes at a deeper depth. Hence we aim to remedy this problem by feeding inputs from fur- ther up the network to revitalize the spike counts at deeper layers. In this paper, we explore the implementation of residual connections in precise time multi-spiking neural networks as a way of solving the disappearing spikes problem that inhibits spike time backpropagation. We explore two al- ternatives in the implementation of the residual connection and we analyze how they affect the accuracy of the network as the depth increases in both Multi-Layer Perceptrons and Convolutional Neural Networks. We also developed an ar- chitecture to allow for swapping of the fuse function in or- der to take full advantage of the flexibility that precise time provides us. Results show that the implemented residual connections allow for deeper training, which has the poten- tial to aid in the network’s performance, although in some cases some hurdles remain.

Generating arbitrarily long, temporally consistent, and visually realistic street-view videos presents a formidable challenge at the intersection of computer vision and graphics. Existing methods, while capable of producing high-quality individual frames or short sequences, often struggle to maintain coherence and realism over extended durations. Parallel generation approaches, though effective in ensuring frame-to-frame consistency, face limitations in generating videos of arbitrary length and adapting pre-trained text-to-image models for the video domain. In response to these challenges, we propose a novel sequential approach that draws inspiration from recent advancements in 3D-aware image generation and object reconstruction. Our model, built upon the foundations of state-of-the-art techniques like MagicDrive, incorporates innovative conditioning mechanisms to leverage the visual context of previous frames. By establishing strong temporal dependencies and ensuring smooth transitions between frames, our approach enables the creation of coherent and controllable video sequences of any desired length. Through extensive experimentation on the challenging nuScenes dataset, we demonstrate the effectiveness of our sequential generation framework. Our model not only achieves competitive performance compared to existing parallel methods but also offers enhanced flexibility and computational efficiency. The ability to generate arbitrarily long videos opens up new possibilities for applications such as autonomous vehicle simulation, virtual reality training, and urban planning, where realistic and diverse visual data is crucial. Furthermore, our research contributes to a broader understanding of the trade-offs between sequential and parallel generation paradigms, highlighting the potential of sequential approaches for addressing the limitations of current methods.

With the advent of large language models (LLMs), developing solutions for Natural Language Processing (NLP) tasks has become more approachable. However, these models are opaque, which presents several challenges, such as prompt engineering, quality assessment, and error analysis. Explainability methods can have several potential benefits, such as improving accuracy, increasing trust, and assessing quality. However, limited research exists on how explainability techniques can be applied to LLMs in practice, particularly using human-centred methodologies. Therefore, this study takes a user-centered approach, investigating the needs and challenges of the NLP data scientist and developing an explainability tool to address these needs. This approach is done by conducting a formative study to deepen our understanding of the user, combined with relevant literature. The observations from the formative study were used to develop a tool tailored to the user’s specific needs. This development was done by creating requirements and a design based on the findings of the formative study, followed by a proof of concept implementation. User satisfaction was assessed through practical interviews with a fairness dataset, providing insights into the usefulness and usability of the explanation techniques and the tool. The tool implements three explanation techniques: uncertainty, token-level feature attribution, and contrastive explanations. These can be viewed using a web application separated from the Python development environment, making it easy to interact with. Other key features are that it can be easily integrated into the user’s existing workflow, is usable in practice and can be presented to different stakeholders within the project. The evaluation concluded that the tool fits the workflow and does indeed help the NLP data scientist to understand the model. However, the evaluation also showed that the explainability techniques did not provide the necessary insights to achieve the user’s goal, mainly to improve the model’s accuracy and make the error analysis actionable. More research should be done to see which other explainability techniques could provide insights that would lead to objectively better performance of these models. Finally, more explainability techniques should be developed that do not focus on debugging the model but rather on revealing its behaviour and thus providing a better understanding of how to improve it.

Object detectors have come a long way and are used for various applications. In pictures and videos, an object detector must deal with the background. In some settings, this background is indicative of the object; in others, it’s not and can even be disruptive. For models trained on data containing correlations between objects and backgrounds (background bias), it makes sense that changing the background can disrupt learned correlations. This paper is interested in how sensitive object detectors are to background changes, specifically when the training data does not contain correlations between objects and backgrounds. Models were trained on carefully controlled synthetic data, so only the backgrounds differed and correlations could be controlled. The results show that models perform better when tested with seen backgrounds than unseen backgrounds. This performance difference diminishes when the model is trained on more unique backgrounds.

Structure-from-Motion (SfM) and Neural Radiance Fields (NeRFs) have significantly advanced 3D reconstruction in multi-view scenarios. Despite their success in handling non-repetitive, texture-rich scenes, applying such techniques to real-world scenarios with texture-less and repetitive structures remains challenging. One such case is in industrial inspection processes, specifically the reconstruction of aircraft engine blades. The goal is to build a 3D model of all the engine blades from a single video. We explore the application of SfM in modeling repetitive and texture-less blades, identify common causes of failure, and improve upon the default SfM by replacing the commonly used exhaustive match with sequential match to handle ambiguity stemming from repetitiveness. Sequential matching enables more precise pose estimation and better 3D reconstruction in our scenario. In addition, we explore the importance of choosing the correct camera model and provide a comparative look at the existing 3D mesh reconstruction solutions, presenting tweaked versions that result in better performance. This work lays the foundation for 3D reconstruction of repetitive and texture-less objects by proposing sequential matching, enabling better 3D modeling of engine blades compared to classic SfM pipelines.

Leprosy is a neglected tropical disease affecting people worldwide. Left untreated or starting therapy too late, the infection can cause permanent damage to the nerves, skin, eyes, and limbs of patients. While current standard diagnostic techniques generally detect the disease once symptoms have manifested, infrared thermography presents a potential tool for the early detection of leprosy-related neuropathy. A research initiative has been established to explore whether autonomic nerve impairment in the hands can be identified using infrared thermography, necessitating a new method for temperature evaluation. This work discusses the development of a semi-automatic temperature analysis method for twelve regions of interest on the hands. The approach utilises real-time landmark tracking based on an existing machine-learning model, adapted to the context of the application. After applying a cold impulse and recording the response with an infrared camera, the rewarming temperature and recovery values are determined within the regions. Results presented in this report indicate no significant improvement compared to the existing manual method. The success of the model outcomes highly depends on the accuracy of the machine-learning-based hand landmark detection, resulting in insufficiently reliable outcomes. The underlying explanations and the potential for future improvements are further discussed in this thesis, considering the evident potential and benefits of the automated approach.

With the increasing popularity of Large Language Models (LLMs), fine-tuning them has become increasingly computationally expensive. Parameter Efficient Fine-Tuning (PEFT) methods like LoRA and Adapters, introduced by Microsoft and Google, respectively, aim to reduce the number of trainable parameters, with the current state-of-the-art combining both methods as LoRA Adapters. This paper introduces Transformer Modules as a PEFT method. These modules utilize Modular Transformer Blocks (MTBs) inserted into a frozen pre-trained model, achieving competitive performance while significantly reducing computation costs. Compared to the current state-of-the-art using GPT-2, BERT, and T5, Transformer Modules further reduced compute time by 39.7\% and training memory by 72.7\%, with a performance cost of 4.5±2.51\% on the GLUE benchmark. Additionally, the paper presents the Transformer Bridge, a continuous vector transformer designed to transfer Transformer Modules across different models. This could enable cross-model fine-tuning, allowing model-agnostic modules, such as an ethics or medical module, to be used across various LLMs without retraining or access to the original dataset. Although the current implementation of the Transformer Bridge did not fully succeed in mapping embedding spaces, analysis of the results suggests that further refinements using traditional model distillation techniques could lead to success in future iterations.

In large-scale ML, data size becomes a critical variable, especially in the context of large companies, where models already exist and are hard to change and fine-tune. Time to market and model quality are essential metrics, thus looking for ways to select, prune and augment the input data while treating the model as a black box can speed up the process from raw data to productionized model.

Datasets can have thousands of features and many redundant/duplicate samples, for various business logic reasons. In some particular ML flows, it might be that only a subset of them provide most of the input to the final accuracy. Also, looking into ways to provide insights on what data points are the most meaningful can help engineers collect more relevant samples, or focus their attention on specific parts of the data distribution.

To address the challenges of manual egg sorting on poultry farms, such as labor intensity and
inconsistent quality standards, Moba is developing a machine for automated egg candling
using computer vision. As part of the prototype phase, this project evaluates various methods
to process the egg images for this sorting task.
This project presents an algorithm that segments the individual eggs within the camera’s
field of view, classifies them into five categories, and tracks them across multiple images
to perform the outlet control prediction. To enhance computational efficiency, an anomaly
detector is introduced, directing classification resources exclusively to eggs with a high like-
lihood of defects.
For segmentation, the number of channels in the YOLOv8n model was significantly re-
duced, resulting in YOLOv8xn. With a 97.2% AP95, this model outperformed a classical
algorithm using domain-specific information, as well as more complex deep learning meth-
ods, such as YOLOv8n and the prompt-based FastSAM and MobileSAM models.
Various low-complexity deep learning models were evaluated and compared to classical
feature extraction combined with ensemble machine learning methods for the classification
task. Balancing accuracy and runtime, MobileNetV3 proved to be the optimal choice, achiev-
ing 95.6% accuracy on the final test set.
For anomaly detection, both classical methods and the autoencoder reconstruction loss
were evaluated. Modeling the edge count of non-defective eggs with an exponential distribu-
tion to set a threshold reduced the computational load for classification by 83%. Furthermore,
using the anomaly detector the batch size can be reduced from 32 to 8 images.
To send the eggs to the correct outlet, the predictions for each egg were combined using
thresholds on count-per-class and normalized cumulative class predictions, both without and
with anomaly detection enabled. This resulted in overall accuracies of 97.3%, 98.4%, and
98.0%, respectively.

Insects such as Diptera are capable of highly complex aerial maneuvers and rapid responses to environmental stimuli, making them a subject of interest for studies in flight dynamics and motor control. To accurately quantify these movements, high-speed cameras are employed, capturing footage at 5000 frames per second. This results in a vast amount of data, necessitating the development of automated analysis models. In this work, we present a segmentation model specifically designed for the wings of Diptera species, aiming to track their wingbeats with high precision. The model’s performance was evaluated on species included in the training set as well as on species only included in the testset, allowing us to assess its generalization capabilities. Our results show that the model achieves a 95th percentile error of 7.8 pixels, while operating at a speed 2500 times faster than manual human analysis. This significant improvement in speed highlights the potential for our model to facilitate large-scale, high-speed analysis of insect flight, with implications for both biological research and bio-inspired engineering applications.

Training data ambiguity, such as the presence of out-of-frame moving objects, introduces significant challenges in deep learning-based optical flow models by causing large loss spikes and training instability. Most models overlook this ambiguity, treating it as a limitation of existing datasets. SEA-RAFT attempts to address these ambiguous areas with an additional network and a modified loss function, yet does so without explicitly verifying the specific drawbacks. In this paper, we investigate the influence of out-of-frame movements on model accuracy by generating the FlyingIcons dataset, which includes in-frame and out-of-frame masks for precise analysis. Using the latter masks, we introduce a weighted masked training scheme that selectively penalizes errors in out-of-frame areas, significantly increasing model accuracy over both standard GMA training and SEA-RAFT. Building on this concept, we propose a weighted partially masked training method, which uses partial out-of-frame masks generated through a simple process that adds the ground truth flow to pixel locations and checks if they fall outside the frame. While this method only yields improvements in error reduction on FlyingThings3D, our findings suggest that incorporating similar masks into other synthetic datasets could improve model stability and accuracy with minimal additional overhead. This highlights a promising direction for further research, particularly in developing more complex mask generation strategies and creating synthetic datasets with out-of-frame masks to enhance generalizability across datasets.

Designing processors for implantable closed-loop neuromodulation systems presents a formidable challenge owing to the constrained operational environment, which requires low latency and high energy efficacy. Previous benchmarks have provided limited insights into power consumption and latency. However, this study introduces algorithmic metrics that capture the potential and limitations of neural decoders for closed-loop intra-cortical brain-computer interfaces in the context of energy and hardware constraints. This study benchmarks common decoding methods for predicting a primate’s finger kinematics from the motor cortex and explores their suitability for low latency and high energy efficient neural decoding. The study found that ANN-based decoders provide superior decoding accuracy, requiring high latency and many operations to effectively decode neural signals. Spiking neural networks have emerged as a solution, bridging this gap by achieving competitive decoding performance within sub-10ms while utilizing a fraction of computational resources. These distinctive advantages of neuromorphic spiking neural networks make them highly suitable for the challenging closed-loop neural modulation environment. Their capacity to balance decoding accuracy and operational efficiency offers immense potential in reshaping the landscape of neural decoders, fostering greater understanding, and opening new frontiers in closed-loop intra-cortical human-machine interaction.

Machine learning, a pivotal aspect of artificial intelligence, has dramatically altered our interaction with technology and our handling of extensive data. Through its ability to learn and make decisions from patterns and previous experiences, machine learning is growing in influence on different aspects of our lives. It is, however, shown that machine learning can be attacked, and by the attacks, its functioning may become completely opposite of what it was designed. A special kind of attack on machine learning models is a backdoor attack. It uses a special pattern that was placed in the training data by malicious users to alter the models’ behaviour. This pattern is called a backdoor trigger, and it can take any possible form. The test data with this trigger will be misclassified, while the clean data will get a correct prediction. This property makes the backdoor attacks stealthy and hard to detect.

The backdoor attacks are mostly created to attack the classification models, where for each data sample, there is a label. In this thesis, we move away from the classification setup and create the first (to our knowledge) backdoor attack on the linear regression. We show that the triggers constructed using different versions of feature selection algorithms can be effective and impose a high error on the linear learning model prediction. Additionally, the study shows that backdoor attacks with the trigger constructed with a feature selection using correlation analysis lead to a higher error than the one using random forest for feature selection.

Furthermore, we also transfer this backdoor attack to the federated learning setup. The results prove to be highly dependent on the number of poisoned nodes, while for all of them, the error for the poisoned region is higher than for the clean data.

Finally, for the attack in both setups, we have adapted popular defence mechanisms that work against backdoor attacks on classification models. For the centralised setup, we have explored the possibility of using the studentized residuals as an outlier detection mechanism. The results are diverse, becoming worse when the poisoning rate of the model increases. To prevent the attacks in the federated setup, we used the FoolsGold defence mechanism, and it proved to be effective against the backdoor attacks on the regression model in all the cases except the one with exactly one attacking node.

Current methods in Federated and Decentralized learning presume that all clients share the same model architecture, assuming model homogeneity. However, in practice, this assumption may not always hold due to hardware differences. While prior research has addressed model heterogeneity in Federated Learning, it remains unexplored in fully decentralized or peer-to-peer settings. Therefore, in this paper, we investigate a real-world yet challenging situation involving model heterogeneity in a fully decentralized context. Furthermore, we introduced a Model Agnostic Peer-to-peer Learning (MAPL) framework, which allows simultaneous learning of heterogeneous personalized models. Additionally, we define a graph learning objective to infer optimal collaboration weights based on task similarity. Experiments reveal that even in this challenging scenario, MAPL delivers competitive results while being communication efficient owing to the sparse collaboration graph in both model homogeneous and heterogeneous settings.