Despite significant advancements in computer vision models, their ability to generalize to novel object-attribute combinations remains limited. In Compositional Zero-Shot Learning (CZSL), the goal is to recognize all possible attribute-object combinations while training on only a subset of these compositions. We demonstrate that Vision-Language Models (VLMs) are suitable for CZSL due to their ability to understand both text and image modalities, and we show the importance of cross-modality fusion. Existing methods for CZSL mainly focus on image classification. In this thesis, we enhance CZSL in object detection without forgetting prior learned knowledge. We use Grounding DINO, a VLM for object detection, and create auxiliary tokens for the words present in the training dataset, training only these to mitigate forgetting. This technique, referred to as Compositional Soft Prompting (CSP), was initially used for image classification, and we incorporate it into Grounding DINO to leverage it for object detection. We show that for CZSL, it is beneficial to anticipate the orientation of embeddings in the embedding space, promoting independence between attributes and objects. Additionally, by anticipating possible compositions that could exist based on the attributes and objects present in the training data, we assign soft labels for partial correctness using a method we call Compositional Smoothing. This approach guides the model to learn what the compositions are composed of rather than learning the compositions themselves. We refer to the combined approach of these methods as Compositional Anticipation. Our approach achieves a 70.5% improvement over CSP on the harmonic mean (HM) between seen and unseen compositions on the CLEVR dataset. Furthermore, we introduce Contrastive Prompt Tuning to incrementally address model confusion between similar compositions. We demonstrate the effectiveness of this method, achieving an increase of 14.5% in HM across the pretrain, increment, and unseen sets. Collectively, these methods provide a framework for learning various compositions with limited data, as well as improving the performance of underperforming compositions when additional data becomes available.

Learning new concepts is a difficult task for autonomous robots. These robots can adapt to changes in the situations. To adapt to a situation, they should be able to determine the usefulness of objects around them. The usefulness of objects is highly dependent on situational context, making pre-programming of adaptation behaviour to all possible situations difficult. Automatic learning of this information by the robot from its own observations during deployment is a more feasible approach. We encode the usefulness of objects in logical statements that symbolise regularities in observations of objects in a Markov Logic Network. This network forms the basis from which the usefulness of newly observed objects or situations can be inferred by a robot. Each logical statement in the network has an associated weight that indicates how strong the robot believes the statement is true. These weights can be adjusted over time based on new observations made by the robot, strengthening its beliefs or creating a new belief for an unknown object in a process called cumulative learning. We extend the Markov Logic Network framework with a cumulative learning algorithm called MLN-CLA. This algorithm contains several different knowledge updating strategies that handle conflicts. We demonstrate the ability of MLN-CLA to learn the usefulness of unknown objects over time. In addition, we demonstrate the abilities of MLN-CLA to incorporate new logical statements to better capture information about objects and situational contexts. Together, these two abilities enable robots to autonomously adapt to changing situations during deployment.

In the open world, machine learning (ML) models can encounter a multitude of unknown or novel classes. In a surveillance, safety, or security use case, unknown samples can pose potential threats that are hard to detect since those samples have never been trained on. At the same time, most of the unknowns that will be encountered by a surveillance ML model will be harmless. This results in too many unwanted alerts and manual analyses, of harmless unknowns that have been flagged.

Through this thesis, for the first time (to the best of our knowledge), a method is developed that can automatically assess the relevance of unknown classes, by modelling their image features as clusters (or distributions) and comparing them using statistical distance measures. Our use case lies in computer vision for military applications, where based on the user input, relevance is defined. We define road vehicles as relevant classes and use those for our training set. Our aim is to build a model that can successfully classify new unseen road vehicles as ‘relevant unknowns’, while also successfully classifying harmless unknown birds that are not part of the training set, as ‘irrelevant unknowns’. On the DomainNet data-set, we demonstrate that our novel method can very accurately determine the relevance of unknown classes at test time for both low and high-dimensional data, with AUC scores ranging from 0.99 to a perfect 1.00.

Visual surveillance technologies are increasingly being used to monitor public spaces. These technologies process the recordings of surveillance cameras. Such recordings contain depictions of human actions such as "running", "waving", and "aggression". In the field of computer vision, automated detection of human actions in videos is known as action detection. Recently, deep learning models have been proposed for the task of action detection. Deep learning models for this task can be grouped into single-frame models and multi-frame models. Single-frame models detect actions using individual frames of videos whereas multi-frame models detect actions using sequences of frames.
This thesis proposes to use multi-frame models as compared to single-frame models for action detection in surveillance videos. To compare multi-frame and single-frame models, we implement the ACT-detector. The ACT-detector is a deep learning model that takes as input a sequence of K frames and outputs tubelets (labeled sequences of bounding boxes). We train and evaluate ACT for various values of K on the VIRAT dataset. In our comparison, K=1 serves as the single-frame model and K>1 as the multi-frame models. When compared qualitatively, we find that multi-frame models have less missed detections. When compared quantitatively, we find that multi-frame models outperform single-frame models in performance measures such as classification accuracy, MABO, frame-mAP, and video-mAP.
To assess whether the improvements of multi-frame models yield purely from the increased number of frames, or also from the temporal order encoded by those frames, we experiment with training multi-frame models on unordered sequences of frames, i.e., sequences for which the frames are shuffled in time. When compared qualitatively, we find that multi-frame models have less precise localization when trained on unordered sequences. When compared quantitatively, we find that multi-frame models perform worse when trained on unordered sequences, indicating that multi-frame models learn temporal dynamics of actions. Nevertheless, even when trained on unordered sequences, multi-frame models outperform single-frame models for action detection in surveillance videos.