Affordances, or action possibilities, have been explored to enable robotic manipulation with everyday objects, however the effect of an agent's embodiment has not received much attention. Here we investigate how embodiment changes affordances between a human and robot. We present a method to automatically generate affordance pseudo-labels from a robotic manipulator for the task of grounding (localising) affordances on an object, as there is no such existing dataset. We then propose a general model for embodiment-conditioned affordance grounding, and explore three ways to condition on the embodiment. Our model learns to perform an affine transformation on image embeddings based on the effect of embodiment on the affordance. We evaluate all three variants of our model and compare them to a variant without embodiment conditioning and a state-of-the-art affordance grounding method. The results show that our best performing model decreases affordance prediction error by 25% when compared to the variant without embodiment conditioning and by 68% when compared to the state-of-the-art method. Through our results we demonstrate that embodiment matters when perceiving affordances.

Despite significant advancements in computer vision models, their ability to generalize to novel object-attribute combinations remains limited. In Compositional Zero-Shot Learning (CZSL), the goal is to recognize all possible attribute-object combinations while training on only a subset of these compositions. We demonstrate that Vision-Language Models (VLMs) are suitable for CZSL due to their ability to understand both text and image modalities, and we show the importance of cross-modality fusion. Existing methods for CZSL mainly focus on image classification. In this thesis, we enhance CZSL in object detection without forgetting prior learned knowledge. We use Grounding DINO, a VLM for object detection, and create auxiliary tokens for the words present in the training dataset, training only these to mitigate forgetting. This technique, referred to as Compositional Soft Prompting (CSP), was initially used for image classification, and we incorporate it into Grounding DINO to leverage it for object detection. We show that for CZSL, it is beneficial to anticipate the orientation of embeddings in the embedding space, promoting independence between attributes and objects. Additionally, by anticipating possible compositions that could exist based on the attributes and objects present in the training data, we assign soft labels for partial correctness using a method we call Compositional Smoothing. This approach guides the model to learn what the compositions are composed of rather than learning the compositions themselves. We refer to the combined approach of these methods as Compositional Anticipation. Our approach achieves a 70.5% improvement over CSP on the harmonic mean (HM) between seen and unseen compositions on the CLEVR dataset. Furthermore, we introduce Contrastive Prompt Tuning to incrementally address model confusion between similar compositions. We demonstrate the effectiveness of this method, achieving an increase of 14.5% in HM across the pretrain, increment, and unseen sets. Collectively, these methods provide a framework for learning various compositions with limited data, as well as improving the performance of underperforming compositions when additional data becomes available.

Visual surveillance technologies are increasingly being used to monitor public spaces. These technologies process the recordings of surveillance cameras. Such recordings contain depictions of human actions such as "running", "waving", and "aggression". In the field of computer vision, automated detection of human actions in videos is known as action detection. Recently, deep learning models have been proposed for the task of action detection. Deep learning models for this task can be grouped into single-frame models and multi-frame models. Single-frame models detect actions using individual frames of videos whereas multi-frame models detect actions using sequences of frames.
This thesis proposes to use multi-frame models as compared to single-frame models for action detection in surveillance videos. To compare multi-frame and single-frame models, we implement the ACT-detector. The ACT-detector is a deep learning model that takes as input a sequence of K frames and outputs tubelets (labeled sequences of bounding boxes). We train and evaluate ACT for various values of K on the VIRAT dataset. In our comparison, K=1 serves as the single-frame model and K>1 as the multi-frame models. When compared qualitatively, we find that multi-frame models have less missed detections. When compared quantitatively, we find that multi-frame models outperform single-frame models in performance measures such as classification accuracy, MABO, frame-mAP, and video-mAP.
To assess whether the improvements of multi-frame models yield purely from the increased number of frames, or also from the temporal order encoded by those frames, we experiment with training multi-frame models on unordered sequences of frames, i.e., sequences for which the frames are shuffled in time. When compared qualitatively, we find that multi-frame models have less precise localization when trained on unordered sequences. When compared quantitatively, we find that multi-frame models perform worse when trained on unordered sequences, indicating that multi-frame models learn temporal dynamics of actions. Nevertheless, even when trained on unordered sequences, multi-frame models outperform single-frame models for action detection in surveillance videos.