Unmanned Ground Vehicle (UGV) navigation in unstructured off-road environments can benefit from accurate traversability estimation. Often, experiments with UGVs use semantic segmentation networks for visual scene understanding. Based on the pixel-wise classification of a semantic segmentation network, the UGV can distinguish traversable from non-traversable terrain. However, it is still an open challenge to design a model which is able to accurately estimate traversability in a variety of environments. Variation in terrain characteristics and different levels of structuredness requires a model with a high level of generalisability. Limited generalisability will result in inaccurate traversability estimation, which in the worst-case scenario can cause the UGV to crash. In order to overcome limited generalisability, a hybrid semantic segmentation framework is presented that can switch between different operation modes. The hybrid framework contains multiple environment-specific segmenters. For each input frame, the hybrid framework selects an environment-specific segmenter, based on a decision parameter. In this work, two hybrid frameworks containing different decision parameters are designed. The first hybrid framework contains multiple Bayesian segmenters, which quantifies prediction uncertainty in addition to the pixel-wise classification. This uncertainty quantification is obtained by Monte Carlo sampling to generate a posterior distribution of pixel class labels. The second hybrid framework consists of multiple environment-specific segmenters and autoencoders. Every segmenter has a corresponding autoencoder trained on the same environmental dataset. The output of the environment-specific autoencoder is a reconstructed image of the input image. The error between the original input image and the reconstructed image is used as a decision parameter for selecting the best performing segmenter. We experimented with a hybrid segmentation framework and observed that it could outperform a single semantic segmentation network with a 2.6% Intersection over Union increase. The hybrid framework with the autoencoder approach resulted in a model selection precision of 99.3% on all the test images. Therefore, we can conclude that UGV navigation can benefit from a hybrid semantic segmentation framework.

Traversability estimation is a key component in autonomous driving tasks. In many applications, semantic segmentation is used to pixel-wise classify a visual scene. The pixel-wise segmented map is used to estimate the traversability of different environments. The semantic segmentation accuracy can drop if environmental conditions change. The introduction of shadow in images can cause the segmentation network to misclassify pixels, which leads to an inaccurate traversability estimation. This inaccurate estimation could lead an autonomous vehicle to deviate from traversable paths, leaving it unable to continue or even cause accidents. To increase the segmentation accuracy in shadow conditions, shadow removal before semantic segmentation is proposed. In this research, the supervised Dual Hierarchical Aggregation Network (DHAN) and unsupervised cycle-based Shadow Generative Adversarial Network (SGAN) are used for shadow removal before semantic segmentation with Segmentation Network (SegNet). The shadow removal networks are evaluated on two datasets, containing image triplets, consisting of shadow, shadow-free and shadow-mask images. The structural similarity is calculated for complete images and the non-shadow regions by inverting the shadow-masks. The networks are applied to the Cambridge-driving Labeled Video Database (CamVid) dataset to evaluate the change in segmentation accuracy. In a second set of experiments, the DHAN shadow removal network is retrained on multiple datasets containing synthetic shadows. The obtained DHAN networks are tested on shadows with increasing intensity. The retrained DHAN networks are compared to evaluate the training dataset for shadow removal to increase segmentation accuracy. The experiments show that the DHAN increases the structural similarity of 99.8% and the SGAN for 74.5% of the datasets. After retraining on the synthetic shadow dataset, segmentation accuracy after DHAN shadow removal increases the segmentation Pixel Accuracy (PA) for a maximum of 91% and the segmentation mean Intersection over Union (mIoU) for a maximum of 88% of the images of the CamVid dataset. We conclude that segmentation accuracy increases after DHAN shadow removal if the DHAN is trained on the synthetic shadow dataset.

Pick and place systems that operate in a warehouse setting have been studied a lot recently due to the high economic value for e-commerce companies. In this thesis, the focus is on the perception pipeline that performs object recognition given a certain input data stream (typically RGB-D images). Impressive results regarding object recognition have been reported in the last years, mainly driven by the development of convolutional neural networks. However, only very few proposed perception pipelines are suitable to adapt quickly to recognize new objects. This is considered a problem since large e-commerce companies add many new products to their inventory every day. In this thesis, efforts are made to solve this problem by proposing the use of a neuro-symbolic model in which concept learning is combined with symbolic reasoning. First, visual attributes are obtained from an input image by passing it through the neural part of the model, which consists of two β-Variational Autoencoders. The key element of the model is that the visual attributes can be recognized even if a particular combination of visual attributes have not been in the training dataset. Next, given a knowledge base and the visual attributes as inferred from the input image, a symbolic reasoner infers the most likely object ID. Hereby, the knowledge base is manually constructed and describes the relationships between object IDs and the corresponding visual attributes. The implementation of the neuro-symbolic model is first tested on a synthetic dataset which is similar to the dataset as used in the study that the neural part is based upon. Thereafter, the neuro-symbolic model is tested on real RGB-D images of the pick-and-place dataset and several iterations of the baseline model are evaluated. Hereby, the main research question is formulated as: Can a neuro-symbolic model be used to recognize unseen objects given RGB-D data as typically seen in pick and place scenarios? The best top-1 accuracy score on unseen images of the synthetic dataset was 79.5%. However, using the same neuro-symbolic model on the pick-and-place dataset, the top-1 accuracy score on unseen images was only 25.5%. In the following iterations of the model, the top-1 accuracy score was improved up to 32.4%. Analysis of the results of the pick-and-place experiments shows that the neural part is not very capable of recognizing the correct visual attributes. This likely be due (to some extend) to the simulation-to-real gap. However, further research is required to identify the exact cause(s) of the performance drop. Concludingly, the proposed neuro-symbolic model is capable of recognizing unseen images of the synthetic dataset, but is not very capable of recognizing unseen images of the pick-and-place dataset.

Virtual reality (VR) creates an exceptional experience in which users can explore virtual environments. Wearing a head-mounted display (HMD), users are able to observe a virtual world that is rendered based on their physical movement and actions. A common solution for capturing the visual and geometric information needed for the construction of virtual environments is the use of RGB-D sensors. These sensors not only capture a collection of RGB data like conventional cameras do, but additionally record a depth value for each pixel. Thus, RGB-D sensors are able to capture both the visual and geometric properties of a space, including any objects or people. This makes immersive social VR experiences possible, where people in different physical locations can be placed in the same virtual environment. However, HMDs obstruct the RGB-D sensor from capturing the wearer's upper face, which severely impacts the social aspects of VR applications. To address this, we proposed a framework that is capable of the virtual removal of head-mounted displays in RGB-D images, which is referred to as the task of HMD removal. Due to its novelty, we took an exploratory approach to this task. We formulated this problem as a joint RGB-D face image inpainting task and proposed a GAN-based coarse-to-fine architecture that is capable of simultaneously filling in the missing color and depth information of face images occluded by an HMD. To preserve the identity features of the inpainted faces, we proposed an RGB-based identity loss function. Leveraging the knowledge of a pretrained identity embedding model, this perceptual loss function stimulates the preservation of identity-specific facial features. Furthermore, we proposed several architectural structures to explore multimodal feature fusion of the color and depth information contained in RGB-D images. To this end, we introduced data-level fusion, which naively combines the color and depth information at network input. In addition, we introduced hybrid fusion, which involves feature-level fusion in the coarse stage of the architecture and data-level fusion in the refinement stage of the architecture. Within the concept of hybrid fusion, we investigated several fusion strategies, including residual fusion. Our findings suggest that data-level fusion achieves similar performance to hybrid fusion. Moreover, to improve surface reproduction in the depth channel, we introduced the employment of a surface normal loss function and contextual surface attention module, which both rely on surface normals that are estimated based on the depth channel of the RGB-D image. We also considered the addition of surface normal information to the discriminator input, which we found to have an adverse effect on the visual quality of the results. In absence of a large scale RGB-D face dataset, we devised a pipeline for the creation of a synthetic RGB-D face dataset for the evaluation of our network. Despite its exploratory nature, our research provides unique insights into the design and behavior of a multimodal image inpainting architecture that can be of interest to future research.