3D human pose estimation is a widely researched computer vision task that could be applied in scenarios such as virtual reality and human-robot interaction. With the lack of depth information, 3D estimation from monocular images is an inherently ambiguous problem. On top of that, unrealistic human poses have been overlooked in the majority of papers since joint detection is the only focus. Our work consists of two parts, an end-to-end 2D-3D lifting pipeline and a novel kinematic human model integrated approach. We start with Pose Estimation using TRansformer (PETR), an approach that does not require temporal information and has the attention mechanism to model the inter-joint relationship from RGB images. In the approach with human model, we emphasize pose similarity rather than focusing on joint detection. We propose a new metric, called Mean Per Bone Vector Error (MPBVE), that evaluates poses regardless of a human body’s gender, weight, or age. We introduce Pose Estimation on Bone Rotation using Transformer (PEBRT), a novel approach that regresses rotation matrices for 16 human bones, assuming labeled 2D poses as input. Our human model encapsulates joint angle and bone length constraints. Existing methods treat these constraints as an additional loss term, which does not guarantee realistic final out- puts. Our method does not require temporal information or receptive fields to generate kinematically realistic human poses. We demonstrate that PEBRT is capable of delivering comparable results on Human3.6M to existing methods.

As robots are becoming a more integral part in our daily lives, it is important to ensure they work in a safe and efficient manner. A large part of perceiving the environment is done through robot vision. Research in computer vision and machine learning lead to great improvements in the past decades and robots are able to outperform humans on certain tasks. However, these tasks are often in closed set condition. This makes translation to a real world application challenging, as this is in open set condition. The open set condition implies that incomplete knowledge of the world is available at training time. It is important for a robot, or agent, to be aware of this limitation. In vision tasks this is defined as open world recognition and allows an agent to detect and incrementally learn unknown objects. The key contributions of this report are an autonomous data collection protocol for synthetic data creation, the open world algorithm Learning to Accept Image Classes (L2AIC) and an on-the-job recognition approach that combines open world recognition with autonomous data collection. L2AIC is a deep meta-learning model that classifies objects by comparing it to its memory in an n-way k-shot manner. New classes can be incrementally added to the memory without needing to retrain the model. The autonomous data collection protocol consists of two steps. First, a 3D model is reconstructed from an unknown object with an RGB-D camera. Secondly, from this 3D model a synthetic dataset is created that is added to the memory of L2AIC. Results show that the on-the-job recognition approach is successful in learning to recognize a single unknown object using the L2AIC model with small-fc architecture an ResNet152 encoder. The encoder is loaded with pretrained weights on the ImageNet dataset. No additional fine-tuning is required, this has an adverse effect on the performance. Using the autonomous data collection protocol two datasets were created, varying the distance from the camera to the object. It was found that the synthetic dataset containing close-up images achieves the best performance. The performance of L2AIC with this closeup synthetic dataset is similar to using a dataset of actual images of the object. This means that for a single object it currently is not efficient to create a synthetic dataset. A more extensive study is required to compare performance of both datasets when increasing the number of encountered objects. Finally, the on-the-job approach shows it is capable of recognizing other instances of the same object class, using only the dataset of a single instance. This shows the on-the-job recognition approach is able to generalize well. Future work could be focused on improving this generalization with, for example, the help of Generative Adversarial Networks (GANs).

Tooth removal is one of the most performed surgical procedures worldwide. Despite the high amount of tooth removal procedures carried out each year, scientific understanding of these procedures is not present. Knowledge of force and torque behaviour is limited and knowledge about movements has never been subject to scientific research before. This study is an initial attempt to describe the factors that influence tooth removal in terms of forces, torques and movements. In-vitro measurements were performed that resulted in a dataset containing force, torque and movement time series of 181 human demonstrations of tooth extractions. This report showed how feature engineering and classification modelling were employed to find tooth removal explaining parameters in the dataset. The feature engineering process led to numerical features describing the force and movement (rotation) time series. The rotation features were found to be most descriptive in describing differences in tooth removal procedures. This introduced five distinct rotation strategies that grouped the human demonstrations based on similarity of extraction strategy. These groups have been used as classification labels in the supervised learning process. A Naive Bayes algorithm and a Logistic Regression algorithm were implemented as prediction models. These models showed that while the rotation features contributed the most to the prediction performance, there was need for additional force features to reach maximum prediction performance. The results showed how feature engineering and classification modelling are the first steps in understanding the procedure of tooth removal.