Self-driving vehicles have shown rapid development in recent years and continue to move towards full autonomy. For high or full automation, self-driving vehicles will have to be able to address and solve a broad range of situations, one of which is interaction with traffic agents. For correct and save maneuvering through these situations, reliable detection of agents followed by an accurate classification of the traffic gestures used by agents is essential. This problem has received limited attention in literature to date. The objective of this work is to establish and investigate a working traffic gesture pipeline by leveraging the latest developments in the fields of computer vision and machine learning. This work investigates and compares how well state-of-the-art methods translate to traffic gesture recognition and what application specific problems are encountered. Multiple configurations based on skeletal features, estimated using OpenPose, and classified using recurrent neural networks (RNN) were investigated. Skeleton estimation using OpenPose and feature representations were evaluated using an action recognition dataset with motion capture ground-truth. Three RNN network architectures, varying in complexity and size, were evaluated on traffic gestures. The robustness of the developed system regarding viewpoint variation is explored, combined with the viability of transfer learning for traffic gestures. To train and validate these methods, a new traffic gesture dataset is introduced, on which an mAP of 0,70 is achieved. The results show that the proposed methods are able to classify traffic gestures within reasonable computation time and illustrate the value of transfer learning for gesture recognition. These promising results validate the methodology used and show that this direction warrants further research.

One of the most promising ideas in autonomous vehicle control systems is letting the vehicle drive autonomously outside the normal, linear, operating region and letting it "drift". By doing so, the maneuverability of the vehicle could be enhanced. To enable systems that can control this behaviour, estimation of certain vehicle states is needed with high accuracy and high frequency.In this project, a new solution to this problem is proposed by combining a mixed dynamic-kinematic observer with a single camera that produces velocity measurements based on tracking the ground plane. To improve filtering of the camera velocity measurements, the measurement error covariance matrix is updated online based on a model of the camera measurement error. Evaluation of the new methodology was done on data recorded from a 1:10 scale test vehicle and performance was assessed based on ground truth data obtained using a Motion Capture System.In normal driving conditions with correctly identified vehicle parameters, an observer without camera performs better by 25% in terms of RMSE on lateral velocity and sideslip angle estimation. However, the online adaptation of the covariance matrix results in an estimate that is at least 45% more accurate in terms of RMSE than the same observer without online covariance adaptation. Next to that, experiments show that the proposed observer with camera has better robustness to uncertainty in model parameters by almost a factor five in terms of RMSE than the observer without camera.When the grip of the tires is physically lowered and the vehicle is drifting, the proposed observer can track large sideslip angles (>30°), where the state-of-the-art observer without camera is not able. The state-of-the-art observer has an increase in RMSE of 75% on all estimated quantities in comparison to the proposed methodology. These results show that adding a camera to an existing sideslip angle observer greatly enhances robustness of the observer to uncertainty in model parameters and violation of model assumptions. This comes dat the cost of losing some accuracy in normal driving conditions.