To be able to understand the dynamic driving environment, an autonomous vehicle needs to predict the mo- tion of other traffic participants in the driving scene. Motion prediction can be done based on experience and recently observed series of past events, and entails reasoning about probable outcomes with these past ob- servations. Aspects that influence driving behavior comprise many factors, such as general driving physics, infrastructure geometry, traffic rules, weather and so on. Models that are used today are incapable of including many of these aspects. These deep learning models often rely heavily on the past driven trajectory as an information source for future prediction. Sparsely, some work has been done to include interaction between vehicles or some infrastructural features to the model. The lack of information regarding the driving scene can be seen as a missed opportunity, because it is a use- ful feature source to predict the motion of a vehicle. Especially when a map of the driving scene is available, reliable information regarding the road layout can be extracted. In this thesis, a model architecture is sought that is aware of the interaction between vehicles, and that un- derstands the geometry of the roads in the scene. Importantly, map information regarding the driving scene should be used as the primary source of information regarding the road geometry. First, a baseline deep learning model is constructed, that can generate predictions based on the past observed trajectory. To add interaction features to the model, a social pooling module is introduced. The social pooling method allows to efficiently include the driving behavior of other vehicles in the scene. To introduce reliable, map-based road information to the model, two novel methods are proposed. In the first, the predictions from a model are used to extract features in a map. These features describe the road scene around the predicted location, and are used to update these predictions. In the second method, the semantic map is only used to extract a road segment ahead of the vehicle. A road-RNN is introduced to con- struct features regarding the road segment, and an attention mechanism to determine what part of the road segment is relevant for the predictions. These modules are referred to as road-refinement and road-attention respectively. The importance of including both road geometry and interaction methods in the model is shown by con- structing 5 different models that vary in their road-geometric and interaction awareness. A baseline deep learning model is used and extended with a road-geometry module, an interaction module, a combination of the two, or none. To test the prediction capabilities of the models, they are trained on two different datasets. The first dataset, called i80, consists of trajectory recordings from a straight highway with dense traffic. The other dataset is a curved version of the i80, called i80c, where the trajectories and road are transformed to introduce road-geometric variations in the data. The prediction performances from the models clearly show the importance of both interaction-aware and road-geometry aware modeling. The road-refinement and road-attention models outperform the road-agnostic baseline model on the i80c data, showing better understanding of the road layout. Comparison between these models learns that the road-attention model is more effective for road-geometry inclusion. Additionally, the performance increase for interaction-aware models compared to the baseline clearly shows the importance of interaction in modeling on both datasets.The model that combines the interaction and road-attention modules shows outstanding prediction performance on the challenging i80c dataset compared to all other models. From the predictions it can be seen that the model understands the road layout ahead of the subject vehicle, as well as the interactive forces that are in play.

Autonomous driving is a development that has gained a lot of attention lately, because it can lead to major improvements in the mobility sector. One example of a research project that aims to develop vehicles that are capable of reaching the highest level of autonomy in driving, is the WEpods project. The goal of this research is in line with this aim, having the thesis objective defined as follows: let the WEpods continue driving in autonomous mode more often than is currently the case. The WEpod shuttles are not yet completely able to drive autonomously due to their inability to handle unexpected behavior (terminology: faults). Currently, such faults need to be detected and solved by a steward, who will manually initiate a safe stop if necessary. The localization module, which is responsible for localizing the vehicle on a map, sometimes generates unreliable location estimates. This poses two challenges. First, the fact that there is a mismatch between reality and the sensor outcomes of the localization module that needs to be detected. Second, the question of how to prevent the system from showing behavior that is different from what is desired (terminology: failure) in case such a fault is present (terminology: fault tolerant contol). Fault tolerant control can be performed in either a passive or an active manner. The passive approach ensures that either the faults are prevented or the system is able to mitigate them by anticipation in the design. The approach evolves from passive to active fault tolerant control when an on-line adaptation of the system control is made. For applications in autonomous driving, it is apparent that it is important to handle not only anticipated faults, but also to be able to deal with unexpected faults in an on-line manner. This on-line fault tolerant control approach involves two fault diagnosis steps that lead to solving the first challenge: detection and isolation. A so-called model-based fault diagnosis approach turned out to be most suitable, as it has been used for similar applications in the past. However, a model-based fault diagnosis approach has not yet been implemented for detecting and isolating faults in a localization module of autonomous driving, indicating the scientific relevance of this research. In the model-based approach, kinematic and dynamic equations of the research vehicle (WEpod) are used to build a computational model. This model is then subjected to an observer, that is able to compare the model outcomes with the actual measurements in an off-line way. A residual is drawn up by taking the difference between the model outcomes and the measurements. A threshold is computed based on noise on the measurements to compare the residual with. When the residual exceeds the threshold, an alarm is raised. This way, the system itself has been enabled to detect faults when they occur internally. Inclusion of the suggested fault diagnosis approach in an on-line manner into the system is a big step towards fully autonomous driving of the WEpods, and therefore the goal of this research is met.

To bring down the number of traffic accidents and increase people’s mobility companies, such as Robot Engineering Systems (RES) try to put automated vehicles on the road. RES is developing the WEpod, a shuttle capable of autonomously navigating through mixed traffic. This research has been done in cooperation with RES to improve the localization capabilities of the WEpod. The WEpod currently localizes using its GPS and lidar sensors. These have proven to be not accurate and reliable enough to safely navigate through traffic. Therefore, other methods of localization and mapping have been investigated. The primary method investigated in this research is monocular Simultaneous Localization and Mapping (SLAM). Based on literature and practical studies, ORB-SLAM has been chosen as the implementation of SLAM. Unfortunately, ORB-SLAM is unable to initialize the setup when applied on WEpod images. Literature has shown that this problem can be solved by adding depth information to the inputs of ORB-SLAM. Obtaining depth information for the WEpod images is not an arbitrary task. The sensors on the WEpod are not capable of creating the required dense depth-maps. A Convolutional Neural Network (CNN) could be used to create the depth-maps. This research investigates whether adding a depth-estimating CNN solves this initialization problem and increases the tracking accuracy of monocular ORB-SLAM. A well performing CNN is chosen and combined with ORB-SLAM. Images pass through the depth estimating CNN to obtain depth-maps. These depth-maps together with the original images are used in ORB-SLAM, keeping the whole setup monocular. ORB-SLAM with the CNN is first tested on the Kitti dataset. The Kitti dataset is used since monocular ORB- SLAM initializes on Kitti images and ground-truth depth-maps can be obtained for Kitti images. Monocular ORB-SLAM’s tracking accuracy has been compared to ORB-SLAM with ground-truth depth-maps and to ORB-SLAM with estimated depth-maps. This comparison shows that adding estimated depth-maps in- creases the tracking accuracy of ORB-SLAM, but not as much as the ground-truth depth images. The same setup is tested on WEpod images. The CNN is fine-tuned on 7481 Kitti images as well as on 642 WEpod images. The performance on WEpod images of both CNN versions are compared, and used in combination with ORB-SLAM. The CNN fine-tuned on the WEpod images does not perform well, missing details in the estimated depth-maps. However, this is enough to solve the initialization problem of ORB-SLAM. The combination of ORB-SLAM and the Kitti fine-tuned CNN has a better tracking accuracy than ORB-SLAM with the WEpod fine-tuned CNN. It has been shown that the initialization problem on WEpod images is solved as well as the tracking accuracy is increased. These results show that the initialization problem of monocular ORB-SLAM on WEpod images is solved by adding the CNN. This makes it applicable to improve the current localization methods on the WEpod. Using only this setup for localization on the WEpod is not possible yet, more research is necessary. Adding this setup to the current localization methods of the WEpod could increase the localization of the WEpod. This would make it safer for the WEpod to navigate through traffic. This research sets the next step into creating a fully autonomous vehicle which reduces traffic accidents and increases the mobility of people.

The development of domestic mobile manipulators for unconstrained environments has driven significant research recently. Robot Care Systems has been pioneering in developing a prototype of a mobile manipulator for elderly care. It has a 6 degrees of freedom robotic arm mounted on their flagship robot LEA, a non-holonomic differential drive platform. In order to utilize the navigation and manipulation capabilities of such mobile manipulators, robot placement algorithm that computes a favorable position and orientation of the mobile base is sought, which enables the end effector to reach a desired target. None of the existing approaches perform robot placement while ensuring a high chance of successful planning to target through a short path, while accounting for sensing and actuation errors typical in real world scenarios. This thesis presents a novel robot placement algorithm DeCOWA (Determining Commutation configuration using Optimization and Workspace Analysis) with these characteristics. Since the approach to robot placement is dependent upon the kind of mobile manipulation, a comparative study of sequential and full body methods is performed with respect to criteria important in domestic settings. Sequential mobile manipulation is found to be most suitable, for which a modular mobile manipulation framework encompassing motion planning and robot placement is presented. With sequential mobile manipulation, the ability to successfully reach a target depends upon the kinematic capabilities of the arm. Accordingly, robot placement with DeCOWA determines a favorable location for the arm, and corresponding platform orientation. To find the position of arm’s base, an offline manipulator workspace analysis is performed generating the Inverse Reachability and Planability maps. During online use, these maps are combined into an Inverse Fusion Map that ranks different locations based on the ability of the arm placed there to find a successful and short motion plan to target. This map is filtered to generate a set of feasible locations at the arm’s height. Through a ranked iterative search, a suitable collision free arm location is determined followed by minimization of the platform distance from robot’s current pose. This approach is evaluated against an unbiased random placement of robot near the target using a sample set of twenty scenes mimicking domestic settings. It is found that DeCOWA is able to generate commutation configurations in fraction of a second, that lead to a high planning success rate, a short path length, and account for goal tolerance of navigation. Also, its modularity allows to use several planability metrics, making it useful for domestic application.

An autonomous vehicle should be able to operate amidst numerous other human-driven vehicles, each driving on its own trajectory. To safely navigate in such a dynamic environment, the autonomous vehicle should be able to predict trajectories of the vehicles operating in its vicinity and use these to plan its own path. Most related work uses a vehicle's past trajectory to model its behavior, based on which the future trajectory is predicted. However, they do not focus on the influence of contextual features such as road structure from the scene that may affect the vehicle's future trajectory. This work proposes an approach to predict a long-term vehicle trajectory using not only the past trajectory of a vehicle but also contextual features from the driving scene. We model the road structure to help prediction on curved road sections. A Recurrent Neural Network is used to learn vehicle behavior from past vehicle trajectories and predict future trajectories while incorporating road structure. Using a trajectory dataset collected from a test vehicle, we compare our model's performance with the conventional prediction approach based on only past vehicle trajectory.