To be able to understand the dynamic driving environment, an autonomous vehicle needs to predict the mo- tion of other traffic participants in the driving scene. Motion prediction can be done based on experience and recently observed series of past events, and entails reasoning about probable outcomes with these past ob- servations. Aspects that influence driving behavior comprise many factors, such as general driving physics, infrastructure geometry, traffic rules, weather and so on. Models that are used today are incapable of including many of these aspects. These deep learning models often rely heavily on the past driven trajectory as an information source for future prediction. Sparsely, some work has been done to include interaction between vehicles or some infrastructural features to the model. The lack of information regarding the driving scene can be seen as a missed opportunity, because it is a use- ful feature source to predict the motion of a vehicle. Especially when a map of the driving scene is available, reliable information regarding the road layout can be extracted. In this thesis, a model architecture is sought that is aware of the interaction between vehicles, and that un- derstands the geometry of the roads in the scene. Importantly, map information regarding the driving scene should be used as the primary source of information regarding the road geometry. First, a baseline deep learning model is constructed, that can generate predictions based on the past observed trajectory. To add interaction features to the model, a social pooling module is introduced. The social pooling method allows to efficiently include the driving behavior of other vehicles in the scene. To introduce reliable, map-based road information to the model, two novel methods are proposed. In the first, the predictions from a model are used to extract features in a map. These features describe the road scene around the predicted location, and are used to update these predictions. In the second method, the semantic map is only used to extract a road segment ahead of the vehicle. A road-RNN is introduced to con- struct features regarding the road segment, and an attention mechanism to determine what part of the road segment is relevant for the predictions. These modules are referred to as road-refinement and road-attention respectively. The importance of including both road geometry and interaction methods in the model is shown by con- structing 5 different models that vary in their road-geometric and interaction awareness. A baseline deep learning model is used and extended with a road-geometry module, an interaction module, a combination of the two, or none. To test the prediction capabilities of the models, they are trained on two different datasets. The first dataset, called i80, consists of trajectory recordings from a straight highway with dense traffic. The other dataset is a curved version of the i80, called i80c, where the trajectories and road are transformed to introduce road-geometric variations in the data. The prediction performances from the models clearly show the importance of both interaction-aware and road-geometry aware modeling. The road-refinement and road-attention models outperform the road-agnostic baseline model on the i80c data, showing better understanding of the road layout. Comparison between these models learns that the road-attention model is more effective for road-geometry inclusion. Additionally, the performance increase for interaction-aware models compared to the baseline clearly shows the importance of interaction in modeling on both datasets.The model that combines the interaction and road-attention modules shows outstanding prediction performance on the challenging i80c dataset compared to all other models. From the predictions it can be seen that the model understands the road layout ahead of the subject vehicle, as well as the interactive forces that are in play.

Predicting when a neonate will fall victim to an infection or a disease allows prevention through early medicine administering. Such physiological conditions can be made visible using infrared thermography (IRT). This is a technique for measuring heat emitted in the infrared spectrum and transforming them into visible signals that can be recorded photographically. This thesis will contribute to the prediction of infection in (pre)term neonates by quantifying the interaction between IRT and a neonatal incubator without (and with) a neonate in it.  A system was designed that consisted of three modules: a measurement (incubator and IRT camera), back-end (embedded system and server), and front-end module. The scope of this thesis is limited to the measurement module and the embedded system of the back-end module. Minimum camera requirements were set up which required the camera to: be inexpensive (i.e. ≤ €1000,-), be mobile, be open-source (for Linux), have a minimum frames-per-second of 5, have a resolution of at least 160x160 pixels with a field of view (FOV) of 27°, sensitivity of < 0.1°C, and safe to the patient. Such a camera was found in the FLIR One Pro. For this thesis a different FLIR camera was used due to lack of budget, namely the FLIR A305sc, which was already available at the TU Delft. The A305sc is not open-source, which required a work-around. The Aravis Open Source Project allowed for communication with the camera. Internal camera parameters had to be determined to calculate temperature based on analogue-to-digital values. ExifTool was used on a file stored by the camera to extract these parameters. This calculated temperature was compared to the temperature as determined by FLIR’s software and led to a difference in the range of 1·106°C. An open-source application was written that can connect with this IRT camera that has a GenICam interface using Aravis. Additionally, this application implemented the temperature calculation based on the internal camera parameters. The hood of the incubator is opaque to infrared, which required the design of a measurement setup to circumvent this. Three different setups were discussed, with the final choice falling on placing the camera in front of an opened incubator porthole on a tripod, and sealing this porthole with high or low density ethyl polyethylene (HDPE/LDPE). Regular H/LDPE used for construction site was found to have a attenuated transmissivity as found in literature. To quantify the interaction between IRT and a neonatal incubator, the IRT measurements were to be compared against the current golden standard sensor, namely thermistors. These sensor values were to be read out from the incubator as this would also be used in the final product. Code was written which allows for automatic detection between the GE GiraffeTM Omnibed and the Dräger Caleo® incubator, automatic connecting, and manipulation of all sensors values to a standard string which allows for easy uploading to the InfluxDB database on the server. To be allowed to perform measurements on human subjects, approval had to be acquired by the human research ethics committee (HREC) of the TU Delft and the respective hospital. A “non-wet medisch wetenschappelijk onderzoek met mensen” (nWMO) request was submitted and approved, which resulted in 25 recorded sick and healthy neonates in incubators divided over two hospitals (the JKZ in The Hague, and the RDGG in Delft), with over 25 hours of recording material. Simultaneously, measurements were performed on an empty incubator to gain an understanding in the behaviour of an incubator when actors from outside interacted with the internal environment. Measurements that were performed included determining the reflected apparent temperature (RAT) for every possible opened porthole and for both incubator types. The RAT for the Caleo was found to be higher for every measurement for the GE. The accuracy of the IRT and hospital skin temperature sensors was compared against a calibrated Pt-100 sensor, which show that the Pt-100 sensor measures an equal value as the hospital skin temperature sensor, whereas The IRT camera measured .6°C higher. The effect of changing the distance on IRT values was measured, which shows that for a distance of 0.2m to 1.2m the accuracy of the IRT camera is within the specified accuracy. Finally, the effect of opening additional portholes on IRT was measured, the effect of the airboost setting on IRT, and the measurement of opening additional portholes was repeated with a different IRT camera. Overall the IRT camera measures a higher temperature than the hospital skin temperature sensors, but follows the skin temperature sensors’ pattern.    

Many robotic systems rely on laser ranging sensors to navigate and map their environment. As robots are getting more advanced, smaller and cheaper, their sensors need to decrease in size and cost, while increasing their reliability and accuracy. Current laser-based sensors have difficulty to meet these properties. They are either large, expensive, contain moving parts or provide insufficient amounts of information. This thesis introduces an integrated approach for the development of an eye-safe, low-cost, solid-state, wide-angle line-laser distance sensor, tackling these challenges. A prototype is derived by constructing a generic, camera independent, triangulation model. The model, combined with a set of pre-defined requirements, is used to select hardware components and predict the sensor limits. A unified calibration step is proposed to estimate both camera intrinsics as well as misalignment errors with the same set of data. Additionally, a microsecond-accurate open loop synchronization system for laser activation and imager exposure is presented. Tests show that the prototyped sensor is able to measure distances up to 3 meters with an error of 4%. At 2 meter, the error is just under 2%. Additionally, the solid-state prototype has a field-of-view of 105 degrees, an angular resolution of 0.8 degrees, an update rate of 10Hz and an estimated cost of just below $35.

Parts of the Southwest Palace of Sennacherib (700 BC), which were located in Nineveh (Iraq), are now destroyed by the ongoing conflict in Iraq and Syria. The palace rooms used to be decorated with numerous Assyrian reliefs. Luckily, digital pictures of the site are available thanks to an Italian expedition in 2002. The goal of this research is to physically reproduce the destroyed reliefs using this photo database. First photogrammetry was used to rebuild the global dimensions of the reliefs. The details are reconstructed by fusing a highly detailed depth map retrieved with Shape from Luminance (SFL), together with a coarser Artificial Reconstruction. The fused reconstruction was evaluated with the eigenvalues of the Hessian. The eigenvalues give an intuitive measure about the shapes of the reliefs. The RMSE of the results improve up to 44.4% compared to the usage of an intensity (grey) image. The final 3D models are used to produce full size reproductions with 3D printing and CNC milling.

To bring down the number of traffic accidents and increase people’s mobility companies, such as Robot Engineering Systems (RES) try to put automated vehicles on the road. RES is developing the WEpod, a shuttle capable of autonomously navigating through mixed traffic. This research has been done in cooperation with RES to improve the localization capabilities of the WEpod. The WEpod currently localizes using its GPS and lidar sensors. These have proven to be not accurate and reliable enough to safely navigate through traffic. Therefore, other methods of localization and mapping have been investigated. The primary method investigated in this research is monocular Simultaneous Localization and Mapping (SLAM). Based on literature and practical studies, ORB-SLAM has been chosen as the implementation of SLAM. Unfortunately, ORB-SLAM is unable to initialize the setup when applied on WEpod images. Literature has shown that this problem can be solved by adding depth information to the inputs of ORB-SLAM. Obtaining depth information for the WEpod images is not an arbitrary task. The sensors on the WEpod are not capable of creating the required dense depth-maps. A Convolutional Neural Network (CNN) could be used to create the depth-maps. This research investigates whether adding a depth-estimating CNN solves this initialization problem and increases the tracking accuracy of monocular ORB-SLAM. A well performing CNN is chosen and combined with ORB-SLAM. Images pass through the depth estimating CNN to obtain depth-maps. These depth-maps together with the original images are used in ORB-SLAM, keeping the whole setup monocular. ORB-SLAM with the CNN is first tested on the Kitti dataset. The Kitti dataset is used since monocular ORB- SLAM initializes on Kitti images and ground-truth depth-maps can be obtained for Kitti images. Monocular ORB-SLAM’s tracking accuracy has been compared to ORB-SLAM with ground-truth depth-maps and to ORB-SLAM with estimated depth-maps. This comparison shows that adding estimated depth-maps in- creases the tracking accuracy of ORB-SLAM, but not as much as the ground-truth depth images. The same setup is tested on WEpod images. The CNN is fine-tuned on 7481 Kitti images as well as on 642 WEpod images. The performance on WEpod images of both CNN versions are compared, and used in combination with ORB-SLAM. The CNN fine-tuned on the WEpod images does not perform well, missing details in the estimated depth-maps. However, this is enough to solve the initialization problem of ORB-SLAM. The combination of ORB-SLAM and the Kitti fine-tuned CNN has a better tracking accuracy than ORB-SLAM with the WEpod fine-tuned CNN. It has been shown that the initialization problem on WEpod images is solved as well as the tracking accuracy is increased. These results show that the initialization problem of monocular ORB-SLAM on WEpod images is solved by adding the CNN. This makes it applicable to improve the current localization methods on the WEpod. Using only this setup for localization on the WEpod is not possible yet, more research is necessary. Adding this setup to the current localization methods of the WEpod could increase the localization of the WEpod. This would make it safer for the WEpod to navigate through traffic. This research sets the next step into creating a fully autonomous vehicle which reduces traffic accidents and increases the mobility of people.

In computer vision pose estimation of objects in everyday scenes is a basic need for a clearunderstanding of the surrounding environment, fields of interests include; augmented reality,surveillance, navigation, manipulation, and robotics in general. Pose estimation is a wellstudied topic, however fast and robust solutions are still hard to obtain. The goal of thisresearch is to robustly and efficiently perform 3D pose estimation of multiple objects withina single RGB image in real-time (>24framespersecond (fps)). To achieve this goal an existing CNN is utilized, more specifically the YOLO network is used.This provides a stable platform for object detection and classification, the network is onlyslightly modified to include pose regression. The YOLOv2 network was originally designedto generate bounding boxes around objects and classify the objects within the boundingboxes. This research shows that using a single confidence value rather than 4 boundingbox parameters is sufficient to determine the relative location of objects within the image,limiting the number of parameters that need to be trained. This has allowed to make thenetwork more efficient and to make the network focus more on training the pose parameters(azimuth,elevation and distance) rather than the bounding box parameters. Using several techniques like data augmentation, data clustering and data selection the state-of-the-art AVP of 50.1% was achieved on the azimuth estimation problem. For the full 3Dpose (azimuth, elevation and distance) problem the AVP is limited to 30.1%, with no directcomparison this is still considered to be state-of-the-art. However, normalizing the confidenceoutput of each image in the post-processing step has increased this accuracy further, improvingbeyond the state-of-the-art results. With a normalization step the AVP score has reached 63.0% and 40.4% respectively. These results show that the pose estimation problem hasimproved significantly, getting closer to a viable solution for real-world applications. However, further improvements can be made in the post-processing step, the data augmenta-tion step and the data selection step. The research conducted has shown that accuracy gainsare not only achieved through better network architecture but are also highly dependent onthe training and processing techniques used. This is evident from the accuracy increase of 38% from an AVP of 25% to an AVP of 63%. Optimizing these techniques specifically for theYOLOposearchitecture, might result in a solution that can be used in real-world applications.

Our world is rapidly changing, leading to various technological achievements in diverse areas of expertise. Among the most evolving are autonomous robots, machines and vehicles, enormously extending human capabilities. While this trend became quite familiar for land-based vehicles, innovations in the underwater domain lag behind, mainly due to the harsh environmental conditions encountered. This in large contrast with worldwide growing needs for autonomous underwater solutions, like dredging. Nevertheless, for underwater vehicles becoming autonomously, various technological challenges must be overcome. The most essential one is finding a proper answer to the question; where in the underwater world am I? Autonomous robots are usually be able to operate in complex environments using external reference systems such as Global Positioning System (GPS) to locate themselves inside their environment. However, these are not accessible in underwater applications, since water strongly attenuates electromagnetic signals, notably GPS. Hence, various alternative methods for Simultaneous Localization And Mapping (SLAM) for underwater applications are employed, among which the method of anchoring inertial measurements on environmental landmarks, perceived with an exteroceptive Sonar. The ultimate goal of this procedure is to correct for errors in the inertial measurements, using the environmental perception available in the Sonar scans. Nevertheless, due to practical underwater issues, it still is not readily available and even barely investigated in open scientific research. This thesis aims at the development of an underwater localization algorithm, using onboard inertial sensors and Sonar. The overall system design consists of several individual software procedures, executing certain assignments, together ultimately solving for the localization task. The developed system was tested using ground truth in form of a real-world dataset obtained several years ago in an abandoned marina. This showed that most individual procedures were able to provide good results, while having a low computational complexity in relation to online operations. However, it is quite a challenge to obtain results accurate enough to properly correct for positional errors, by using the designed system. Although showing potential, it still is not entirely decent in terms of accuracy and computational burden. This is likely due to the sparsity of the extracted Sonar measurements. Data is lacking to further verify the system, hence several questions remain open regarding its universal applicability in case of different scenarios, dynamic environments and long trajectories. In this context, more in-depth research is needed into the algorithms that are influenced by the Sonar measurements, to increase the accuracy of the overall system, while gaining more insight in its computational behaviour. Additionally, also more real-world experiments are essential, to extensively verify the designed system. In conclusion, the results showed that it is possible to correct for inertial errors, using the system developed. Furthermore, it demonstrates that most procedures individually produce good results in real-time. Hence, this study can be seen as a positive step in the right direction, forming a basis for future research in solutions that are generically applicable in online real underwater world operations.

New measures have to be taken to combat fatalities caused by traffic accidents. Intelligent vehicles have the potential to increase safety, but depend heavily on their automated perception ability. Acoustic perception, an unused sensing modality in this field, has potential for the detection of nearby vehicles, an ability both human drivers and autonomous vehicles could use assistance with. In this thesis two existing datasets, AudioSet a large general purpose dataset and RoadCube a small dedicated vehicle recognition set, are evaluated. Furthermore commonly used acoustic features and classifier algorithm are evaluated. Special attention is given to the influence of a moving listener vehicle on the performance. For the evaluation a new dataset, DriveSound, is captured. It contains samples captured from a listener car, both when its moving or idle. Results show that RoadCube can be used for the detection of road vehicles, but only when the listener is idle. The best performing classifier from RoadCube, a Gaussian Mixture Model classifier surpassed classifiers trained on the evaluation dataset itself with a Matthews Correlation Coefficient (MCC) of 0.34. None of the classifiers performed well on the samples captured by a moving listener, except for the DriveSound-driving classifiers. The Support Vector Machine trained on this dataset attained a MCC of 0.56.

This thesis presents a visual route following method that minimizes memory consumption to the point that even Micro Aerial Vehicles (MAV) equipped with only a simple microcontroller can traverse distances of a few hundred meters. Existing Simultaneous Localization and Mapping (SLAM) algorithms are too complex for use on a microcontroller. Instead, the route is modeled by a sequence of snapshots that can be followed back using a combination of visual homing and odometry. Three visual homing methods are evaluated to find and compare their memory efficiency. Of these methods, Fourier-based homing performed best: it still succeeds when snapshots are compressed to less than twenty bytes. Visual homing only works from a small region surrounding the snapshot, therefore odometry is used to travel longer distances between snapshots. The proposed route following technique is tested in simulation and on a Parrot AR.Drone 2.0. The drone can successfully follow long routes with a map that consumes only 17.5 bytes per meter.

An autonomous vehicle should be able to operate amidst numerous other human-driven vehicles, each driving on its own trajectory. To safely navigate in such a dynamic environment, the autonomous vehicle should be able to predict trajectories of the vehicles operating in its vicinity and use these to plan its own path. Most related work uses a vehicle's past trajectory to model its behavior, based on which the future trajectory is predicted. However, they do not focus on the influence of contextual features such as road structure from the scene that may affect the vehicle's future trajectory. This work proposes an approach to predict a long-term vehicle trajectory using not only the past trajectory of a vehicle but also contextual features from the driving scene. We model the road structure to help prediction on curved road sections. A Recurrent Neural Network is used to learn vehicle behavior from past vehicle trajectories and predict future trajectories while incorporating road structure. Using a trajectory dataset collected from a test vehicle, we compare our model's performance with the conventional prediction approach based on only past vehicle trajectory.

Recent advancements in computation power and artificial intelligence have allowed the creation of advanced reinforcement learning models which could revolutionize, between others, the field of robotics. As model and environment complexity increase, however, training solely through the feedback of environment reward becomes more difficult. From the work on robotic priors by R.Jonschkowski et al. we present robotic auxiliary losses for continuous reinforcement learning models. These function as additional feedback based on physics principles such as Newton’s laws of motion, to be utilized by the reinforcement learning model during training in robotic environments. We furthermore explore the issues of concurrent optimization on several losses and present a continuous loss normalization method for the balancing of training effort between main and auxiliary losses. In all continuous robotic environments tested, individual robotic auxiliary losses show consistent improvement over the base reinforcement learning model. The joint application of all losses during training however did not always guarantee performance improvements, as the concurrent optimization of several losses of different nature proved to be difficult.

The subject of this thesis is to investigate whether the Cerebellar Model Articulation Controller (CMAC) can be used to anticipate controller corrections and increase performance by reducing delays in humanoid robots. This question can be divided into two subquestions. Firstly, whether the CMAC is a suitable architecture for the prediction of controller actions for a humanoid soccer robot. Using a 2D model of a robotic leg, the results of this thesis show that the CMAC can indeed learn to anticipate a corrective control signal 30 ms ahead. Secondly, whether the architecture of the aforementioned setup can increase the performance of adequately passing a ball by reducing delays. The experiments show that the use of a CMAC can increase the performance of the robotic setup.

One of the remaining challenges in the development of intelligent vehicles is the topic of behavior planning in urban scenarios. Based on perception of the environment around the intelligent vehicle, driving behavior has to be optimized to achieve a comfortable driving experience without sacrificing safety. This work covers the development of a novel longitudinal behavior planning method for an intelligent passenger vehicle in urban scenarios. A multi-layer situation analysis architecture consisting of localization, maneuver prediction and trajectory prediction is used to predict traffic object trajectories on urban intersections and roundabouts. A universal model predictive controller is developed to determine the optimal longitudinal acceleration behavior of the intelligent vehicle online in a variety of urban scenarios, based on the predicted trajectories. Real-time simulations show that the behavior planning method is able to maintain safe driving conditions with reasonable comfort during scenarios with varying traffic maneuvers.