Pose estimation provides accurate position and orientation information of the intelligent agents in real time. The accuracy of the estimation directly affects the performance of sequential tasks such as mapping, motion planning, and control. EKF (Extended Kalman Filter) is a standard theory for nonlinear pose estimation by modeling state uncertainty to Gaussian distribution. However, EKF has requirements for proper initial estimate and system noise to obtain bounded optimal estimate. Meanwhile, model nonlinearity and non-gaussian noise modeling affect the performance of EKF significantly in practical applications. In this thesis, we focus on improving the performance of nonlinear pose estimation by reinforcement learning. By formulating an EKF measurement update as a Markov Decision Process (MDP), reinforcement learning agents can be trained to learn the estimator gain through data samples and executed as the online estimator for pose estimation tasks. Based on the above idea, we propose a novel reinforcement learning-compensated EKF estimator (RLC-EKF), where the RL agent serves as a second-time measurement update that subsides the residual error from the standard EKF estimate. The estimator is developed and testified on two specific pose estimation scenarios. Firstly, as a continuous work from the previous study, a framework for 3 DOF orientation estimation using inertial sensor and magnetometer is replicated. Then, the framework is extended by different RL algorithms training and multi-scale robustness validation. Besides, we implement the estimator on a feature-based 2D plane localization framework. The proposed framework shows the feasibility of the underlying algorithm on a localization task with a known map. As a result, the RLC-EKF estimator gives superior performance and convincible robustness compared to classical methods in severe conditions such as varying initial states, degree of noise intensities, and model covariance.

Object pushing in robotics has numerous applications, but it often relies on room-bound object tracking systems such as Motion Capture (MoCap) for accurate object pose acquisition. Such systems limit the potential use scenarios, since they add complexity and cost and require expansion of the sensor infrastructure for expanding the operational areas. To address these limitations, we propose a framework that integrates object pushing and vision-guided object pose estimation using a monocular RGB camera that is placed on the pushing agent. By leveraging the temporal coherence of the scene, we accelerate the 2D object detection stage that is common in the field of object pose estimation. Our contributions include the development of a novel framework that continuously pushes an object while simultaneously tracking its pose. We incorporate dynamics by efficiently using a Model Predictive Path Integral controller’s predictive model to predict the object location in the forthcoming image, resulting in an extreme reduction of computational cost for 2D object detection, without sacrificing the accuracy of the pose estimation. Our framework achieves real-time object pushing with accurate object pose estimation, which we demonstrate with real-world experiments. Furthermore, we provide a novel object pose estimation dataset in the widely used BOP format, specifically for robot planar pushing with an on-board camera. Our approach pushes towards eliminating the need for room-bound sensor systems, expanding the potential use cases. By considering temporal coherence and scene dynamics, our framework challenges recent object pose estimation methods that fully process each image as uncorrelated individuals, while providing a promising solution for real-time object pushing.

Simulation environments are useful for a wide range of applications and their functionalities continue to improve every year. The aim of this thesis project was to create a simulation environment with high levels of realism and assess its capabilities through the use case of generating distributed drone traffic rules. This thesis project was executed at the company Almende in cooperation with its sub-company DoBots. The distributed drone traffic rule generation use case was a contribution to the ADACORSA project which Almende takes part in. The goals of the thesis were the following. A simulation environment incorporating elements of realism was created using the software tools Almende has experience with. The capabilities of this environment were tested through the said use case. According to company requirements, this process was realised using an incremental complexity approach, and it utilized machine learning techniques. It was expected that the incremental complexity approach would help speed up the learning process of more complex scenarios, while machine learning would provide the benefit of smart automation and impartiality to solutions. For this process, a model was also devised for creating such rules, incorporating the incremental complexity approach and a standard machine learning technique. Finally, the benefits of the incremental complexity approach were also assessed. The four building blocks of the implemented simulation environment were Gazebo, PX4, ROS2 and the iris drone model. The code was built upon the OpenAI ROS software package, with extensive modifications. The main work done during the implementation consisted of multiple parts. The original software package was cleaned of unnecessary elements, the PX4 software and the iris drone model were integrated into the environment and the simulation speed was increased. The whole setup was migrated from ROS to ROS2. The use of multiple drones in the same environment was also solved. An implementation of the deep Q-learning algorithm, which was selected as the learning method, was added. Different scenarios were created, as well as some logging and visualization tools. Additionally, the system was set up to run in docker containers and was highly parameterized for future use. Different elementary traffic scenarios were implemented in the simulation and then executed. Based on these the following points were determined. It was concluded that the created simulation environment was well-fitted for running scenarios for traffic rule generation. It was also suitable for executing the scenarios using the incremental complexity approach, although its usability also depended on how the neural networks of the scenarios were set up. The environment was also feasible to work with machine learning from an engineering viewpoint, however, it did not perform well for its scientific research. This was due to not having statistically well-supported results, which was caused by the long time it took to execute a scenario. The benefits of the incremental complexity approach were also examined. For the simple case tackled in this project, it was found that the incremental complexity approach provided no significant advantages with regard to learning speed.

Mobile robots are getting more common in warehouses, distribution centers and factories, where they are used to boost productivity. At the same time, they are moving into the everyday world, where they need to operate in uncontrolled and cluttered environments. In order to extend the set of tasks that a robot can autonomously accomplish in such environments, we can equip mobile bases with a pushing skill. Possible use cases are delivering packages by pushing objects to a goal location, or clearing a path by pushing movable obstacles out of the way. This thesis considers pushing with a non-holonomic mobile robot in cluttered environments. We develop a learning-based method, based on ensembles of probabilistic neural networks for learning the object’s dynamics. The models capture the inherent stochasticity of pushing from the variability of friction. In addition, the ensembles can capture the uncertainty that arises from a lack of data, so we can reason about which regions to avoid during the pushing and do not come into unrecoverable states. The model is combined with a Model Predictive Path Integral (MPPI) controller. Each timestep the controller optimizes for a finite time horizon, which allows us to recover from slight model inaccuracies. The MPPI controller is capable of handling complex cost functions, which allows us to include the estimated uncertainty from the model to avoid uncertain regions. For obstacle-aware pushing, we provide the MPPI controller with a costmap grid, that contains the free pushing space, which the robot and object can use to maneuver to the goal location. Both the robot and object are constrained to keep within the free pushing space, which is constructed using LiDAR measurements. We verify the method in simulation and compare it to two baseline methods: the state-of-the-art adaptive pushing controller proposed by Krivic et al. [31] and a pushing policy trained with the model-free reinforcement learning baseline Soft Actor-Critic (SAC). All three baselines reach comparable success rates in simulation of 100%, 98% and 96%, respectively. Furthermore, the learning-based MPPI shows improvements of 50% and 23% in accuracy compared to the baselines. The results are consistent for pushing in cluttered environments, where we show an increase of accuracy by 43%, compared to Krivic et al. [31]. Lastly, we validate the approach in a real-world setting, where the robot and object are tracked with a motion capture system. The approach has an overall success rate of 94% and performs successful pushing in a cluttered environment.

Simultaneous Localisation and Mapping (SLAM) provide a novel solution for the robots to localise and navigate an unknown environment. Initial SLAM research focused mainly on the indoor environment, assuming the background to be primarily static. In contrast, the real world has dynamic interactions that restrict the implementation of SLAM to limited scenarios. This brings a higher requirement to deal with the moving objects in dynamic environments for robust SLAM performance. Semantic understanding of the environment helps in filtering out the influence of dynamic objects in the vicinity. An instance segmentation based on two-stage neural architecture is used for this purpose, which is hard to operate in real-time navigation. In this project, the benefits of single stage neural architecture are studied in terms of speed and accuracy for improving the efficiency of dynamic features removal in the application of SLAM. Although Instance segmentation architecture helps to identify the potentially dynamic object by learning from the dataset, it cannot differentiate moving objects from non-moving objects in the dynamic class. Hence, all the features corresponding to the predicted dynamic class are removed even when the objects remain stationary, affecting the quality of SLAM performance. A two-stream encoder-decoder architecture is developed to segment the moving masks using RGB and optical flow input, improving feature tracking without affecting robustness. The feasibility of encoding dynamic information to enhance quality semantic mapping is also studied.