Task and Motion Planning (TAMP) has progressed significantly in solving intricate manipulation tasks in recent years, but the robust execution of these plans remains less touched. Particularly, generalizing to diverse geometric scenarios is still challenging during execution. In this work, we propose a reactive TAMP method to deal with disturbances and geometric ambiguities by combining an active inference planner (AIP) for online action selection with a proposed multi-modal model predictive path integral controller (M3P2I) for low-level control. The AIP generates online alternative plans, each of which is translated into a cost function to be sampled for the proposed method. The proposed M3P2I then uses a parallelizable physics simulator for throwing different rollouts, leading to a coherent optimal solution by averaging the weighted samples based on their costs. Our method empowers real-time adaptation of action sequences to rectify failed plans, while also computing low-level motions to address dynamic obstacles or disturbances that could potentially invalidate the existing plan.

Theoretical findings are validated in simulation and in the real world. We show that the framework exhibits reactiveness in different scenarios, including battery charging, push-pull among obstacles, and pick-place with disturbances. We show that our framework outperforms an off-the-shelf RL method in the reactive pick-place task in terms of position error and orientation error. We also show that M3P2I is generalizable in combining different constraints, such as generating hybrid motions of push and pull for the mobile robot, and grasping objects with different grasping poses. The real-world experiments show that the system exhibits reactiveness and robustness against human disturbances in a variety of manipulation tasks.

The supporting videos can be found at https://sites.google.com/view/m3p2i-aip.

Achieving human-like action planning requires profound reasoning and context-awareness capabilities. It is especially true for autonomous robotic mobile manipulation in dynamic environments. In the case of component failure, the autonomous robotic system requires reliable adaptation capabilities combined with a consistent understanding of the task, environment, and the robot’s capabilities for successful task completion. Recent research has shown that
Active Inference, a unifying neuroscientific theory of the brain, has the potential to intrinsically handle substantial uncertainties in the system, resembling the adaptability of humans. These works, however, have the following limitations: (1) no distinction is made between actions with some commonality, capable of satisfying similar tasks, and (2) actions are assumed to be always feasible when preconditions are satisfied, regardless of their context. Given the situation, certain actions satisfying a task might not lead to task succession. This work
proposes the AI for retail (Airet) framework, a novel extension of action planning through Active Inference for mobile manipulation. The Airet framework uses Bayesian networks and Ontological Reasoning to facilitate context-awareness in action planning through Active Inference. Reasoning on robot components, action-, manipulation- and environmental constraints is facilitated through a description-logic-based reasoner and an OWL-based ontology containing concepts relevant for action selection in a retail context. The capabilities of the Airet framework are demonstrated through the following cases (1) irrecoverable task & component. Failure prevention when dealing with ill-defined tasks, (2) Selection of the best action given the situation & the component capabilities through context-awareness (3) failure recovery & adaptation when dealing with component failure. Lastly, these situations are compared with
research on reactive task planning through Active Inference without context-awareness. This thesis represents a leap forward from the current state-of-the-art in Active Inference for task planning in robotics, laying the foundations for further research in the direction of this thesis

This thesis is a contribution to the research on Active Inference for Robotics. Active Inference is an intricate, intriguing theory from neuroscience, a field in which it has already gained a greater following and popularity. This theory, based on the underlying Free Energy Principle, provides a unified account of perception, action and learning in the biological brain. It has great explanatory power of the function of the biological brain and furthermore it is mathematically well-defined. This property makes the theory suitable for a translation to robotics, in which it can also provide a unified account of action and perception. This is not only elegant, but potentially very powerful too. The research for Active Inference in robotics is young, but the current research already shows that Active Inference indeed has great potential for robotics control. Literature on Active Inference is narrow and complex, and provides a lot of concepts to work with in a translation to robotics control. Once such concept are the generalised coordinates of motion, which are the instantaneous derivatives of a dynamic variable. The incorporation of generalised coordinates, especially in combination with the assumption that the noise encountered in a dynamic environment is coloured, has great potential to be beneficial for both action and perception when it comes to robot control in real environments. Generalised coordinates provide a reference frame for the gradient descent that is applied to provide the action and perception laws, which in a dynamic setting has to `hit a moving target'. Furthermore, in combination with coloured noise the generalised coordinates are advantageous for dealing with such noise. In this thesis, detailed research is provided with regards to the application of generalised coordinates in Active Inference for robotics. Current research for robotics in which Active Inference has been applied doesn't exploit the full potential of generalised coordinates. Therefore, this research aims to explore the constructs necessary to apply generalised coordinates of motion in an on-line Active Inference control loop of an LTI State Space system. A detailed derivation of the generalised precision, which relates generalised coordinates and coloured noise, is provided. A method for obtaining generalised output by means of finite differences is proposed, that constructs generalised coordinates from the on-line data in scenarios in which the environment does not provide the required generalised coordinates naturally. The method is implemented in the simulation of a one degree of freedom SISO LTI State Space scenario which highlights the potential but also the difficulties still faced when applying Active Inference for on-line robotics control. Besides the detailed derivations of some aspects of Active Inference for robotics, open problems are identified and suggested for future research that can potentially yield methods to apply Active Inference in robotics at full capacity, providing a true biologically plausible robot control method.