Mobile manipulators operating in dynamic environments shared with humans and robots must adapt in real time to environmental changes to complete their tasks effectively. While global planning methods are effective at considering the full task scope, they lack the computational efficiency required for reactive adaptation. In contrast, local planning approaches can be executed online but are limited by their inability to account for the full task's duration. To tackle this, we propose Globally-Guided Geometric Fabrics (G3F), a framework for real-time motion generation along the full task horizon, by interleaving an optimization-based planner with a fast reactive geometric motion planner, called Geometric Fabrics (GF). The approach adapts the path and explores a multitude of acceptable target poses, while accounting for collision avoidance and the robot's physical constraints. This results in a real-time adaptive framework considering whole-body motions, where a robot operates in close proximity to other robots and humans. We validate our approach through various simulations and real-world experiments on mobile manipulators in multi-agent settings, achieving improved success rates compared to vanilla GF, Prioritized Rollout Fabrics and Model Predictive Control. ... Read more

Deployment of robots in dynamic environments requires reactive trajectory generation. While optimization-based methods, such as Model Predictive Control focus on constraint verificaction, Geometric Fabrics offer a computationally efficient way to generate trajectories that include all avoidance behaviors if the environment can be represented as a set of object primitives. Obtaining such a representation from sensor data is challenging, especially in dynamic environments. In this letter, we integrate implicit environment representations, such as Signed Distance Fields and Free Space Decomposition into the framework of Geometric Fabrics. In the process, we derive how numerical gradients can be integrated into the push and pull operations in Geometric Fabrics. Our experiments reveal that both, ground robots and robotic manipulators, can be controlled using these implicit representations. Moreover, we show that, unlike the explicit representation, implicit representations can be used in the presence of dynamic obstacles without further considerations. Finally, we demonstrate our methods in the real-world, showing the applicability of our approach in practice. ... Read more

We present a sampling-based model predictive control method that uses a generic physics simulator as the dynamical model. In particular, we propose a Model Predictive Path Integral controller (MPPI) that employs the GPU-parallelizable IsaacGym simulator to compute the forward dynamics of the robot and environment. Since the simulator implicitly defines the dynamic model, our method is readily extendable to different objects and robots, allowing one to solve complex navigation and contact-rich tasks. We demonstrate the effectiveness of this method in several simulated and real-world settings, including mobile navigation with collision avoidance, non-prehensile manipulation, and whole-body control for high-dimensional configuration spaces. This is a powerful and accessible open-source tool to solve many contact-rich motion planning tasks. ... Read more

As robotics will play a crucial role in the future of our modern societies, the need for advancements in the field is more pronounced than ever. While robots are already present in industrial settings, they are noticeably absent from dynamic environments. Dynamic environments are characterized by other moving agents, such as humans, varying tasks and high safety requirements. With the aim to deploy robots in such environments, Trajectory Generation (TG) becomes crucial. TG approaches aim to compute sequences of control commands that bring the robot from its current configuration to a desired goal state while avoiding collisions with obstacles and itself. Thus, it is directly placed between task planning, the problem of defining what high level tasks should be executed in which order, and control, the problem of executing motor commands. A good solution to TG must be fast, to cope with changes in the environment, flexible to different goal definitions, and it should promote safety. Most advancements in the field of robotics in TG focus either on manipulators or on mobile robots. However, the combination of both systems seems inevitable for deployment in human-shared environments.

TG for mobile manipulation is usually formulated as an optimization problem of a finite time horizon. This approach is known as Model Predictive Control (MPC) and is widely used in the field of autonomous driving thanks to its feasibility and stability guarantees. In Chapter 4, we present a method to bring MPC to mobile manipulation. The method formulates the TG problem for the entire kinematic chain and relies on Free Space Decomposition (FSD) for collision avoidance. This leads to reasonable control frequencies of 10Hz independent on the amount of collision obstacles in the environment. Importantly, this approach allows for coupled motion of the mobile base and the manipulator. This is beneficial in situations where synchronization of the two subsystems is crucial, such as opening doors or moving obstacles around. Despite these simplifications on the environment representations, computational costs limit the applicability of MPC to mobile manipulation as motion is not considered truly reactive and different components, such as goal attraction and collision avoidance, are not easily separable.

A recent novel approach to receding horizon control is the formulation as a purely geometric problem. Early successes in this direction, including Cartesian Impedance Control (IC) and Artificial Potential Fields (APF), led to the formulation as sets of dynamical systems on smooth manifolds in the configuration space. The framework of Optimization Fabrics (fabrics) unifies such ideas, offers stability guarantees in static environments, and results in highly reactive behavior similar to simple low-level controllers. This framework relies on non-Riemannian geometry to shape a smooth manifold of the configuration space with individual behaviors, such as collision avoidance, joint-limit avoidance, and goal attraction. In Chapter 5, we present a generalization of fabrics to dynamic environments. We refer to the resulting framework as Dynamic Fabrics (DF). The generalization uses time-parameterized manifolds to integrate moving obstacles and time-parameterized reference trajectories. The latter is especially important for long-horizon TG that may exhibit local minima. Importantly, Chapter 5 shows that the dynamic component of DF is required when coping with moving obstacles. As repulsive forces in fabrics are proportional to the approaching speed of obstacle and robot, collision avoidance in a pseudo-static fashion is not sufficient when the robot is moving slowly. Finally, we deploy the general framework of DF to several real-world settings showing the applicability of the framework to mobile manipulation. First, we present a way to integrate non-holonomic constraints into the framework. Despite loosing formal guarantees on convergence, the method is shown to be the natural extension to wheeled mobile robots characterized by non-holonomic constraints. Second, Chapter 6 presents a symbolic implementation of fabrics to achieve higher control frequencies. Symbolic implementations are possible because the framework of fabrics is based on differential equations of second order, for which a closed-form solution exists. For changing environments, obstacles states are then only concretized at runtime. Additionally, we show symbolic hyperparameters can be tuned automatically to achieve expert-level tuning performance. Third, to overcome high requirements on the perception pipeline, Chapter 7 integrates different implicit environment representations into the framework. Using Signed Distance Field (SDF) and FSD for example is widely used in mobile robotics when formulating TG as MPC. We show that the same representations can be used in fabrics while achieving faster solver times. Finally, Chapter 8 deploys a mobile manipulator controlled by fabrics in a supermarket. Dexterous manipulation is programmed using learning-from-demonstration, with fabrics as the underlying encoding. That allows to teach rather than program complicated behaviors while maintaining properties on collision avoidance.

This thesis presents insights into aspects of motion planning, advances the framework of fabrics for TG, and compares it extensively to the more commonly used method of MPC. Through the ideas presented in this thesis, we hope to encourage the usage of geometric properties of robotic systems deployed to human-shared environments. This approach does not only provide reactive TG but also may act as a compact encoding of trajectories for learning-based methods in the future.
... Read more

In this paper, we address the problem of real-time motion planning for multiple robotic manipulators that operate in close proximity. We build upon the concept of dynamic fabrics and extend them to multi-robot systems, referred to as Multi-Robot Dynamic Fabrics (MRDF). This geometric method enables a very high planning frequency for high-dimensional systems at the expense of being reactive and prone to deadlocks. To detect and resolve deadlocks, we propose Rollout Fabrics where MRDF are forward simulated in a decentralized manner. We validate the methods in simulated close-proximity pick-and-place scenarios with multiple manipulators, showing high-success rates and real-time performance. Code, video: https://github.com/tud-amr/multi-robot-fabrics ... Read more

Adaptation to external and internal changes is of major importance for robotic systems in uncertain environments. Here, we present a novel multisensory active inference (AIF) torque controller for industrial arms that shows how prediction can be used to resolve adaptation. Our controller, inspired by the predictive brain hypothesis, improves the capabilities of current AIF approaches by incorporating learning and multimodal integration of low- and high-dimensional sensor inputs (e.g., raw images) while simplifying the architecture. We performed a systematic evaluation of our model on a 7DoF Franka Emika Panda robot arm by comparing its behavior with previous AIF baselines and classic controllers, analyzing both qualitatively and quantitatively adaptation capabilities and control accuracy. The results showed improved control accuracy in goal-directed reaching with high noise rejection due to multimodal filtering, and adaptability to dynamical inertial changes, elasticity constraints, and human disturbances without the need to relearn the model or parameter retuning. ... Read more

Optimization fabrics are a geometric approach to real-time local motion generation, where motions are designed by the composition of several differential equations that exhibit a desired motion behavior. We generalize this framework to dynamic scenarios and nonholonomic robots and prove that fundamental properties can be conserved. We show that convergence to desired trajectories and avoidance of moving obstacles can be guaranteed using simple construction rules of the components. In addition, we present the first quantitative comparisons between optimization fabrics and model predictive control and show that optimization fabrics can generate similar trajectories with better scalability, and thus, much higher replanning frequency (up to 500 Hz with a 7 degrees of freedom robotic arm). Finally, we present empirical results on several robots, including a nonholonomic mobile manipulator with 10 degrees of freedom and avoidance of a moving human, supporting the theoretical findings. ... Read more

In this paper, we present an automated parameter optimization method for trajectory generation. We formulate parameter optimization as a constrained optimization problem that can be effectively solved using Bayesian optimization. While the approach is generic to any trajectory generation method, we showcase it using optimization fabrics. Optimization fabrics are a geometric trajectory generation method based on non-Riemannian geometry. By symbolically pre-solving the structure of the tree of fabrics, we obtain a parameterized trajectory generator, called symbolic fabrics. We show that autotuned symbolic fabrics reach expert-level performance in a few trials. Additionally, we show that tuning transfers across different robots, motion planning problems and between simulation and real world. Finally, we qualitatively showcase that the framework could be used for coupled mobile manipulation. ... Read more

This paper presents a real-time method for whole-body trajectory optimization of mobile manipulators in simplified dynamic and unstructured environments. Current trajectory optimization methods typically use decoupling of the mobile base and the robotic arm, which reduces flexibility in motion, does not scale to unstructured environments, and does not consider the future evolution of the environment, which is crucial to avoid dynamic obstacles. Given a goal configuration, such as waypoints generated by a global path planner, we formulate a receding horizon trajectory optimization minimizing the distance-to-target while avoiding collisions with static and dynamic obstacles. The presented method unifies the control of a robotic arm and a non-holonomic base to allow coupled trajectory planning. For collision avoidance, we propose to compute three convex regions englobing the robot's major body parts (i.e., base, shoulder-link and wrist-link) and thus reducing and limiting the number of inequality constraints, regardless of the number of obstacles in the environment. Moreover, our approach incorporates predicted trajectory information to smoothly, and in advance, avoid dynamic obstacles. The presented results show that trajectory optimization for the coupled system can reduce the total execution time by 48% and that applying the convex region generation for individual links allows keeping the computational costs low, even for complex scenarios, enabling onboard implementation. ... Read more