Distributed model predictive control (DMPC) is promising in achieving optimal cooperative control in multirobot systems (MRS). However, real-time DMPC implementation relies on numerical optimization tools to periodically calculate local control sequences online. This process is computationally demanding and lacks scalability for large-scale, nonlinear MRS. This article proposes a novel distributed learning-based predictive control framework for scalable multirobot control. Unlike conventional DMPC methods that calculate open-loop control sequences, our approach centers around a computationally fast and efficient distributed policy learning algorithm that generates explicit closed-loop DMPC policies for MRS without using numerical solvers. The policy learning is executed incrementally and forward in time in each prediction interval through an online distributed actor-critic implementation. The control policies are successively updated in a receding-horizon manner, enabling fast and efficient policy learning with the closed-loop stability guarantee. The learned control policies could be deployed online to MRS with varying robot scales, enhancing scalability and transferability for large-scale MRS. Furthermore, we extend our methodology to address the multirobot safe learning challenge through a force field-inspired policy learning approach. We validate our approach's effectiveness, scalability, and efficiency through extensive experiments on cooperative tasks of large-scale wheeled robots and multirotor drones. Our results demonstrate the rapid learning and deployment of DMPC policies for MRS with scales up to 10 000 units.

This letter explores the problem of delivering unwieldy objects using nonholonomic mobile bases. We propose a new approach called free pushing to address this challenge. Unlike previous stable pushing methods which maintain a stiff robot-object contact, our approach allows the robot to maneuver around the object while pushing it. It aims to execute continuous pushes without losing contact for improved pushing maneuverability. Additionally, to ensure the feasibility of the planned pushes, a robot-object contact model is developed to account for the shape and kinematics of the robot in pushing modeling and planning. A Model Predictive Controller solves the pushing planning problem in real time. Experimental results show that the proposed method achieves an average success rate of 83% with an accuracy of 0.085 m when pushing to the selected goals. Compared to the baselines, this approach improves the agility and efficiency of mobile pushers. Furthermore, it is robust in achieving the task while tolerating modeling errors.

Integrating rule-based policies into reinforcement learning promises to improve data efficiency and generalization in cooperative pursuit problems. However, most implementations do not properly distinguish the influence of neighboring robots in observation embedding or inter-robot interaction rules, leading to information loss and inefficient cooperation. This letter proposes a cooperative pursuit algorithm named Decentralized Adaptive COOperative Pursuit via Attention (DACOOP-A) by empowering reinforcement learning with artificial potential field and attention mechanisms. An attention-based framework is developed to emphasize important neighbors by concurrently integrating the learned attention scores into observation embedding and inter-robot interaction rules. A KL divergence regularization is introduced to alleviate the resultant learning stability issue. Improvements in data efficiency and generalization are demonstrated through numerical simulations. Extensive quantitative analyses are performed to illustrate the advantages of the proposed modules. Real-world experiments are performed to justify the feasibility of DACOOP-A in physical systems.

This article explores deep reinforcement learning (DRL) for the flocking control of unmanned aerial vehicle (UAV) swarms. The flocking control policy is trained using a centralized-learning-decentralized-execution (CTDE) paradigm, where a centralized critic network augmented with additional information about the entire UAV swarm is utilized to improve learning efficiency. Instead of learning inter-UAV collision avoidance capabilities, a repulsion function is encoded as an inner-UAV 'instinct.' In addition, the UAVs can obtain the states of other UAVs through onboard sensors in communication-denied environments, and the impact of varying visual fields on flocking control is analyzed. Through extensive simulations, it is shown that the proposed policy with the repulsion function and limited visual field has a success rate of 93.8% in training environments, 85.6% in environments with a high number of UAVs, 91.2% in environments with a high number of obstacles, and 82.2% in environments with dynamic obstacles. Furthermore, the results indicate that the proposed learning-based methods are more suitable than traditional methods in cluttered environments.

In recent years, safe reinforcement learning (RL) with the actor-critic structure has gained significant interest for continuous control tasks. However, achieving near-optimal control policies with safety and convergence guarantees remains challenging. Moreover, few works have focused on designing RL algorithms that handle time-varying safety constraints. This article proposes a safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. The algorithm's novelty lies in two key aspects. Firstly, the approach introduces a unique barrier force-based control policy structure to ensure control safety during learning. Secondly, a multistep policy evaluation mechanism is employed, enabling the prediction of policy safety risks under time-varying constraints and guiding safe updates. Theoretical results on learning convergence, stability, and robustness are proven. The proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. It is also applied to the real-world problem of integrated path following and collision avoidance for two intelligent vehicles - a differential-drive vehicle and an Ackermann-drive one. The experimental results demonstrate the impressive sim-to-real transfer capability of our approach, while showcasing satisfactory online control performance.

This paper investigates the deep reinforcement learning based secure control problem for cyber–physical systems (CPS) under false data injection attacks. We describe the CPS under attacks as a Markov decision process (MDP), based on which the secure controller design for CPS under attacks is formulated as an action policy learning using data. Rendering the soft actor–critic learning algorithm, a Lyapunov-based soft actor–critic learning algorithm is proposed to offline train a secure policy for CPS under attacks. Different from the existing results, not only the convergence of the learning algorithm but the stability of the system using the learned policy is proved, which is quite important for security and stability-critical applications. Finally, both a satellite attitude control system and a robot arm system are used to show the effectiveness of the proposed scheme, and comparisons between the proposed learning algorithm and the classical PD controller are also provided to demonstrate the advantages of the control algorithm designed in this paper.

This letter addresses the problem of pushing manipulation with nonholonomic mobile robots. Pushing is a fundamental skill that enables robots to move unwieldy objects that cannot be grasped. We propose a stable pushing method that maintains stiff contact between the robot and the object to avoid consuming repositioning actions. We prove that a line contact, rather than a single point contact, is necessary for nonholonomic robots to achieve stable pushing. We also show that the stable pushing constraint and the nonholonomic constraint of the robot can be simplified as a concise linear motion constraint. Then the pushing planning problem can be formulated as a constrained optimization problem using nonlinear model predictive control (NMPC). According to the experiments, our NMPC-based planner outperforms a reactive pushing strategy in terms of efficiency, reducing the robot's traveled distance by 23.8% and time by 77.4%. Furthermore, our method requires four fewer hyperparameters and decision variables than the Linear Time-Varying (LTV) MPC approach, making it easier to implement. Real-world experiments are carried out to validate the proposed method with two differential-drive robots, Husky and Boxer, under different friction conditions.

The problem of learning-based control for robots has been extensively studied, whereas the security issue under malicious adversaries has not been paid much attention to. Malicious adversaries can invade intelligent devices and communication networks used in robots, causing incidents, achieving illegal objectives, and even injuring people. This article first investigates the problems of optimal false data injection attack scheduling and countermeasure design for car-like robots in the framework of deep reinforcement learning. Using a state-of-the-art deep reinforcement learning approach, an optimal false data injection attack scheme is proposed to deteriorate the tracking performance of a robot, guaranteeing the tradeoff between the attack efficiency and the limited attack energy. Then, an optimal tracking control strategy is learned to mitigate attacks and recover the tracking performance. More importantly, a theoretical stability guarantee of a robot using the learning-based secure control scheme is achieved. Both simulated and real-world experiments are conducted to show the effectiveness of the proposed schemes.

This article proposes a fuzzy adaptive design solving the finite-time constrained tracking for hypersonic flight vehicles (HFVs). Actuator dynamics and asymmetric time-varying constraints are considered when solving this problem. The main features of the proposed design lie in 1) introducing a novel piecewise but differentiable switching control law, with an appropriate design thought to avoid the singularity issues typical of finite-time control; 2) handling actuator magnitude, bandwidth, and rate constraints, thanks to the introduction of an auxiliary compensating system counteracting the adverse effects caused by actuator physical constraints, while guaranteeing the closed-loop stability; and 3) handling asymmetric time-varying state constraints, thanks to the introduction of tan-type barrier Lyapunov functions working for both constrained and unconstrained scenarios. Comparative simulation results illustrate the effectiveness of the proposed strategy over existing methods for HFVs in terms of convergence, smoothness, actuator performance, and constraints satisfaction.

Koopman operators are of infinite dimension and capture the characteristics of nonlinear dynamics in a lifted global linear manner. The finite data-driven approximation of Koopman operators results in a class of linear predictors, useful for formulating linear model predictive control (MPC) of nonlinear dynamical systems with reduced computational complexity. However, the robustness of the closed-loop Koopman MPC under modeling approximation errors and possible exogenous disturbances is still a crucial issue to be resolved. Aiming at the above problem, this paper presents a robust tube-based MPC solution with Koopman operators, i.e., r-KMPC, for nonlinear discrete-time dynamical systems with additive disturbances. The proposed controller is composed of a nominal MPC using a lifted Koopman model and an off-line nonlinear feedback policy. The proposed approach does not assume the convergence of the approximated Koopman operator, which allows using a Koopman model with a limited order for controller design. Fundamental properties, e.g., stabilizability, observability, of the Koopman model are derived under standard assumptions with which, the closed-loop robustness and nominal point-wise convergence are proven. Simulated examples are illustrated to verify the effectiveness of the proposed approach.

Multi-robot formation control has been intensively studied in recent years. In practical applications, the multi-robot system's ability to independently change the formation to avoid collision among the robots or with obstacles is critical. In this study, a multi-robot adaptive formation control framework based on deep reinforcement learning is proposed. The framework consists of two layers, namely the execution layer and the decision-making layer. The execution layer enables the robot to approach its target position and avoid collision with other robots and obstacles through a deep network trained by a reinforcement learning method. The decision-making layer organizes all robots into a formation through a new leader-follower configuration and provides target positions to the leader and followers. The leader's target position is kept unchanged, while the follower's target position is changed according to the situation it encounters. In addition, to operate more effectively in environments with different levels of complexity, a hybrid switching control strategy is proposed. The simulation results demonstrate that our proposed formation control framework enables the robots to adjust formation independently to pass through obstacle areas and can be generalized to different scenarios with unknown obstacles and varying number of robots.

This paper proposes a sparse Bayesian treatment of deep neural networks (DNNs) for system identification. Although DNNs show impressive approximation ability in various fields, several challenges still exist for system identification problems. First, DNNs are known to be too complex that they can easily overfit the training data. Second, the selection of the input regressors for system identification is nontrivial. Third, uncertainty quantification of the model parameters and predictions are necessary. The proposed Bayesian approach offers a principled way to alleviate the above challenges by marginal likelihood/model evidence approximation and structured group sparsity-inducing priors construction. The identification algorithm is derived as an iterative regularised optimisation procedure that can be solved as efficiently as training typical DNNs. Remarkably, an efficient and recursive Hessian calculation method for each layer of DNNs is developed, turning the intractable training/optimisation process into a tractable one. Furthermore, a practical calculation approach based on the Monte-Carlo integration method is derived to quantify the uncertainty of the parameters and predictions. The effectiveness of the proposed Bayesian approach is demonstrated on several linear and nonlinear system identification benchmarks by achieving good and competitive simulation accuracy. The code to reproduce the experimental results is open-sourced and available online.

This article investigates the zero-sum game-based secure control problem for cyber-physical systems (CPS) under the actuator false data injection attacks. The physical process is described as a linear time-invariant discrete-time model. Both the process noise and the measurement noise are addressed in the design process. An optimal Kalman filter is given to estimate the system states. The adversary and the defender are modeled as two players. Under the zero-sum game framework, an optimal infinite-horizon quadratic cost function is defined. Employing the dynamic programming approach, the optimal defending policy and the attack policy are derived. The convergence of the cost function is proved. Moreover, the critical attack probability is derived, beyond which the cost cannot be bounded. Finally, simulation results are provided to validate the proposed secure scheme.

Reinforcement learning (RL) is promising for complicated stochastic nonlinear control problems. Without using a mathematical model, an optimal controller can be learned from data evaluated by certain performance criteria through trial-and-error. However, the data-based learning approach is notorious for not guaranteeing stability, which is the most fundamental property for any control system. In this paper, the classic Lyapunov's method is explored to analyze the uniformly ultimate boundedness stability (UUB) solely based on data without using a mathematical model. It is further shown how RL with UUB guarantee can be applied to control dynamic systems with safety constraints. Based on the theoretical results, both off-policy and on-policy learning algorithms are proposed respectively. As a result, optimal controllers can be learned to guarantee UUB of the closed-loop system both at convergence and during learning. The proposed algorithms are evaluated on a series of robotic continuous control tasks with safety constraints. In comparison with the existing RL algorithms, the proposed method can achieve superior performance in terms of maintaining safety. As a qualitative evaluation of stability, our method shows impressive resilience even in the presence of external disturbances.