This paper presents a novel approach to reinforcement learning (RL) for control systems that provides probabilistic stability guarantees using finite data. Leveraging Lyapunov's method, we propose a probabilistic stability theorem that ensures mean square stability using only a finite number of sampled trajectories. The probability of stability increases with the number and length of trajectories, converging to certainty as data size grows. Additionally, we derive a policy gradient theorem for stabilizing policy learning and develop an RL algorithm, L-REINFORCE, that extends the classical REINFORCE algorithm to stabilization problems. The effectiveness of L-REINFORCE is demonstrated through simulations on a Cartpole task, where it outperforms the baseline in ensuring stability. This work bridges a critical gap between RL and control theory, enabling stability analysis and controller design in a model-free framework with finite data.

This article is devoted to the antibump switched linear parameter varying (sLPV) controller design for morphing aircraft under delayed scheduling variables (or parameters), which is typically caused by lagging measurements of the morphing extent. Such delayed scheduling is formulated as the control disturbance and the asynchronous control in the sLPV scheme, according to whether the current mode governed by scheduling variables is correctly detected or not. The persistent dwell time (PDT) switching signals are utilized in this article to describe inherent slow and rapid switching phenomena for steady flight and fast morphing, respectively, which is more applicable than the conventional average dwell time (DT) or DT and covers them as special cases. By adopting the detected-mode-based Lyapunov functions and a smooth function, the stability condition is obtained for the underlying system, upon which the antibump sLPV controller allowing for delayed scheduling is designed, in contrast to the existing studies that simply ignore the detection lag to allow the use of overlapped partitions and scheduling-variable-dependent Lyapunov functions for different modes. By an aircraft with a variable-sweep wing and an aircraft with a deformable wingspan, the effectiveness and the superiority of the proposed approach are demonstrated via simulations.

Reinforcement learning (RL) is promising for complicated stochastic nonlinear control problems. Without using a mathematical model, an optimal controller can be learned from data evaluated by certain performance criteria through trial-and-error. However, the data-based learning approach is notorious for not guaranteeing stability, which is the most fundamental property for any control system. In this paper, the classic Lyapunov's method is explored to analyze the uniformly ultimate boundedness stability (UUB) solely based on data without using a mathematical model. It is further shown how RL with UUB guarantee can be applied to control dynamic systems with safety constraints. Based on the theoretical results, both off-policy and on-policy learning algorithms are proposed respectively. As a result, optimal controllers can be learned to guarantee UUB of the closed-loop system both at convergence and during learning. The proposed algorithms are evaluated on a series of robotic continuous control tasks with safety constraints. In comparison with the existing RL algorithms, the proposed method can achieve superior performance in terms of maintaining safety. As a qualitative evaluation of stability, our method shows impressive resilience even in the presence of external disturbances.

This article investigates the stability of continuous-time switched linear systems with dwell-time constraints. A fresh insight into this established problem is provided via novel stability conditions that require the solution to a family of differential Lyapunov equations and algebraic Lyapunov equations. The proposed analysis, which leads to a peculiar Lyapunov function that is decreasing in between and at switching instants, enjoys the following properties: it achieves the same dwell time as the well-known result in the research 'stability and stabilization of continuous time switched linear systems' by Geromel and Colaneri; it removes the increasing computational complexity of the linear interpolation method; it leads to a straightforward counterpart for discrete-time switched linear systems.We show the application of this methodology to the problem of adaptive control of switched linear systems with parametric uncertainties.

Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this letter, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.

This paper proposes a tube-based method for the asynchronous observation problem of discrete-Time switched linear systems in the presence of amplitude-bounded disturbances. Sufficient stability conditions of the nominal observer error system under mode-dependent persistent dwell-Time MPDT switching are first established. Taking the disturbances into account, a novel asynchronous MPDT robust positive invariant RPI set and an asynchronous MPDT generalized RPI GRPI set are determined for the difference system between the nominal and disturbed observer error systems. Further, the global uniform asymptotical stability of the observer error system is established in the sense of converging to the asynchronous MPDT GRPI set, i.e., the cross section of the tube of the observer error system. Finally, the proposed results are validated on a space robot manipulator example.

In the presence of discontinuous time-varying delays, neither Krasovskii nor Razumikhin techniques can be successfully applied to adaptive stabilization of uncertain switched time-delay systems. This paper develops a new adaptive control scheme for switched time-delay systems that can handle impulsive behavior in both states and time-varying delays. At the core of the proposed scheme is a Lyapunov function with a dynamically time-varying coefficient, which allows the Lyapunov function to be non-increasing at the switching instants. The control scheme, guaranteeing global uniformly ultimate boundedness of the closed-loop system, substantially enlarges the class of uncertain switched systems for which the adaptive stabilization problem can be solved. A two-tank system is used to illustrate the effectiveness of the method.

This paper addresses robust adaptive stabilization of uncertain switched time-delay systems with unknown disturbances in a high-order form. In addition to parametric uncertainty, an extra source of uncertainty arises from having time-varying delays. In particular, the upper bound of the changing rate of the delay is assumed to be unknown. To this purpose, a new reparametrization method is proposed, which incorporates both sources of uncertainty in a single scalar parameter. Therefore, a single scalar adaptive law is designed for parametric uncertainties and time-varying delays, which is combined with a new dynamic gain embedded in the control action. The new dynamic gain is designed to dominate the nonlinear effects caused by both the parametric uncertainty and the unknown variation of the time delay. The proposed design guarantees global asymptotic stability for arbitrary switching. A numerical example illustrates the effectiveness of the method.

In this paper, a novel Lyapunov function is proposed to study switched linear systems with a switching delay between activation of system modes and activation of candidate controller modes. The novelty consists in continuity of the Lyapunov function at the switching instants and discontinuity when the system modes and controller modes are matched. This structure is exploited to construct a time-varying Lyapunov function that is non-increasing at time instants of discontinuity. Stability criteria based on the novel Lyapunov function are developed to guarantee global asymptotic stability in the noiseless case. Most importantly, when exogenous disturbances are considered, the proposed Lyapunov function can be used to guarantee a finite non-weighted L₂ gain for asynchronously switched systems, for which Lyapunov functions proposed in literature are inconclusive. A numerical example illustrates the effectiveness of the proposed method.

This letter introduces a new switched adaptive control mechanism that can cope with parametric uncertainty while using discrete and saturated actuators. Control of air handling units (AHUs), where air and water supply have discrete and saturated characteristics, is the motivational drive behind this letter. We show that the cheap actuation and low computational requirements of building automation installations can be met after recasting the AHU thermal dynamics as a switched linear system with discrete working modes. Adaptive laws with anti-windup compensation and a switching law based on dwell time are introduced to cope with the uncertainties and input constraints of the switched linear system. Tracking performance is shown analytically and demonstrated via a numerical test case.