Improved PER-DDPG based nonparametric modeling of ship dynamics with uncertainty

Abstract

This study contributes to addressing the challenge of quickly obtaining an effective and accurate nonparametric model for describing ship maneuvering motion in three degrees of freedom (3-DOF). To achieve this, an intelligent ship dynamics nonparametric modeling method named improved PER-DDPG is proposed. This method leverages the deep deterministic policy gradient algorithm (DDPG) and prioritized experience replay mechanism (PER) and analyzes the characteristics between the goal of deep reinforcement learning (DRL) and the modeling process of the nonparametric model. The PER mechanism is utilized to enhance the agent's understanding of the overall mechanism of ship motion by improving the utilization of samples. The meaning of target value is redefined due to transforming DRL aiming at maximizing cumulative rewards into maximizing the set of immediate rewards at each time step. To validate the performance of the proposed modeling method, we conduct simulation studies using a benchmark ship model i.e., a Mariner cargo ship dynamic model, and experimental studies using a real unmanned surface vehicle (USV). In the simulation test, we demonstrate the effectiveness and generalization of the proposed method through zigzag and turning circle tests. Furthermore, we verify the robustness and applicability of the proposed method by using datasets with uncertain environmental disturbances and datasets with different sampling frequencies. Additionally, the experimental tests conducted on the USV indicate the consistency of the proposed approach.