Improved PER-DDPG based nonparametric modeling of ship dynamics with uncertainty

Improved PER-DDPG based nonparametric modeling of ship dynamics with uncertainty

Zhu, Man (Wuhan University of Technology)
Tian, Kang (Wuhan University of Technology)
Wen, Yuan Qiao (Wuhan University of Technology)
Cao, Ji Ning (Wuhan University of Technology)
Huang, L. (TU Delft Safety and Security Science; Wuhan University of Technology)

2023

This study contributes to addressing the challenge of quickly obtaining an effective and accurate nonparametric model for describing ship maneuvering motion in three degrees of freedom (3-DOF). To achieve this, an intelligent ship dynamics nonparametric modeling method named improved PER-DDPG is proposed. This method leverages the deep deterministic policy gradient algorithm (DDPG) and prioritized experience replay mechanism (PER) and analyzes the characteristics between the goal of deep reinforcement learning (DRL) and the modeling process of the nonparametric model. The PER mechanism is utilized to enhance the agent's understanding of the overall mechanism of ship motion by improving the utilization of samples. The meaning of target value is redefined due to transforming DRL aiming at maximizing cumulative rewards into maximizing the set of immediate rewards at each time step. To validate the performance of the proposed modeling method, we conduct simulation studies using a benchmark ship model i.e., a Mariner cargo ship dynamic model, and experimental studies using a real unmanned surface vehicle (USV). In the simulation test, we demonstrate the effectiveness and generalization of the proposed method through zigzag and turning circle tests. Furthermore, we verify the robustness and applicability of the proposed method by using datasets with uncertain environmental disturbances and datasets with different sampling frequencies. Additionally, the experimental tests conducted on the USV indicate the consistency of the proposed approach.

Deep deterministic policy gradient (DDPG)
Nonparametric modeling
Prioritized experience replay (PER)
Ship dynamic model
Uncertainty

http://resolver.tudelft.nl/uuid:203364df-f8fe-44e2-9f09-acc81f9e8f51

https://doi.org/10.1016/j.oceaneng.2023.115513

2024-02-03

0029-8018

Ocean Engineering, 286

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

journal article

© 2023 Man Zhu, Kang Tian, Yuan Qiao Wen, Ji Ning Cao, L. Huang