Title
Optimizing deep reinforcement learning policies for deteriorating systems considering ordered action structuring and value of information
Author
Andriotis, C. (TU Delft Structural Design & Mechanics)
Papakonstantinou, K.G. (Pennsylvania State University)
Contributor
Li, J. (editor)
Spanos, Pol D. (editor)
Chen, J.B. (editor)
Peng, Y.B. (editor)
Date
2022
Abstract
Inspection and maintenance (I&M) optimization entails many sources of computational complexity, among others, due to high-dimensional decision and state variables in multi-component systems, long planning horizons, stochasticity of objectives and constraints, and inherent uncertainties in measurements and models. This paper studies how the above can be addressed within the context of constrained Partially Observable Markov Decision Processes (POMDPs) and Deep Reinforcement Learning (DRL) in a unified fashion. Special emphasis is paid on how ordered action structuring of I&M actions can be exploited to decompose the respective policy parametrizations in actor-critic DRL schemes, resulting into fully decoupled maintenance and inspection actors. It is shown that the Value of Information (VoI) is naturally utilized in such POMDP control frameworks, as directly associated with the DRL advantage functions that emerge in the gradient computations of the inspection policy parameters. Overall, the presented approach, following the natural flow of engineering decisions, results in new architectural configurations for policy networks, facilitating more efficient training, while alleviating further the dimensionality burdens related to combinatorial definitions of I&M actions. The efficiency of the methodology is demonstrated in numerical experiments of a structural system subject to corrosion, where the optimization problem is formulated to concurrently account for state and model uncertainties as well as long-term probability of failure exceedance constraints. Results showcase that the obtained DRL policies considerably outperform standard decision rules.
Subject
inspection & maintenance
deep reinforcement learning
partially observable Markov decision processes
value of information
stochastic constraints
decision theory
To reference this document use:
http://resolver.tudelft.nl/uuid:b64f6296-e747-47e9-81e3-622db0ebc781
Embargo date
2023-07-01
Source
Proceedings of the 13th International Conference on Structural Safety and Reliability (ICOSSAR)
Event
International Conference on Structural Safety and Reliability, 2022-09-13 → 2022-09-17, Tongji University, Shanghai, China
Bibliographical note
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Part of collection
Institutional Repository
Document type
conference paper
Rights
© 2022 C. Andriotis, K.G. Papakonstantinou