Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management

Abstract

Life-cycle management of large-scale transportation systems is a computationally intensive task. It requires determining a sequence of inspection and maintenance decisions to minimize long-term risks and costs while dealing with multiple uncertainties and constraints that lie in high-dimensional spaces. Traditional approaches, such as static age- or condition-based maintenance and risk-based or periodic inspection plans, have been widely applied but often suffer from limitations related to optimality, scalability, and the ability to properly handle uncertainty in the loop of optimization. Moreover, many existing methods rely on unconstrained formulations that overlook critical hard and soft operational constraints. We address these issues in this work by casting the optimization problem within the framework of constrained Partially Observable Markov Decision Processes (POMDPs), which provide a robust mathematical foundation for stochastic sequential decision-making under observation uncertainties, in the presence of risk and resource limitations. To tackle the high dimensionality of state and action spaces, we propose DDMAC-CTDE, a Deep Decentralized Multi-Agent Actor-Critic (DDMAC) reinforcement learning architecture with Centralized Training and Decentralized Execution (CTDE). To demonstrate the utility of the proposed framework, we also develop a new comprehensive benchmark environment representing an existing transportation network in Virginia, U.S., with heterogeneous pavement and bridge assets undergoing nonstationary degradation. This environment incorporates multiple practical constraints related to budget limits, performance guidelines from State and Federal agencies, traffic delays, and risk considerations. On this benchmark, DDMAC-CTDE consistently outperforms standard transportation management baselines, producing more reliable, cost- and risk-efficient policies that respect operational constraints. Together, the proposed framework and benchmark provide (i) a scalable, constraint-aware methodology for sequential decision-making under uncertainty, and (ii) a realistic, rigorous testbed for comprehensive evaluation of Deep Reinforcement Learning (DRL) solutions for transportation infrastructure management.