Print Email Facebook Twitter Interval Markov Decision Processes with Continuous Action-Spaces Title Interval Markov Decision Processes with Continuous Action-Spaces Author Delimpaltadakis, Giannis (Eindhoven University of Technology) Lahijanian, Morteza (University of Colorado) Mazo, M. (TU Delft Team Manuel Mazo Jr) Laurenti, L. (TU Delft Team Luca Laurenti) Date 2023 Abstract Interval Markov Decision Processes (IMDPs) are finite-state uncertain Markov models, where the transition probabilities belong to intervals. Recently, there has been a surge of research on employing IMDPs as abstractions of stochastic systems for control synthesis. However, due to the absence of algorithms for synthesis over IMDPs with continuous action-spaces, the action-space is assumed discrete a-priori, which is a restrictive assumption for many applications. Motivated by this, we introduce continuous-action IMDPs (caIMDPs), where the bounds on transition probabilities are functions of the action variables, and study value iteration for maximizing expected cumulative rewards. Specifically, we decompose the max-min problem associated to value iteration to |Q| max problems, where |Q| is the number of states of the caIMDP. Then, exploiting the simple form of these max problems, we identify cases where value iteration over caIMDPs can be solved efficiently (e.g., with linear or convex programming). We also gain other interesting insights: e.g., in certain cases where the action set A is a polytope, synthesis over a discrete-action IMDP, where the actions are the vertices of A, is sufficient for optimality. We demonstrate our results on a numerical example. Finally, we include a short discussion on employing caIMDPs as abstractions for control synthesis. Subject bounded-parameter Markov decision processescontrol synthesisplanning under uncertaintyuncertain Markov decision processesvalue iteration To reference this document use: http://resolver.tudelft.nl/uuid:814b2ec0-11a1-4080-a782-c43d100ab130 DOI https://doi.org/10.1145/3575870.3587117 Publisher Association for Computing Machinery (ACM) ISBN 979-8-4007-0033-0 Source HSCC 2023 - Proceedings of the 26th ACM International Conference on Hybrid Systems: Computation and Control, Part of CPS-IoT Week Event 26th ACM International Conference on Hybrid Systems: Computation and Control, HSCC 2023, Part of CPS-IoT Week 2023, 2023-05-10 → 2023-05-12, San Antonio, United States Part of collection Institutional Repository Document type conference paper Rights © 2023 Giannis Delimpaltadakis, Morteza Lahijanian, M. Mazo, L. Laurenti Files PDF 3575870.3587117.pdf 1.04 MB Close viewer /islandora/object/uuid:814b2ec0-11a1-4080-a782-c43d100ab130/datastream/OBJ/view