Acceleration of Q-Learning via Second Order Methods
M. Sunil (TU Delft - Mechanical Engineering)
M.A. Sharifi Kolarijani – Mentor (TU Delft - Mechanical Engineering)
M. Khosravi – Graduation committee member (TU Delft - Mechanical Engineering)
T. Ok – Graduation committee member (TU Delft - Mechanical Engineering)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The convergence rate of standard Q-learning is notoriously slow, demanding an immense volume of sample transitions to isolate the optimal policy in practical implementations. To mitigate this sluggish convergence, acceleration techniques utilizing second-order stochastic approximation methods have been introduced. This thesis proposes a novel preconditioned matrix gain framework built upon the foundations of Zap Q-learning. The proposed method directly addresses the high computational bottleneck associated with standard Zap Q-learning updates, reducing the per-iteration complexity to O(n2m2). Extensive empirical evaluations across diverse Markov Decision Process topologies demonstrate that the proposed framework yields significantly accelerated convergence to the optimal policy compared to existing baselines. Furthermore, a rigorous mathematical analysis ensures that the algorithm’s iterates are stable and asymptotically converge to the true optimal value function.