A Lyapunov Theory for Finite-Sample Guarantees of Markovian Stochastic Approximation
Zaiwei Chen, Siva T. Maguluri, Sanjay Shakkottai, Karthikeyan Shanmugam
Operations Research · 2024 · Volume 72 · Issue 4
Abstract
This paper develops a unified Lyapunov framework for finite-sample analysis of a Markovian stochastic approximation (SA) algorithm under a contraction operator with respect to an arbitrary norm. The main novelty lies in the construction of a valid Lyapunov function called the generalized Moreau envelope . The smoothness and an approximation property of the generalized Moreau envelope enable us to derive a one-step Lyapunov drift inequality, which is the key to establishing the finite-sample bounds. Our SA result has wide applications, especially in the context of reinforcement learning (RL). Specifically, we show that a large class of value-based RL algorithms can be modeled in the exact form of our Markovian SA algorithm. Therefore, our SA results immediately imply finite-sample guarantees for popular RL algorithms such as n -step temporal difference (TD) learning, TD ( 𝜆 ) ( λ ) , off-policy V -trace, and Q -learning. As byproducts, by analyzing the convergence bounds of n -step TD and TD ( 𝜆 ) ( λ ) , we provide theoretical insight into the problem about the efficiency of bootstrapping. Moreover, our finite-sample bounds of off-policy V -trace explicitly capture the tradeoff between the variance of the stochastic iterates and the bias in the limit. Funding: This work was supported by RTX, the National Science Foundation [Grants 2019844, 2107037, 211247, 2112533, 2144316, and 2240982], and the Machine Learning Laboratory at University of Texas at Austin. Supplemental Material: The online appendix is available at https://doi.org/10.1287/opre.2022.0249 .
Keywords
Machine Learning and Data Science, Markovian stochastic approximation, finite-sample analysis, Lyapunov drift method, generalized Moreau envelope, reinforcement learning
Enable JavaScript for citation tools and additional paper interactions.