Deep approximate policy iteration
Yuling Jiao, Lican Kang, Jin Liu, Xiliang Lu, Jerry Zhijian Yang
The Annals of Statistics · 2025 · Volume 53 · Issue 2
Abstract
In this paper, we consider deep approximate policy iteration (DAPI) with the Bellman residual minimization in reinforcement learning. In each iteration of DAPI, we apply convolutional neural networks (CNNs) with ReLU activation, called ReLU CNNs, to estimate the fixed point of the Bellman equation by minimizing an unbiased minimax loss. To bound the estimation error in each iteration, we control the statistical and approximation errors using the tools of the empirical process theory with dependent data and deep approximation theory, respectively. We establish a novel statistical error bound for ReLU CNNs on dependent data that is C-mixing, and an approximation error bound for ReLU CNNs on Hölder class. Combining with error propagation, we obtain a nonasymptotic error bound between the optimal action-value function Q∗ and the estimated Q function induced by the greedy policy in DAPI. This bound depends on the sample size and ambient dimension of the data, as well as the size, weight bound, and depth of the CNNs, providing prior guidance on how to set these hyperparameters to achieve the desired convergence rate when training DAPI in practice. Moreover, this bound circumvents the curse of dimensionality if the distribution of state-action pairs is supported on a set with a low intrinsic dimension.
Keywords
62G05, 68T07, Approximate policy iteration, Approximation error, C-mixing, low intrinsic dimension, ReLU CNNs
Enable JavaScript for citation tools and additional paper interactions.