Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate
Yifan Lin, Yuhao Wang, Enlu Zhou
Operations Research · 2025 · Volume 73 · Issue 6
Abstract
Reinforcement learning provides a mathematical framework for learning-based control, the success of which largely depends on the amount of data that it can utilize. The efficient utilization of historical trajectories obtained from previous policies is essential for expediting policy optimization. Empirical evidence has shown that policy gradient methods based on importance sampling work well. However, the existing literature often neglects the interdependence between trajectories from different iterations, and the good empirical performance lacks a rigorous theoretical justification. In this paper, we study a variant of the natural policy gradient method with reusing historical trajectories via importance sampling. We show that the bias of the proposed estimator of the gradient is asymptotically negligible, the resultant algorithm is convergent, and reusing past trajectories helps improve the convergence rate. We further apply the proposed estimator to popular policy optimization algorithms, such as trust region policy optimization. Our theoretical results are verified on classical benchmarks. Funding: The authors are grateful for the support of the Air Force Office of Scientific Research [Grant FA9550-22-1-0244], the Division of Electrical, Communications and Cyber Systems [Grant 2419562], and the National Science Foundation [Grant 2112533]. Supplemental Material: All supplemental materials, including the code, data, and files required to reproduce the results, are available at https://doi.org/10.1287/opre.2024.0854 .
Keywords
Machine Learning and Data Science, importance sampling, reinforcement learning, policy gradient
Enable JavaScript for citation tools and additional paper interactions.