New technique analyzes Q-learning convergence and bias in stochastic approximation

Researchers have developed a new technique for analyzing nonsmooth contractive stochastic approximation (SA) dynamics, particularly relevant to Q-learning. The study establishes weak convergence of iterates to a stationary limit distribution in Wasserstein distance for both additive noise and synchronous/asynchronous Q-learning with additive and multiplicative noise. A novel prelimit coupling method is introduced to demonstrate steady-state convergence and characterize the limit distribution as the stepsize approaches zero, revealing an asymptotic bias proportional to the square root of the stepsize, unlike smooth SA. AI

IMPACT Introduces a novel method for analyzing the convergence and bias of Q-learning, potentially improving reinforcement learning algorithm performance.

RANK_REASON Academic paper detailing a new theoretical approach to analyzing machine learning algorithms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Q-learning
• Richardson-Romberg extrapolation
• stochastic approximation
• Wasserstein metric
• Yixuan Zhang

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →