New Research Finds Monte Carlo Policy Iteration Fails with Nonuniform Updates

A new paper presents a certified counterexample to the convergence of Monte Carlo optimistic policy iteration when using nonuniform update frequencies. The research demonstrates that fixed nonuniform state-selection probabilities can lead to a stochastic recursion that fails to converge, instead becoming trapped near a periodic orbit. This finding highlights a geometric obstruction where uniform sampling provides radial contraction, while nonuniform sampling can distort dynamics and create attracting cycles. AI

IMPACT Highlights theoretical limitations in reinforcement learning algorithms, potentially impacting future algorithm design.

RANK_REASON Academic paper published on arXiv detailing a theoretical counterexample in reinforcement learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Markov decision process
• Monte Carlo Optimistic Policy Iteration
• Robbins--Monro stepsizes
• Tsitsiklis

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →