Fast Non-Episodic Finite-Horizon RL with K-Step Lookahead Thresholding
Researchers have developed a novel approach to reinforcement learning in non-episodic, finite-horizon Markov decision processes (MDPs). The method introduces a modified Q-function that limits planning to a K-step lookahead and incorporates a thresholding mechanism to select actions only when their estimated value exceeds a dynamic threshold. An efficient tabular learning algorithm is proposed, demonstrating fast finite-sample convergence and achieving minimax optimal constant regret for K=1, with improved regret bounds for K>=2. Empirical evaluations on synthetic MDPs and environments like JumpRiverswim, FrozenLake, and AnyTrading show superior cumulative rewards compared to existing tabular RL methods. AI
IMPACT Introduces a novel algorithm for reinforcement learning that improves sample efficiency and convergence in finite-horizon, non-episodic environments.