QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
Researchers have introduced QPILOTS, a novel method designed to improve the efficiency of reinforcement learning (RL) for flow-matching and diffusion policies. This technique steers the denoising process at inference time by projecting intermediate actions to an estimate of the final clean action, thereby avoiding numerical instability associated with direct gradient backpropagation. QPILOTS offers two variants, QPILOTS-U and QPILOTS-M, and has demonstrated superior performance on offline-to-online RL benchmarks, achieving a 90% success rate across 50 tasks. The method has also been successfully applied to a large, pre-trained Vision-Language Action (VLA) foundation model, outperforming existing inference-time approaches. AI
IMPACT Enhances reinforcement learning efficiency for complex policy generation, potentially improving robotics and autonomous systems.