3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding
Researchers have introduced 3D-RFT, a novel framework that applies Reinforcement Learning with Verifiable Rewards (RLVR) to video-based 3D scene understanding. Unlike traditional Supervised Fine-Tuning (SFT) methods that use indirect optimization, 3D-RFT directly optimizes models using task-specific metrics like 3D IoU and F1-Score through a Group Relative Policy Optimization (GRPO) approach. This method has demonstrated state-of-the-art performance, outperforming larger models on benchmarks for 3D video detection, visual grounding, and spatial reasoning. AI
IMPACT This new reinforcement learning approach could advance AI's ability to interpret complex 3D environments from video data.