A recent paper by Shao et al. demonstrates that training language models with randomly assigned rewards can yield significant performance gains, particularly on math-related tasks. When applied to Qwen2.5-Math-7B using the GRPO algorithm, random rewards achieved approximately 75% of the accuracy improvement seen with genuine rewards. This phenomenon is attributed to GRPO's clipping term, which amplifies pre-existing model behaviors, such as code reasoning, even when the reward signal is uninformative. However, the effectiveness of this 'spurious rewards' technique is model-dependent, showing limited gains on families like Llama 3 and OLMo2. AI
IMPACT Suggests that RL training gains may not always correlate with reward signal quality, prompting re-evaluation of baseline methods.
RANK_REASON Research paper detailing a novel finding about RL training methods. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
