Small-population ES fine-tuning for LLMs shows promise with reward adjustments

A new research paper explores the effectiveness of Evolutionary Strategies (ES) for fine-tuning large language models, particularly when using binary rewards. The study found that the perceived need for large population sizes in ES for binary-reward training might be an artifact of reward design and normalization rather than an intrinsic limitation. By disabling z-score advantage normalization, the researchers demonstrated that ES with a small population size (N=2) could significantly improve performance on benchmarks like GSM8K and TREC, even outperforming normalized variants that collapsed or degraded. This suggests that careful reward design and normalization can enable efficient fine-tuning with minimal computational resources. AI

IMPACT Suggests that efficient LLM fine-tuning may be achievable with smaller computational footprints through optimized reward mechanisms.

RANK_REASON The cluster contains a research paper detailing a novel method for fine-tuning LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• binary-reward
• cross-entropy
• GSM8K
• large language models
• Qwen2.5-Instruct

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →