Researchers have developed EGGROLL, a method to make evolution strategies (ES) practical for large language models (LLMs) by using low-rank Gaussian products instead of dense weight perturbations. While computationally efficient, this approach has geometric limitations. The study analyzes the update field and estimator error, showing that EGGROLL is exact for quadratic objectives and provides bounds for smooth objectives. A new leave-one-out estimator, LOO-ROLL, is introduced to preserve the finite-rank population field, improving accuracy on Transformer blocks and significantly boosting performance on the GSM8K test set for models up to 8B parameters. AI
IMPACT This research could lead to more efficient training of large language models, potentially improving their performance on complex reasoning tasks.
RANK_REASON The cluster contains a research paper detailing a new method for training LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
