Researchers have developed a new method called Wanda-Importance Sparsity-MeZO (WIS-MeZO) to improve the efficiency of zeroth-order (ZO) optimization for large language models. This technique utilizes Wanda weight-activation saliency to create task-specific sparse perturbation masks, which reduce the dimensionality of the search space. Tested on LLaMA and Qwen models across 16 benchmarks, WIS-MeZO demonstrated superior aggregate scores and faster convergence compared to existing methods like MeZO and random or magnitude-based sparse perturbations. AI
IMPACT This method could lead to more efficient fine-tuning of large language models, reducing computational costs and time.
RANK_REASON The cluster describes a new research paper detailing a novel method for optimizing large language models. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
