Researchers have developed AR1-ZO, a novel method for fine-tuning large language models using Zeroth-Order optimization and Low-Rank Adaptation (LoRA). This technique addresses the challenge of effectively increasing LoRA rank without compromising the signal-to-noise ratio in ZO queries. AR1-ZO achieves this by querying alternating rank-1 atoms with topology-aware scaling, which restores a rank-invariant active signal without requiring additional bases or forward passes. Experiments on OPT and Qwen3 models demonstrate that AR1-ZO enables high-rank LoRA fine-tuning to be effective within standard ZO query budgets. AI
IMPACT Enables more efficient and effective fine-tuning of large language models by improving Zeroth-Order optimization techniques with LoRA.
RANK_REASON The cluster contains an academic paper detailing a new method for fine-tuning large language models. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
