New AR1-ZO method boosts LoRA fine-tuning with Zeroth-Order optimization

Researchers have developed AR1-ZO, a novel method for fine-tuning large language models using Zeroth-Order optimization and Low-Rank Adaptation (LoRA). This technique addresses the challenge of effectively increasing LoRA rank without compromising the signal-to-noise ratio in ZO queries. AR1-ZO achieves this by querying alternating rank-1 atoms with topology-aware scaling, which restores a rank-invariant active signal without requiring additional bases or forward passes. Experiments on OPT and Qwen3 models demonstrate that AR1-ZO enables high-rank LoRA fine-tuning to be effective within standard ZO query budgets. AI

影响 Enables more efficient and effective fine-tuning of large language models by improving Zeroth-Order optimization techniques with LoRA.

排序理由 The cluster contains an academic paper detailing a new method for fine-tuning large language models. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

• AR1-ZO
• LoRA
• Qwen3
• Zeroth-Order optimization

• 论文
• 基础设施

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →