AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
Researchers have developed AR1-ZO, a novel method for fine-tuning large language models using Zeroth-Order optimization and Low-Rank Adaptation (LoRA). This technique addresses the challenge of effectively increasing LoRA rank without compromising the signal-to-noise ratio in ZO queries. AR1-ZO achieves this by querying alternating rank-1 atoms with topology-aware scaling, which restores a rank-invariant active signal without requiring additional bases or forward passes. Experiments on OPT and Qwen3 models demonstrate that AR1-ZO enables high-rank LoRA fine-tuning to be effective within standard ZO query budgets. AI
IMPACT Enables more efficient and effective fine-tuning of large language models by improving Zeroth-Order optimization techniques with LoRA.