Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
Researchers have introduced AdaNAGED, a novel parameter-free optimization method designed for efficient fine-tuning of large language models (LLMs). This approach unifies gradient-free training, adaptive parameter tuning, and geometry-aware updates, addressing the memory overhead associated with traditional backpropagation methods. The method has demonstrated convergence guarantees and has been validated on the OPT-1.3B model for large-scale LLM fine-tuning tasks. AI
IMPACT This new optimization technique could significantly reduce the computational resources required for fine-tuning large language models, making advanced AI more accessible.