New federated learning method halves overhead for fine-tuning large AI models

Researchers have developed SPRA-LoRA, a novel federated learning strategy designed to optimize the fine-tuning of large-scale foundation models on resource-constrained devices. This method addresses challenges in distributed clients by theoretically demonstrating that freezing and periodically resampling LoRA's A matrices approximates full fine-tuning while reducing computational and communication overhead by 50%. SPRA-LoRA also dynamically scales learning rates based on adaptation ranks, improving gradient signal-to-noise ratio and overall utility. Experiments show SPRA-LoRA consistently outperforms existing methods with significantly lower client-side overhead. AI

IMPACT This approach could enable more efficient fine-tuning of large AI models on edge devices, accelerating the deployment of advanced AI capabilities in resource-limited environments.

RANK_REASON The cluster contains a research paper detailing a new method for federated learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• federated learning
• LoRA
• Saber Malekmohammadi
• SPRA-LoRA

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →