Researchers have introduced SeFoRA, a novel algorithm designed for federated parameter-efficient fine-tuning of large neural networks using low-rank adaptation (LoRA). SeFoRA addresses challenges arising from clients using different LoRA ranks by enabling direct aggregation of linear sketches of local updates at the federator. This approach alleviates bilinear mismatches and allows aggregation in a smaller subspace. A rank-homogeneous version, SeFoRA-Ho, further simplifies adapter aggregation. Theoretical analysis shows convergence to a stationary point, and experiments on RoBERTa-Large fine-tuning for GLUE datasets demonstrate that SeFoRA outperforms existing state-of-the-art methods. AI
IMPACT Enhances federated learning techniques for large models, potentially improving efficiency and performance in distributed AI training.
RANK_REASON The cluster describes a new algorithm presented in an arXiv paper for federated learning. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
