New FedAbA framework improves federated LLM fine-tuning with heterogeneous ranks

Researchers have introduced FedAbA, a novel framework designed to improve federated fine-tuning of large language models (LLMs) when clients have varying capacities. This method addresses the issue of basis inconsistency in local LoRA factors by aligning client-specific bases to a global reference basis before aggregation. FedAbA constructs basis-consistent reconstructions of local matrix updates, aggregates them using a weighted approach that considers data size and consistency, and then refactors for dispatch. Experiments demonstrate that FedAbA surpasses existing federated LoRA baselines, particularly in scenarios with significant rank heterogeneity. AI

IMPACT Enhances federated learning efficiency for LLMs, potentially enabling more robust distributed training across diverse hardware.

RANK_REASON This is a research paper detailing a new technical framework for federated learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• FedAbA
• Gotit.pub
• Hugging Face
• large language models
• LoRA
• ScienceCast

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →