FedSLM framework enables federated learning for large foundation models

Researchers have introduced FedSLM, a novel framework designed to address the resource asymmetry challenge in federated learning for large foundation models. This approach utilizes Singular Value Decomposition (SVD) to create compressed client models that are compatible for aggregation, enabling institutions with limited hardware to participate effectively. FedSLM employs a two-stage protocol for synchronizing adapters and fusing full-rank reconstructions, along with a weak-to-strong elicitation step to transfer knowledge to the server model while mitigating compression artifacts. Experiments demonstrate that FedSLM surpasses existing federated baselines in both natural language and vision-language tasks, with client models operating at approximately half the GPU memory of the full model. AI

IMPACT Enables more institutions to participate in training large foundation models by reducing hardware requirements.

RANK_REASON Academic paper detailing a new method for federated learning of foundation models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• FedSLM
• Gotit.pub
• graphics processing unit
• Hugging Face
• natural language
• ScienceCast
• singular value decomposition
• vision-language model

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →