FedACT method improves federated Transformer training with heterogeneous data

Researchers have introduced FedACT, a novel method designed to enhance the robustness of federated Transformer training, particularly when dealing with heterogeneous client data. This approach addresses the issue of "coordinate trust mismatch" in adaptive optimizers like AdamW by reallocating update magnitudes based on a coordinate-wise trust score. FedACT prioritizes updates for coordinates that are consistently supported by both local gradients and global corrections, while still allowing smaller updates for others. Experiments across various models, including vision Transformers and LLMs, demonstrate that FedACT outperforms existing federated adaptive baselines, especially under significant data heterogeneity. AI

IMPACT Enhances robustness in federated Transformer training, potentially improving LLM pre-training and fine-tuning efficiency.

RANK_REASON Academic paper detailing a new method for federated learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• AdamW
• alphaXiv
• arXiv
• CatalyzeX
• CNNs
• DagsHub
• FedACT
• Gotit.pub
• Hugging Face
• LLM
• ScienceCast
• SGD
• transformers

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →