FLoKD framework enables federated LLM training with reduced communication costs

Researchers have developed FLoKD, a novel framework for federated learning of large language models (LLMs) over wireless networks. This approach utilizes adaptive knowledge distillation by transmitting intermediate LoRA activations instead of full parameters or token-level logits, significantly reducing communication overhead. FLoKD further optimizes this by selectively transmitting informative transformer blocks and employing dataset selection strategies to discard irrelevant samples. Experiments show FLoKD can decrease communication costs by 50-65% while achieving competitive accuracy. AI

IMPACT This research could enable more efficient and private training of large language models on distributed devices over wireless networks.

RANK_REASON The cluster contains a research paper detailing a new method for federated LLM training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• dialogue
• federated learning
• FLoKD
• knowledge distillation
• Labour Party of Belgium
• LLM
• LoRA+
• WikiText-103

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →