FraQ method improves federated LoRA for LLMs with efficient recompression

Researchers have introduced FraQ, a novel method for efficient coordinate-space recompression in federated Low-Rank Adaptation (LoRA) for large language models. This approach addresses the aggregation mismatch inherent in federated LoRA by recompressing the aggregated factors. FraQ achieves this by factorizing the stacked factors into an orthonormal basis and a compact coordinate matrix, then recovering the singular spectrum from a Gram matrix. Experiments demonstrate that FraQ maintains accuracy comparable to uncompressed baselines while significantly reducing downlink communication and server-side recompression overhead. AI

IMPACT This method could enable more efficient collaborative training of large language models across distributed datasets without compromising accuracy.

RANK_REASON The cluster contains a research paper detailing a new method for federated learning with large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• commonsense reasoning
• Gramian matrix
• large-language models
• LoRA
• Low-Rank Adaptation
• text classification

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →