Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs
Researchers have introduced Federated Sketching LoRA (FSLoRA), a new framework designed to improve the collaborative fine-tuning of large language models (LLMs) across devices with varying computational capabilities. FSLoRA allows clients to selectively update parts of global LoRA modules, adapting to individual resource constraints by adjusting sketching ratios that determine submatrix ranks. A theoretical analysis confirms the impact of these ratios on convergence rates, and experimental results show FSLoRA outperforms existing methods in training efficiency and stability. AI
IMPACT Enables more efficient and adaptable LLM fine-tuning on diverse hardware, potentially accelerating distributed AI development.