FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA
Researchers have developed FedRot-LoRA, a new framework designed to improve the efficiency and stability of federated learning for large language models. The method addresses rotational misalignment, a problem where semantically equivalent updates can be represented in different latent subspaces across clients, leading to aggregation errors. By aligning client updates via orthogonal transformations before aggregation, FedRot-LoRA preserves the semantic update and reduces subspace mismatch without increasing communication costs. Experiments show FedRot-LoRA outperforms existing federated LoRA baselines across various heterogeneity levels and LoRA ranks. AI