PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity
Researchers have introduced PreLort, a novel method for federated fine-tuning of large language models that addresses challenges posed by heterogeneous hardware. PreLort utilizes a prefix-nested low-rank formulation to organize adapter dimensions, ensuring that lower-rank dimensions capture task-relevant information while higher-rank dimensions provide additional capacity. The approach includes a segment-wise aggregation rule and a prefix-nested training strategy to encourage consistent learning and aggregation of information across different rank capacities. Experiments show PreLort outperforms existing heterogeneous federated LoRA methods in accuracy and ROUGE-L scores. AI
IMPACT This research could enable more efficient and privacy-preserving adaptation of large language models across diverse hardware environments.