Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging
Researchers have developed MERIT, a novel decentralized instruction tuning pipeline designed to overcome gradient interference and synchronization bottlenecks in large language models. This method involves estimating dataset-level gradient conflicts, partitioning the model mixture along these conflict axes, and then fine-tuning each partition independently before a single merge step. MERIT demonstrated improved performance on multimodal and text-only tasks, matching or exceeding centralized training with significantly reduced communication overhead. AI
IMPACT Decentralized training methods like MERIT could significantly reduce the computational and communication costs associated with fine-tuning large models.