New research identifies and solves subspace contention in MoE+LoRA fine-tuning

A new research paper titled "Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning" explores the limitations of combining Mixture-of-Experts (MoE) routing with Low-Rank Adaptation (LoRA) for multi-domain fine-tuning. The study found that even with distinct expert routing, negative transfer can occur due to competing gradients within the same low-rank adapter subspace. To address this, the researchers developed SpawnLoRA, a method that dynamically adds gated sub-adapters inside MoE experts when contention is detected, showing improved performance on models like Phi-tiny-MoE-instruct and OLMoE-1B-7B. AI

IMPACT Introduces a technique to mitigate negative transfer in multi-domain fine-tuning, potentially improving efficiency and performance of specialized LLMs.

RANK_REASON The cluster contains a research paper detailing a novel method for fine-tuning large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Innu-aimun
• LoRA+
• OLMoE-1B-7B
• Phi-tiny-MoE-instruct
• Python
• SpawnLoRA

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →