Researchers have introduced CoPre-MoE, a novel method for multimodal continual instruction tuning designed to prevent catastrophic forgetting in large language models. This approach addresses parameter redundancy and linear growth in Mixture-of-Experts (MoE) architectures by consolidating overlapping expert functions into a shared pool and factorizing task-specific experts into shared bases and private cores. CoPre-MoE utilizes learnable retrieval keys for routers to maintain historical knowledge while incorporating new capabilities, demonstrating significant performance gains and reduced parameter overhead in experiments. AI
IMPACT This method could improve the efficiency and capability retention of multimodal LLMs in sequential learning tasks.
RANK_REASON The cluster contains a research paper detailing a new method for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- CoPre-MoE
- DagsHub
- Gotit.pub
- Hugging Face
- Mixture-of-experts (MoE)
- Multimodal large language models
- ScienceCast
- ScienceQA
- TextVQA
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →