Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Researchers have developed Monarch-MoE, a novel approach to multimodal continual instruction tuning that addresses catastrophic forgetting. By structuring LoRA experts with block-diagonal factors and fixed permutations, Monarch-MoE achieves blockwise gradient isolation, meaning updates to one block do not affect others. This method significantly reduces adapter parameters, by approximately 75% and 50% in evaluations on MLLM-DCL and UCIT respectively, while maintaining high first-task retention. The system is optimized for memory-limited hardware, enabling fine-tuning and inference on a single 24 GB RTX 4090 with competitive runtime and lower memory usage. AI

IMPACT Enables more efficient and effective continual learning for multimodal models on memory-constrained hardware.

RANK_REASON This is a research paper detailing a new method for multimodal continual instruction tuning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• LLaVA-1.5-7B
• LoRA
• MLLM-DCL
• Monarch-MoE
• RTX 4090

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →