Researchers have developed novel methods for fine-tuning large multimodal models (LMMs) to acquire new skills without degrading existing capabilities. By analyzing the output token distribution and identifying a 'counting bias,' they discovered that selectively updating only the self-attention projection layers or the MLP Gate&Up components significantly reduces performance loss on previously learned tasks. These selective tuning approaches, demonstrated across LLaVA-OneVision, LLaVA-NeXT, and Qwen2.5-VL models, outperform traditional full-model fine-tuning and other forgetting mitigation techniques like LwF, LoRA, and WiSE-FT in terms of the learning-stability trade-off. AI
IMPACT Offers more efficient and stable methods for adapting large multimodal models to new tasks without compromising existing knowledge.
RANK_REASON The cluster contains an academic paper detailing novel methods for fine-tuning large multimodal models. [lever_c_demoted from research: ic=1 ai=1.0]
- Large Multimodal Models
- Learning without Forgetting
- LLaVA-NeXT
- LLaVA-onevision
- LoRA+
- mixture of experts
- MLP Gate&Up
- Qwen2.5-VL
- SA Proj.
- WISE-FTP
- Zhen Zhu
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
