New framework combats catastrophic forgetting in MLLMs

Researchers have introduced Curvature-Guided Mixing (CGM), a new framework designed to improve the adaptation of Multimodal Large Language Models (MLLMs). This method addresses the issue of catastrophic forgetting, where fine-tuning on specific tasks degrades general capabilities. CGM utilizes a second-order approximation of loss landscapes to analytically determine an optimal mixing ratio for blending pre-trained and fine-tuned models, based on the curvature of their respective loss landscapes. An enhanced variant, CGM$ exttt{ ext{ extdagger}}$, offers robust parameter selection guided by a novel curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL demonstrated that CGM consistently enhances the balance between task specialization and general knowledge retention compared to existing techniques. AI

IMPACT This research offers a novel approach to mitigate catastrophic forgetting in MLLMs, potentially improving their versatility and performance on specialized tasks without sacrificing general knowledge.

RANK_REASON The cluster contains an academic paper detailing a new method for adapting MLLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• CGM$\dagger$
• Curvature-Guided Mixing
• LLaVA-1.5
• Multimodal Large Language Models
• Qwen2.5VL

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →