Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration
Researchers have introduced Pareto LoRA, a novel method to address modality imbalance in unified multimodal models (UMMs) during parameter-efficient fine-tuning. This imbalance, particularly prevalent in LoRA-based tuning, causes language gradients to overshadow image generation, leading to degraded visual quality. Pareto LoRA reframes multimodal instruction tuning as a bi-objective optimization problem, integrating text and image gradients using a Pareto-optimal strategy to balance their direction and strength. Experiments on the CoMM benchmark with Emu2 showed Pareto LoRA significantly improved multimodal generation balance, yielding up to a 44.9% increase in perceptual image quality while preserving text performance. AI
IMPACT This method could improve the quality and balance of image generation in multimodal AI systems.