MLLM retrieval heads are sparse, shared across modalities, and optimizable

Researchers have investigated the retrieval heads within multimodal large language models (MLLMs), finding that these heads, responsible for locating relevant information in long inputs, remain sparse and largely shared across different modalities like vision and audio. Their study across 18 checkpoints revealed that these shared heads, though fewer than 2% of all heads, are crucial for tasks including document understanding and spatial perception. By optimizing these retrieval heads with a ranking objective, the researchers improved the performance of Qwen3 VL 8B on the MMDocIR benchmark by over 6 points, demonstrating their potential as both an analytical tool and an optimization target for enhancing MLLM capabilities. AI

IMPACT Reveals key architectural components in MLLMs that can be targeted for performance improvements across various modalities.

RANK_REASON Academic paper detailing novel findings about MLLM architecture and optimization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• MMDocIR
• multimodal large language model
• Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
• Qwen3 VL 8B
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →