MLLMs use shared retrieval heads for audio-visual evidence, boosting document understanding

Researchers have investigated the function of "retrieval heads" within Multimodal Large Language Models (MLLMs), finding that these sparse attention heads are crucial for locating relevant information across various modalities, including audio and visual inputs. The study revealed that these heads are largely shared across different modalities and are essential for tasks beyond simple retrieval, impacting document understanding and spatial perception. By optimizing these retrieval heads, performance on document retrieval tasks, such as with the Qwen3 VL 8B model on the MMDocIR benchmark, saw significant improvements. AI

IMPACT Reveals key architectural components in MLLMs that can be optimized for improved performance across multiple modalities.

RANK_REASON Academic paper detailing novel findings about MLLM architecture. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• Hugging Face
• MMDocIR
• Multimodal Large Language Models
• Qwen3 VL 8B
• Retrieval Heads

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →