When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
Researchers have identified a new failure mode in retrieval-augmented large vision-language models (LVLMs) called Attention Distraction (AD). This occurs when highly relevant retrieved text globally suppresses visual attention, causing models to shift focus away from image regions crucial for answering questions they could previously handle. To address this, a new method called MAD-RAG has been proposed, which uses a dual-question formulation and attention mixing to separate visual grounding from context integration. Experiments on OK-VQA, E-VQA, and InfoSeek datasets show MAD-RAG significantly improves performance over standard RAG, rectifying a substantial percentage of failure cases with minimal computational cost. AI
IMPACT This research introduces MAD-RAG, a technique to improve the accuracy of retrieval-augmented LVLMs by mitigating attention distraction, potentially leading to more reliable AI systems for visual question answering.