PulseAugur
EN
LIVE 13:41:48

New methods enhance multimodal models for visual question answering · 6 sources tracked

By PulseAugur Editorial · [6 sources] ·

Researchers have developed several new methods to improve the performance of multimodal large language models (MLLMs) in knowledge-based visual question answering (KB-VQA). One approach, 'Look Twice,' is a training-free framework that uses a model's internal attention to highlight relevant visual and textual evidence, improving accuracy by up to 12.5 points across various benchmarks and MLLMs. Another method, Bayesian Data Reweighting, enhances multimodal retrieval by adaptively downweighting potentially irrelevant negative examples during training. Additionally, UniHEAR offers a unified framework for heterogeneous-source entity retrieval and reranking, addressing limitations of single-modality retrieval and improving recall. AI

IMPACT These advancements could lead to more accurate and efficient AI systems capable of understanding and responding to complex visual and textual information.

RANK_REASON Multiple arXiv papers introducing new methods for multimodal AI tasks.

Read on arXiv cs.IR (Information Retrieval) →

AI-generated summary · Google Gemini · from 6 sources. How we write summaries →

New methods enhance multimodal models for visual question answering · 6 sources tracked

COVERAGE [6]

  1. arXiv cs.AI TIER_1 English(EN) · Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara ·

    Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

    arXiv:2604.01280v2 Announce Type: replace-cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noi…

  2. arXiv cs.LG TIER_1 English(EN) · Jingchen Sun, Shaobo Han, Ruiyi Zhang, Naresh Kumar Devulapally, Ming Liu, Yitao Long, Vishnu Suresh Lokhande, Changyou Chen ·

    Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

    arXiv:2608.02907v1 Announce Type: new Abstract: Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs. However, existing contrastive training methods typically treat all unmatched query-do…

  3. arXiv cs.CL TIER_1 English(EN) · Ganzhong Luo, Yang Ren, Hanyong Wang, Shuyu Zheng, Menglong Yang ·

    UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    arXiv:2608.01147v1 Announce Type: cross Abstract: Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving relevant entity knowledge from external sources to answer visually grounded questions. Existing retrieval-augmented systems suffer from two critical limitation…

  4. arXiv cs.IR (Information Retrieval) TIER_1 English(EN) · Menglong Yang ·

    UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving relevant entity knowledge from external sources to answer visually grounded questions. Existing retrieval-augmented systems suffer from two critical limitations. First, relying on a single retrieval modality c…

  5. arXiv cs.IR (Information Retrieval) TIER_1 English(EN) · Menglong Yang ·

    UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving entity knowledge from external sources to answer visually grounded questions. Existing retrieval-augmented systems suffer from two critical limitations. First, relying on a single retrieval modality creates a …

  6. arXiv cs.CV TIER_1 English(EN) · Haokun Wen, Xuemeng Song, Haoyu Zhang, Weili Guan, Xiangyu Zhao, Liqiang Nie ·

    UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval

    arXiv:2604.20318v2 Announce Type: replace Abstract: Composed image retrieval, multi-turn composed image retrieval, and composed video retrieval all share a common paradigm: composing the reference visual with modification text to retrieve the desired target. Despite this shared s…

RELATED ENTITIES

RELATED TOPICS