New VinQA dataset enhances multimodal LLMs for document question answering

Researchers have introduced VinQA, a new dataset designed to improve multimodal large language models (MLLMs) for question answering in real-world documents. Unlike previous models that often produce text-only responses, VinQA focuses on generating long-form answers that integrate cited visual elements like images and charts with supporting text. The study also explores two encoding methods for document page images and proposes M-GroSE, a multimodal evaluation framework to assess answer quality, including visual citation accuracy. AI

IMPACT Enhances multimodal LLMs' ability to process and generate answers incorporating visual elements from documents.

RANK_REASON The cluster describes a new dataset and evaluation framework for multimodal document QA, presented in an academic paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• M-GroSE
• MLLMs
• Modality Encoding
• Page Encoding
• Qwen2.5-VL
• Visual G-Eval

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →