New TGIF module reduces hallucinations in multimodal LLMs

Researchers have developed TGIF (Text-Guided Inter-layer Fusion), a novel module designed to reduce hallucinations in multimodal large language models (MLLMs). Unlike previous methods that focus on text or static visual feature fusion, TGIF dynamically fuses visual features from different layers of a vision encoder based on the input query. This approach requires no updates to the vision encoder and adds minimal computational overhead. When integrated with LLaVA-1.5-7B, TGIF demonstrated consistent improvements in reducing hallucinations and enhancing performance on OCR and VQA tasks, while maintaining or improving results on other benchmarks like ScienceQA and MMBench. AI

IMPACT This research offers a method to improve visual grounding and reduce hallucinations in multimodal LLMs, potentially leading to more reliable AI systems.

RANK_REASON The cluster contains an academic paper detailing a new method for improving multimodal LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Chenchen Lin
• DagsHub
• GQA
• Hugging Face
• LLaVA-1.5-7B
• MMBench
• ScienceQA

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →