New benchmark Tri-PvP reveals modality bias in omni-modal LLMs

Researchers have developed Tri-PvP, a new benchmark designed to expose modality bias in omni-modal large language models (OLLMs). This benchmark addresses a limitation in previous evaluations by separating perceptual signals (like images or audio recordings) from propositional signals (declarative statements), allowing for a clearer understanding of how OLLMs handle conflicting information across vision, audio, and text. Initial evaluations on five OLLMs revealed a significant visual bias and a systematic asymmetry where models favor perceptual evidence in vision but propositional evidence in audio. Further analysis indicated that this bias is detectable in early model layers and difficult to fully mitigate. AI

IMPACT This research highlights a critical area for improving multimodal AI by identifying and quantifying modality bias, potentially leading to more robust and reliable OLLMs.

RANK_REASON The cluster describes a new academic paper introducing a novel benchmark for evaluating large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Audio
• Contrastive decoding
• dog
• layer-wise linear probing
• omni-modal large language models
• perceptual signals
• plain text
• propositional signals
• Tri-PvP
• visual perception

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →