Researchers have identified a critical issue in Omni-Large Language Models (Omni-LLMs), termed Perceptual-Decision Misalignment (PDM). This problem means that despite strong performance, the models' decisions are not faithfully aligned with their multi-modal perceptions. To address this, a new metric called Causal Modality Sensitivity (CMS) has been developed, using Answer Retention Rate (ARR) and Logit Angular Discrepancy (LAD) to diagnose PDM. A new benchmark dataset, CausalMSBench, was also created to isolate language priors and test CMS. The study found that current Omni-LLMs exhibit low CMS, and introduced Modality Subspace Activation (MSA), a training-free method using Singular Value Decomposition (SVD) to improve modal activation and restore CMS. AI
IMPACT Introduces new methods for diagnosing and improving the faithfulness of multi-modal LLMs to their inputs.
RANK_REASON Academic paper detailing a new diagnostic method and mitigation technique for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
- Answer Retention Rate
- arXiv
- Causal Modality Sensitivity
- CausalMSBench
- Hugging Face
- Logit Angular Discrepancy
- Modality Subspace Activation
- Omni-LLMs
- Perceptual-Decision Misalignment
- singular value decomposition
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
