New ICAP-Gate mechanism controls audio influence in large audio-language models

Researchers have developed a new mechanism called ICAP-Gate to control the influence of audio on large audio-language models (LALMs). This method addresses the issue where task-irrelevant audio can alter a model's text-reasoning decisions, a problem that can be masked by aggregate accuracy metrics. ICAP-Gate selectively controls audio pathways within the model, demonstrating lower influence rates and answer flips across various LALMs and benchmarks compared to standard inference. Notably, it preserves automatic speech recognition performance while offering competitive stabilization with reduced latency compared to methods like self-consistency. AI

IMPACT Enhances the robustness of multimodal AI systems by mitigating unwanted audio interference in reasoning tasks.

RANK_REASON The cluster contains a research paper detailing a new mechanism for controlling audio influence in large audio-language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Aggregate Accuracy
• Answer Flip
• arXiv
• Hugging Face
• ICAP-Gate
• Influence Rate
• Large Audio-Language Models
• Selective Listening: Mechanism-Guided Control of Audio Influence in Large Audio-Language Models
• Self-Consistency In Llms

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →