New MMAC benchmark evaluates AudioLLMs on captioning reliability

Researchers have introduced MMAC, a new benchmark designed to evaluate audio captioning models across multiple dimensions. This benchmark includes 5,638 audio clips from diverse sources and assesses captions based on information coverage and reliability across 15 evaluation dimensions and 6 capability categories. Initial evaluations using MMAC revealed significant variations in performance among both open-source and proprietary AudioLLMs. AI

IMPACT This benchmark could lead to more robust and reliable audio captioning models, improving applications that rely on understanding and describing audio content.

RANK_REASON The item describes a new academic benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• AudioLLMs
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• ScienceCast
• Weijie Wu

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →