OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning
Researchers have introduced OmniCap-IF, a new benchmark designed to evaluate how well omni-modal large language models can follow complex instructions for video captioning. The benchmark assesses captions on both format and content correctness across various modalities and constraint types. Initial evaluations showed significant performance gaps in existing models and revealed a trade-off where increased formatting complexity degrades reasoning abilities. To address these limitations, a new dataset and an improved model, OmniCaptioner-IF, were developed, demonstrating enhanced instruction adherence and captioning performance. AI
IMPACT This benchmark could drive improvements in LLMs' ability to understand and execute nuanced instructions for multimodal tasks.