Researchers have introduced DIVE, a new benchmark designed to evaluate multimodal large language models (MLLMs) in continuous, real-time interaction scenarios. Unlike existing benchmarks that focus on timestamped question answering, DIVE assesses a model's ability to identify relevant cues, decide when to respond, and maintain engagement throughout evolving audio-visual streams. The benchmark includes six tasks across three interaction scenarios: proactive intervention, continuous engagement, and interaction coordination, utilizing 525 annotated audio/video samples. Initial experiments with current real-time models revealed sub-second response latencies but demonstrated challenges in adapting ongoing responses to new information, with average window response rates not exceeding 53.77%. AI
IMPACT This benchmark could drive progress towards more natural and capable real-time multimodal interactive AI systems.
RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
