Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
Researchers have developed a new benchmark, Ego-MC-Bench, to evaluate the ability of video large language models (LLMs) to provide real-time guidance and correct mistakes during task execution. The benchmark, focused on cooking scenarios, revealed that current state-of-the-art video LLMs struggle with this capability due to a lack of suitable training data. To address this, a synthetic dataset called Ego-CoMist was created, which demonstrated performance improvements when used for fine-tuning, particularly for smaller, more efficient LLMs. AI
IMPACT This research could lead to more helpful AI assistants capable of providing real-time, corrective guidance for complex tasks.