StreamingVLM: Real-Time Understanding for Infinite Video Streams
Researchers have developed StreamingVLM, a novel model designed to process and understand long, continuous video streams in real-time. Unlike previous methods that struggle with latency and memory issues on extended videos, StreamingVLM employs a compact KV cache by reusing attention states and incorporating short and long windows of recent vision and text tokens. This approach, validated on the new Inf-Streams-Eval benchmark featuring videos over two hours long, allows the model to achieve stable, real-time performance up to 8 FPS on an NVIDIA H100, outperforming GPT-4O mini in many scenarios. AI
IMPACT Enables real-time AI assistants and agents to process continuous video feeds without performance degradation.