English(EN)Visual Token Coding for Video Multimodal Large Language Models
新研究优化长视频多模态大语言模型的视觉标记处理
作者PulseAugur 编辑部·[12 个来源]·
研究人员正在探索优化多模态大语言模型（MLLMs）处理视觉信息的方法，尤其是在处理长视频方面。多篇论文介绍了用于选择、压缩和修剪视觉标记以提高效率和准确性的技术。一项研究强调，帧选择是最关键的因素，经典的算法如正交匹配追踪（Orthogonal Matching Pursuit）与新方法相比表现相当。其他方法则侧重于自适应标记化、基于注意力熵的秩保持修剪以及为持续学习动态生成标记。
AI
arXiv:2609.03820v1 Announce Type: cross Abstract: Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessin…
Seeing frames in order does not mean representing time. Modern VideoLMs receive ordered video streams, yet their main supervision acts on generated text rather than video-token representations where event dynamics should first emerge. This mismatch allows models to learn temporal…
arXiv cs.LG
TIER_1English(EN)·Toan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim·
Frame selection dominates accuracy in long-video language models, while spatial compression is nearly free if savings are reinvested into more frames, and a unified harness is needed to compare selectors fairly.
Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning essential for practical deployment. Existing pruning methods retain tokens by attention saliency, yet we show that saliency is system…
While MLLMs have made significant strides in chart comprehension and video understanding, current evaluations largely isolate these capabilities, leaving a critical gap in understanding temporally evolving structured visual information. To address this gap, we introduce DVBench, …
KATok is an adaptive transformer-based video tokenizer that selectively drops uninformative tokens to achieve data-dependent compression while preserving spatial consistency for diffusion-based generation.
arXiv:2609.04110v1 Announce Type: new Abstract: Seeing frames in order does not mean representing time. Modern VideoLMs receive ordered video streams, yet their main supervision acts on generated text rather than video-token representations where event dynamics should first emerg…
arXiv:2609.00667v1 Announce Type: cross Abstract: Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning essential for practical deployment. Existing pruning methods retain tokens by atten…
arXiv:2609.01224v1 Announce Type: new Abstract: Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we obser…
arXiv:2608.28138v1 Announce Type: new Abstract: Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this …
arXiv cs.CV
TIER_1English(EN)·Chenxin Fang, Tao Chen, JunChao You, Jun Peng, Yiyi Zhou, Rongrong Ji·
arXiv:2608.28008v1 Announce Type: new Abstract: In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured comp…
