New research explores vision-language model advancements and safety
ByPulseAugur Editorial·[46 sources]·
Multiple research papers published on arXiv explore advancements in vision-language models (VLMs). ViLFlow introduces a parallel generation approach for improved VQA and image captioning. MoMoVL focuses on modality-aware adaptive compression for on-device VLMs, particularly those using mixture-of-experts architectures. RePlan-VLA offers a framework for enhancing VLA models in long-horizon robotic manipulation tasks by replanning based on progress. Other research investigates the grounding capabilities of VLMs, the impact of preference optimization on object coverage, and scaling laws for vision-language fusion, alongside methods to diagnose and mitigate safety failures in these models.
AI
IMPACT
These papers advance VLM capabilities in areas like parallel generation, on-device efficiency, robotic control, and safety, potentially leading to more robust and versatile AI systems.
RANK_REASON
Multiple research papers published on arXiv detailing new models and methodologies in vision-language processing.
Vision-Language-Action (VLA) models have recently been equipped with tactile sensing to perceive visually subtle contact events in contact-rich manipulation. However, adding tactile input yields inconsistent gains, and some visuo-tactile VLA models rely little on tactile input, b…
arXiv:2611.22709v1 Announce Type: cross Abstract: Online vision-language models often miss small details in large images, because the service shrinks the image, or rejects it, before the model ever sees it. A car 20 pixels wide in an 8000 by 6000 drone photo reaches Claude about …
arXiv cs.AI
TIER_1English(EN)·Tongxi Wu, Yuyang Cheng, Jiacheng Xian, Menglin Yang, Kai Chen, Jian Zhang, Yang Gao·
arXiv:2611.23206v1 Announce Type: cross Abstract: Vision-language models (VLMs) can provide harmful assistance by interpreting visual events, even without an explicit request for the underlying harmful objective. We identify this \textbf{scene-state safety vulnerability} and intr…
arXiv:2611.23179v1 Announce Type: cross Abstract: Preference optimization (PO) is the standard way to reduce object hallucination in large vision-language models. We find that four public PO checkpoints of LLaVA-1.5-7B also lower object coverage. They name fewer object categories…
arXiv cs.LG
TIER_1English(EN)·Kevin Zhai, Bian Sun, Zhenyi Wang, Song Wang·
arXiv:2611.23843v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for general-purpose robotic manipulation, yet their performance often degrades on long-horizon tasks that require multiple interdependent manipulations. Inst…
arXiv:2611.23695v1 Announce Type: cross Abstract: The rise of the agentic web has been accompanied by growing demand for on-device Vision-Language Models (VLMs) that perform diverse cross-modal web tasks with enhanced privacy and reduced latency. Mixture-of-Experts (MoE) models a…
arXiv:2611.23446v1 Announce Type: cross Abstract: Compact convolutional backbones such as EfficientNet achieve strong parameter efficiency through local depthwise separable convolutions, yet two structural questions remain central to hybrid design: where does global self-attentio…
arXiv cs.LG
TIER_1English(EN)·Marcel Aach, Jiangtao Wang, Stefan Kesselheim, Andreas Lintermann·
arXiv:2611.24331v1 Announce Type: new Abstract: The Gradient Noise Scale (GNS) metric measures the signal-to-noise ratio of gradients in stochastic gradient descent and, as shown by McCandlish et al. (2018), can be used to approximate the Critical Batch Size (CBS) during neural n…
arXiv cs.CL
TIER_1English(EN)·Basavaraj R M, Harshal Chandel·
arXiv:2611.23127v1 Announce Type: cross Abstract: Retrieval-augmented vision-language models (VLMs) answer from two sources of evidence that can each fail: the retrieved text can be wrong or noisy, and the image can be degraded or contradicted. We introduce GIB-200K, 200,000 imag…
Vision-language-action (VLA) models enable robots to perform diverse tasks from visual observations and language instructions. While prior work has explored task suppression and refusal, stopping prohibited behavior once execution is underway and controlling the resulting physica…
Vision-language models (VLMs) integrate visual and textual information, but their outputs do not reveal internal concept representations. Global workspace theory (GWT) proposes that information is selected from parallel brain processes, integrated in a shared workspace, and made …
We introduce $Z$-sight, a unified robot foundation model that couples vision-language-action (VLA) and generative latent world modeling. Video-based world action models (WAMs) show the value of joint future-and-action generation for control, yet inherit the heavy cost of video ge…
arXiv:2611.22351v1 Announce Type: new Abstract: Vision-language models (VLMs) have emerged as central foundation models combining multimodal perception with text comprehension. While VLMs are traditionally built via late fusion by continued pre-training an off-the-shelf language …
A new benchmark and agentic workflows assess vision-language models for automatic video rough-cut, revealing current models struggle to match expert cut plans.
Online vision-language models often miss small details in large images, because the service shrinks the image, or rejects it, before the model ever sees it. A car 20 pixels wide in an 8000 by 6000 drone photo reaches Claude about 6 pixels wide, and OpenAI's default rejects the ph…
Vision-language models (VLMs) are widely used as zero-shot image quality assessors, yet their good-vs-poor answers agree only moderately with human opinion. We ask whether these models fail to see quality or fail to say it. Layer-wise probes on five VLMs point to the second: qual…
Physical agents must use what they previously saw or touched after visibility or contact is lost. Their actions change the implications of these remembered facts. We introduce executable perceptual memory (EPM), a world-model framework that advances explicit perceptual state alon…
Existing Large Vision-Language Model benchmarks primarily evaluate visual understanding in general everyday scenarios, leaving historical consistency with a specified era and country largely underexplored. To address this gap, we propose AnachroBench, a controlled discriminative …
Pretrained video representations provide rich temporal information for robotic manipulation, yet effectively integrating them into existing vision-language-action (VLA) policies remains challenging. We present Flow-Aware Visual Residual Alignment (FAVR), a lightweight framework f…
arXiv:2511.21542v3 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models offer a unified framework for robotic manipulation by integrating visual perception, language understanding, and control generation. However, existing VLA systems still struggle to gener…
arXiv:2605.13276v3 Announce Type: replace Abstract: The rapid evolution of Embodied AI has enabled Vision-Language-Action (VLA) models to excel in multimodal perception and task execution. However, applying Reinforcement Learning (RL) to these massive models in large-scale distri…
arXiv:2602.20659v3 Announce Type: replace Abstract: Vision-language-action models must enable agents to execute long-horizon tasks under partial observability. However, most existing approaches remain observation-driven, relying on short context windows or repeated queries to vis…
arXiv:2611.18372v1 Announce Type: cross Abstract: Vision-language-action (VLA) models adapt web-pretrained vision-language models to output robot actions, so that web knowledge carries over to physical control. In under 4 years the field has moved from RT-2, the first VLA, to ope…
arXiv:2601.01547v3 Announce Type: replace-cross Abstract: Spatial intelligence is central to embodied cognition, yet contemporary AI systems still struggle to reason about physical interactions in open-world human environments. Despite strong performance on controlled benchmarks,…
Frontier Vision-Language Models (VLMs) such as GPT-6 Astra have demonstrated remarkable capabilities in robot control, suggesting that broad pretraining provides knowledge valuable for physical action. Yet most Vision-Language-Action (VLA) policies still rely on much smaller VLM …
arXiv cs.AI
TIER_1English(EN)·Wen-Han Hsieh, Ken Jiankun Zheng, Chia-Chien Chen, David M. Chan, Trevor Darrell·
arXiv:2611.14834v1 Announce Type: cross Abstract: Vision-language-action (VLA) models have made significant progress in general-purpose robotic manipulation. However, sampled action sequences can be ineffective, while evaluating alternatives with computationally expensive verifie…
arXiv:2602.07045v3 Announce Type: replace Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have enabled complex reasoning. However, existing remote sensing (RS) benchmarks remain heavily biased toward perception tasks, such as object recognition and scene…
arXiv cs.CV
TIER_1English(EN)·Xue Wang, Yubin Xu, Kun Tan·
arXiv:2611.16413v2 Announce Type: replace Abstract: Advances in vision-language models (VLMs) have improved remote sensing image analysis and supported geospatial decision-making. However, fine-grained understanding of remote sensing imagery remains challenging for VLMs. Large va…
arXiv cs.CV
TIER_1English(EN)·Dingqiang Ye, Jiacong Xu, Wen Zhang, Yiqun Mei, Vishal M. Patel·
arXiv:2611.25210v1 Announce Type: new Abstract: Tracking a hidden object through a video is effortless for humans, yet it remains a bottleneck for vision language models (VLMs). In the shell game, where a ball is hidden under one of several identical cups that are then swapped, V…
arXiv cs.CV
TIER_1English(EN)·Jiajun Fan, Neil He, Bangji Yang, Qinglong Zheng, Ziwen Wang, Chaoran Cheng, Changyi Yang, Hanqi Mao, Ge Liu·
arXiv:2611.25257v1 Announce Type: new Abstract: Can a frontier model repair a vision-language model (VLM) by rewriting its weights? Small reasoning VLMs such as Qwen3.5 need such a repair: in thinking mode they fall into loops on many image questions, repeating themselves until t…
arXiv:2611.24578v1 Announce Type: new Abstract: Recursive self-improvement (RSI) is moving from the harness of prompts and tools around a base model toward the model itself. However, existing RSI still faces key challenges, including harness-bound gains that stay outside the base…
arXiv:2611.24327v1 Announce Type: new Abstract: Embodied AI systems, such as drones and autonomous vehicles, improve navigation by enhancing perception in environments containing unseen objects. While Visual Language Models (VLMs) offer strong image understanding, their real-worl…
arXiv:2611.15314v2 Announce Type: replace Abstract: Language-guided place recognition requires recovering geographic identity across visual observations beyond the image from which a description originates. However, aggregate retrieval scores often entangle observation transfer, …
arXiv cs.CV
TIER_1English(EN)·Junjie Ye, Yiqi Zhao, Muchen Xu, Celina Shiyu Wang, Hao Dong, Ruohai Ge, Rong Xue, Benjamin J Kolligs, Aditya Dutt, Sasanka Nagavalli, Nassim A. Samad, Gaurav S. Sukhatme, Jyotirmoy V. Deshmukh, Yue Wang, Yuliang Guo·
arXiv:2611.24940v1 Announce Type: cross Abstract: We introduce $Z$-sight, a unified robot foundation model that couples vision-language-action (VLA) and generative latent world modeling. Video-based world action models (WAMs) show the value of joint future-and-action generation f…
arXiv:2611.24360v1 Announce Type: cross Abstract: Fine-tuning a vision-language-action (VLA) model on a few dozen demonstrations is the standard way to put a manipulation policy on a robot, yet the resulting policy silently assumes that the camera and the robot base stay exactly …
arXiv:2611.24785v1 Announce Type: new Abstract: Vision-language models (VLMs) are often asked several questions about the same image in a single prompt (co-asking), yet existing evaluations report conflicting accuracy gains and hallucinations at the prompt level without tracking …
arXiv cs.CV
TIER_1English(EN)·Ruisi Zhao, You Li, Sihan Chen, Aosong Cheng, Chixiang Ma, Sen Pei, Zhen Yang, Hehe Fan, Dongliang He, Yi Yang·
arXiv:2611.24098v1 Announce Type: new Abstract: Video rough-cut is a core stage in media production workflows, organizing raw media materials into an initial video cut. Despite recent advances in large vision-language models (LVLMs) for video-related tasks, automatic rough-cut re…
arXiv:2611.23615v1 Announce Type: new Abstract: In weight-space programming, a language model edits a network's weights directly, without gradients. We ask what a successful repair of this kind relies on. We plant faults in the weights of two vision-language models, SmolVLM-256M …
arXiv:2611.23516v1 Announce Type: new Abstract: In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention …
arXiv cs.CV
TIER_1English(EN)·Ling Lin, Heng Su, Xiangzeng Liu, Yang Bai, Congcong Zhu, Jingrun Chen·
arXiv:2611.22360v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved impressive performance on a wide range of vision-language tasks. However, it remains unclear whether their predictions are grounded in the visual content itself or are largely d…
arXiv cs.CV
TIER_1English(EN)·Qixuan Wu (School of Artificial Intelligence, Beihang University, Beijing, China), Xiaole Yang (School of Software, Beihang University, Beijing, China), Yuguang Yang (School of Electronic and Information Engineering, Beihang University, Beijing, China), …·
arXiv:2611.20971v2 Announce Type: replace Abstract: Pretrained video representations provide rich temporal information for robotic manipulation, yet effectively integrating them into existing vision-language-action (VLA) policies remains challenging. We present Flow-Aware Visual …
arXiv:2611.22564v1 Announce Type: new Abstract: Vision-language models (VLMs) are widely used as zero-shot image quality assessors, yet their good-vs-poor answers agree only moderately with human opinion. We ask whether these models fail to see quality or fail to say it. Layer-wi…
Vision-language models selectively access internal concepts An arXiv preprint finds vision-language models selectively access concept information as tasks require, echoing global workspace theory from https://www. notatechguy.com/vision-languag e-models-selectively-access-interna…