The Hidden Evolution of Disguised Visual Context inside the VLM
A new arXiv paper explores how visual tokens are transformed within vision-language models (VLMs). Researchers compared two integration paradigms: in-context prompting and layer-wise injection, under identical training conditions. The study reveals that visual tokens evolve into "disguised visual context" within the LLM, with their internal representation and frequency characteristics differing based on the integration method. This evolution dictates which visual features the VLM can effectively utilize and how well its visual representations align with the language space, ultimately impacting performance across various tasks. AI
IMPACT Provides insights into VLM architecture, potentially guiding future model development for better visual understanding.