How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
Researchers have developed three zero-shot auxiliary reasoning methods to improve the ability of vision-language models (VLMs) to ground themselves within graphical user interfaces (GUIs). These methods involve providing explicit spatial cues like axes, grids, and labeled intersections within the input image, enabling VLMs to better articulate their implicit spatial understanding without costly fine-tuning. Experiments across four GUI grounding benchmarks and seven VLMs demonstrated significant performance gains, with one method, Mark-Grid Scaffold, boosting Gemini-3.1-Pro's accuracy on ScreenSpot-v2 from 11.72% to 95.20% and achieving state-of-the-art results on ScreenSpot. AI
IMPACT Enhances VLM capabilities for GUI interaction, potentially accelerating the development of autonomous agents.