New frameworks and benchmarks advance mobile GUI agent capabilities
ByPulseAugur Editorial·[12 sources]·
Researchers have developed several new frameworks and benchmarks to advance the capabilities of mobile GUI agents. STAMP introduces explicit memory training for agents in virtual environments, improving task resilience. PhoneWorld provides a scalable pipeline for converting real mobile trajectories into controllable environments for training and evaluation. MIRAGE highlights a vulnerability in VLM-driven agents, demonstrating how prompt injection can be achieved through user-generated content. MobileExplorer focuses on accelerating on-device inference for these agents by exploring UI elements in parallel and using contextual hints. MobileGym offers a verifiable and highly parallel simulation platform for mobile GUI agent research, enabling deterministic evaluation and scalable reinforcement learning. SimuWoB presents a fully synthetic benchmark for mobile GUI agents, revealing significant weaknesses in current agents on complex, long-horizon tasks.
AI
IMPACT
These advancements in mobile GUI agents and their evaluation frameworks could accelerate the development and deployment of more capable and secure AI assistants on mobile devices.
RANK_REASON
Multiple research papers introducing new frameworks, benchmarks, and techniques for mobile GUI agents.
arXiv:2605.29324v1 Announce Type: new Abstract: Mobile GUI agents excel at immediate reactive control but frequently fail in realistic, long-horizon tasks that require memory. This failure stems from a fundamental conflict between limited context windows and token-heavy screensho…
arXiv:2605.29486v1 Announce Type: cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale. Existing mobile-agent benchmarks have made important progress on evaluation, but t…
arXiv cs.AI
TIER_1English(EN)·Ruoqi Guo, Yi Liu, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Ji Jie, Yuxiao Lu·
arXiv:2605.28116v1 Announce Type: cross Abstract: Mobile graphical user interface (GUI) agents driven by vision-language models (VLMs) perceive the screen as rendered pixels and choose actions from what they see, so they cannot reliably separate trusted interface elements from us…
PhoneWorld is a pipeline that transforms real GUI trajectories and screenshots into controllable mobile environments, executable tasks, and automated verifiers, enabling scalable creation of phone-use benchmarks.
arXiv:2605.26546v1 Announce Type: new Abstract: Mobile graphical user interface (GUI) agents enable AI models to autonomously operate smartphones on behalf of users. However, most existing systems focus primarily on optimizing task accuracy and rely on cloud-hosted models for inf…
arXiv:2605.26114v1 Announce Type: new Abstract: We present MobileGym, a browser-hosted, lightweight, fully controllable environment for everyday mobile use, targeting interaction fidelity without replicating proprietary backends. It enables two capabilities previously out of reac…
arXiv:2605.25160v1 Announce Type: new Abstract: Mobile GUI agents powered by large language models have progressed rapidly, creating urgent needs for realistic and comprehensive evaluation. Existing benchmarks prioritize reproducibility but are often limited to open-source apps o…
We present MobileGym, a browser-hosted, lightweight, fully controllable environment for everyday mobile use, targeting interaction fidelity without replicating proprietary backends. It enables two capabilities previously out of reach for everyday apps: verifiable outcome signals …
arXiv:2605.10347v2 Announce Type: replace Abstract: Recent advances in vision-language models have enabled mobile GUI agents to perceive visual interfaces and execute user instructions, but reliable prediction of action consequences remains critical for long-horizon and high-risk…
MobileGym presents a browser-based mobile environment enabling deterministic evaluation and scalable reinforcement learning through JSON-based state management and parallel execution.
A synthetic benchmark for mobile GUI agents with 120 challenging tasks is introduced, featuring high-fidelity virtual environments with automatic reward generation and revealing significant limitations in current agent performance on complex, long-horizon interactions.
arXiv:2605.27761v1 Announce Type: new Abstract: The rapid development of GUI foundation models and mobile GUI agents has spurred numerous evaluation benchmarks, yet most rely on simulated environments or open-source applications, leaving real-world closed-source applications larg…
