Researchers have introduced UI2App, a new benchmark designed to evaluate the ability of large language models to infer application behavior from screenshots alone. This benchmark addresses a gap in existing image-driven LLM approaches, which often focus on visual fidelity rather than functional interaction. UI2App includes 327 screenshots across 45 web applications and assesses executability, navigation, visual fidelity, and interaction inference. Experiments with six leading vision-language models revealed a significant disparity between visual reconstruction capabilities and interaction realization, with models struggling particularly with cross-page state management. AI
IMPACT Highlights a key challenge in developing LLMs capable of understanding and generating functional web applications from visual inputs.
RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating AI models.
AI-generated summary · Google Gemini · from 3 sources. How we write summaries →
