Researchers have introduced UI2App, a novel benchmark designed to evaluate the interaction inference capabilities of vision-language models in generating executable web applications from screenshots. Unlike previous benchmarks that focused on visual fidelity, UI2App specifically assesses a model's ability to infer and implement interactive behaviors without textual or behavioral guidance. The benchmark includes 327 screenshots across 45 web applications, evaluating executability, navigation, visual fidelity, and interaction inference. Experiments with six leading models revealed a significant gap between visual reconstruction and interaction realization, with models struggling particularly with cross-page state management. AI
IMPACT This benchmark highlights current limitations in AI's ability to generate functional, interactive web applications from visual input, indicating a need for further research in interaction inference.
RANK_REASON The item describes a new benchmark paper for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
