Researchers have developed EvoGenUI-Bench, a new benchmark designed to evaluate the ability of large language models (LLMs) to act as multi-turn generative UI assistants. The benchmark consists of 150 five-turn tasks across three scenarios: information presentation, executable interaction, and tool-grounded external state. Evaluations are conducted by executing generated artifacts in a browser and assessing their performance using various metrics, including turn-level success, episode-level completion, and cross-turn retention via Adjacent Pass Retention (APR). Initial tests across eight models revealed that even the top performer achieved only 74.9% turn pass rate and 37.3% success on full five-turn episodes, highlighting significant challenges in maintaining synchronized interface behavior as user requests evolve. AI
IMPACT Highlights a key bottleneck in generative UI development: maintaining consistent behavior across multiple turns, which could guide future LLM training and evaluation.
RANK_REASON The cluster describes a new academic benchmark for evaluating LLMs in a specific task, published on arXiv.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
