研究人员开发了AniMINT，一个包含300个带注释的UI动画视频的新数据集，用于评估视觉语言模型（VLMs）对动态界面的理解程度。目前的VLMs可以检测UI动画中的基本运动，但在解释其目的和含义方面存在困难，与人类相比表现出显著的性能差距。该研究确定了VLM在运动、上下文和感知线索方面的关键性能瓶颈，为未来提高VLM在UI交互方面的能力指明了方向。 AI
影响 突出了当前VLMs在理解动态UI元素方面的局限性，为未来多模态AI在界面代理方面的研究提供了指导。
排序理由 学术论文，介绍了用于VLMs的新数据集和评估方法。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2604.26148v1 Announce Type: cross Abstract: AI agents operating on user interfaces must understand how interfaces communicate state and feedback to act reliably. As a core communicative modality, animations are increasingly used in modern interfaces, serving critical functi…
AI agents operating on user interfaces must understand how interfaces communicate state and feedback to act reliably. As a core communicative modality, animations are increasingly used in modern interfaces, serving critical functional purposes beyond mere aesthetics. Thus, unders…