Researchers have introduced iOSWorld, a novel benchmark designed to evaluate personalized mobile agent capabilities on iOS devices. This benchmark is the first to incorporate a persistent user identity across 26 distinct applications, allowing agents to reason over personal data such as transactions, messages, and financial activity. iOSWorld comprises 133 tasks, ranging from single-app operations to complex multi-app scenarios and personalization challenges. Initial evaluations show that even frontier models achieve only 52% overall accuracy, with multi-app tasks proving particularly difficult at 37%. The benchmark and its associated code have been released as open-source. AI
IMPACT This benchmark could drive the development of more context-aware and personalized AI agents for mobile devices.
RANK_REASON The cluster contains a research paper introducing a new benchmark. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
