Researchers have introduced OSReward, a new benchmark designed to evaluate the reliability of vision-language models (VLMs) when used as judges for computer-using agents (CUAs). The benchmark includes realistic CUA trajectories across various platforms, with ground-truth verdicts derived from multi-stage human annotation. Initial evaluations reveal that even state-of-the-art VLMs exhibit a leniency bias, misclassifying failed tasks as successful, and the most reliable models are prohibitively expensive. To address this, the team developed OS-Shepherd-100K, an open corpus of annotated judgments, and trained OS-Shepherd reward models that offer comparable performance to commercial judges at a significantly lower cost. AI
IMPACT This benchmark and the OS-Shepherd models could improve the reliability and cost-effectiveness of evaluating AI agents, potentially accelerating research and development in the field.
RANK_REASON The cluster describes a new academic paper introducing a benchmark and associated models for evaluating AI agents. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
- Computer-Using Agents
- Hugging Face
- Microsoft Windows
- OSReward
- OSReward-Hard
- OSReward-Multi
- OS-Shepherd
- OS-Shepherd-100K
- Vision--Language Models
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
