A new benchmark called BioStudyBench has been developed to evaluate AI agents' ability to replicate findings from biomedical studies published after their knowledge cutoffs. The benchmark consists of 25 tasks derived from PubMed records, requiring agents to find and analyze public data and search literature without access to post-cutoff information. Results show that access to data and tools significantly improves agent performance, increasing pass rates by an average of 47 percentage points. However, open-weight models generally underperformed closed-weight models, with the top open-weight model achieving an 81.3% pass rate compared to 94.7% for the best closed-weight model. AI
IMPACT This benchmark could drive improvements in AI agents' ability to perform complex, long-horizon research tasks, particularly in scientific domains.
RANK_REASON The item describes a new benchmark for evaluating AI agents on specific research tasks, which falls under the 'research' category. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
