Researchers have developed EDB-Bio, a new benchmark designed to test language models' ability to evaluate scientific evidence. The benchmark, derived from real-world biological studies, presents models with experimental results and asks them to predict the probability of a hypothesis being true. Initial tests showed that current frontier and open-weight models struggled with underconfidence and poor calibration, performing similarly to a basic base-rate forecaster. However, post-training using proper scoring rules significantly improved model performance, leading to a more efficient model called Ibis0 that was 2.5 times more effective in a sequential experiment selection task. AI
IMPACT This benchmark could lead to more reliable AI systems for scientific discovery by improving how models interpret and weigh evidence.
RANK_REASON The cluster describes a new scientific benchmark and associated model training, fitting the research category. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
