Researchers have introduced AIR-BENCH Live, an evolving safety benchmark designed to keep pace with the rapid advancements in foundation models and the development of AI safety legislation. This updated benchmark expands upon AIR-BENCH 2024 by incorporating a pipeline that monitors global regulations, classifies new policies, and generates multilingual prompts with minimal human intervention. The system has already increased the benchmark's scope to 335 granular risks, with new prompts proving more challenging for current models and revealing a tendency for models to be less safe with non-English inputs. AI
IMPACT This evolving benchmark aims to provide a more accurate and up-to-date assessment of foundation model safety as regulations and model capabilities advance.
RANK_REASON The item describes a new academic benchmark for AI safety, published on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
