Researchers have developed a new framework for creating multilingual evaluations of large language models (LLMs) that aims to preserve the meaning of benchmarks across different languages. This reasoning-guided translation approach separates strategy selection from generation, using task-aware validation and quality screening. The resulting collection covers seven safety, conversational, and mathematical reasoning resources in 11 languages, revealing significant language-dependent vulnerabilities in open-source LLMs, such as inconsistent safety refusals and reasoning degradation in lower-resource languages. AI
IMPACT Highlights the need for robust multilingual evaluations to understand and mitigate LLM vulnerabilities beyond English.
RANK_REASON The cluster contains an academic paper detailing a new methodology for LLM evaluation. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- English
- LLMs
- lower-resource languages
- mathematical reasoning
- open source LLMs
- reasoning-guided translation
- seven safety
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
