Researchers have introduced The Shared Word (TSW-NL), a new benchmark designed to evaluate language models on Dutch word puzzles. Unlike many existing benchmarks for non-English languages, TSW-NL is language-specific, open-vocabulary, human-curated, and deterministically scored. The benchmark consists of 222 instances, each providing three clues to describe different senses of a Dutch word, requiring models to synthesize information to identify the correct word. Human performance on TSW-NL ranges from 49.5% to 61.0%, while the highest-scoring open-weight model achieved only 39.3%. In contrast, closed-source models like GPT-4.1 scored 59.2%, and GPT-5.6 Sol reached 88.8% with advanced reasoning. AI
IMPACT Highlights the need for more language-specific benchmarks and reveals current limitations of open-weight models in non-English tasks.
RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating language models. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →