New MORFES benchmark tests AI models on Modern Greek inflectional competence

A new benchmark called MORFES has been developed to evaluate the inflectional competence of language models in Modern Greek. This benchmark consists of 500 expert-verified items designed to test both the recognition and production of Greek inflected forms, with a focus on lower-frequency lemmas to assess rule-based understanding rather than memorization. Several open language models, including LLaMA, Qwen3, DeepSeek-R1, Magistral, and Kimi K2, were evaluated on MORFES, highlighting the growing multilingual capabilities in the open-weight ecosystem but also the under-measurement of grammatical competence in morphologically rich languages. Sophea-Genesis-1, an open-weight model, demonstrated leading performance in inflectional morphology while maintaining general capabilities comparable to similarly sized models. AI

IMPACT This benchmark could drive improvements in multilingual LLM capabilities for morphologically rich languages.

RANK_REASON The cluster describes a new academic benchmark for evaluating language model performance on a specific linguistic task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• DeepSeek-R1
• Kimi K2
• llama
• Magistral
• Modern Greek
• Qwen3
• Sophea-Genesis-1

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →