New LLM benchmark MÖVE evaluates German public sector needs

A new evaluation framework called MÖVE has been developed to assess Large Language Models (LLMs) for the German public sector. Unlike existing benchmarks that focus on English and US contexts, MÖVE considers energy consumption, provider transparency, and knowledge of German political party positions. Initial findings indicate significant trade-offs among models, with no single LLM performing best across all evaluated dimensions. Energy consumption varied widely and was not solely dependent on model size, while information disclosure differed by provider, and European models did not show superior knowledge of German party stances. AI

IMPACT This framework could guide public sector organizations in selecting more contextually appropriate and responsible LLMs, moving beyond simple performance metrics.

RANK_REASON The item is a research paper presenting a new evaluation framework for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• English
• German public sector
• Hugging Face
• LLMs
• MÖVE
• US-centric

• paper
• policy

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →