New benchmark ITEM evaluates machine translation metrics for Indian languages

Researchers have developed a new benchmark called ITEM to evaluate the reliability of automatic metrics for machine translation and summarization in Indian languages. The study found that LLM-based evaluators performed best in aligning with human judgments, while outliers significantly impacted metric agreement. The research also highlighted differences in how metrics capture fluency versus content fidelity across translation and summarization tasks, and noted variations in metric robustness to perturbations. AI

IMPACT Provides critical guidance for improving evaluation metrics in machine translation and summarization for under-resourced languages.

RANK_REASON The cluster contains an academic paper detailing a new benchmark and evaluation findings for machine translation and summarization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Amir Hossein Yari
• LLM-based evaluators
• Machine Translation

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →