Researchers have developed a new framework to evaluate large language models (LLMs) on literary translation, assessing both comprehension and creativity. Their paired-task approach uses human annotations and automatic scoring of creative units to benchmark 23 models. The study found that even models with strong comprehension struggled with creative translation, often producing literal or contextually inappropriate renderings, especially between distant language pairs like English and Chinese. While creativity-focused prompts offered minor improvements, only Mistral-Large approached human-level creativity. AI
Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →
RANK_REASON Academic paper introducing a new framework for evaluating LLM capabilities in literary translation.