New benchmark evaluates LLMs' context-sensitive code-switching in creative writing

Researchers have introduced CSWBench, a new benchmark designed to evaluate the context-sensitive code-switching abilities of large language models (LLMs) specifically within creative writing. The benchmark includes two settings, Focused Narrative and Extended Narrative, and is accompanied by CSWEval, a framework for assessing language switching at both event and whole-text levels. Initial evaluations of twelve LLMs revealed varied but notable code-switching competence, with models showing particular difficulty in multilingual literary integration compared to general narrative quality. AI

IMPACT This benchmark could lead to more nuanced evaluations of multilingual LLM capabilities, driving improvements in models used for creative and cross-cultural content generation.

RANK_REASON The cluster describes a new academic paper introducing a benchmark for evaluating LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• CSWBench
• CSWEval
• Hugging Face
• Large language models

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →