LLMs struggle with Vietnamese dialects, new benchmark reveals

A new benchmark, VialectBench, has been developed to evaluate the robustness of Large Language Models (LLMs) to Vietnamese dialects. The benchmark includes 2,400 dialectal rewrites across six dialect groups for tasks such as emotion recognition, natural language inference, and question answering. Results indicate that dialectal inputs reduce average LLM performance by 2.82%, with no model demonstrating complete dialect invariance. Specific dialects, particularly those in the Central region, cause significant performance degradation and higher rates of harmful flips, highlighting a gap between standard Vietnamese performance and real-world dialectal usage. AI

IMPACT Highlights the need for LLM training and evaluation to account for linguistic diversity beyond standard forms.

RANK_REASON Academic paper introducing a new benchmark for LLM evaluation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Hugging Face
• LLMs
• Minh Tran Ta Quang
• VialectBench
• Vietnamese

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →