LLMs replicate human bias in evaluating non-native Japanese

A study published on Hugging Face investigated how Large Language Models (LLMs) evaluate Japanese written by non-native speakers. Researchers found that human raters consistently scored L2 Japanese lower than L1 Japanese across fluency, status, and solidarity dimensions. Six LLM judges mirrored this bias, though they understated the solidarity gap and differentiated between L1 backgrounds in ways humans did not. The findings suggest LLMs replicate native speaker biases in an attenuated form, highlighting the need for auditing LLM evaluations beyond English. AI

IMPACT Highlights potential biases in LLMs when evaluating non-native language, impacting fairness in high-stakes applications like hiring.

RANK_REASON Academic paper detailing LLM evaluation of language attitudes. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• Hugging Face
• Japanese
• L1-written Japanese
• L2-written Japanese
• Large language models

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →