A new evaluation, hinglishbench, has measured the performance of three large language models on problems presented in Hinglish, a romanized Hindi-English code-mix common in India. The study found that Beam-1.0-501B-A23B exhibited a token tax, requiring approximately 1.5 times more output tokens for Hinglish answers compared to English ones for the same math problems. In contrast, Qwen3-32B showed improved accuracy on Hinglish inputs, and Gemma-3-27B-IT demonstrated no significant difference in accuracy or token usage. The evaluation was limited to a single run per problem and relied on automated quality checks, with translations performed by the Beam model itself. AI
IMPACT Highlights potential cost and accuracy trade-offs for LLMs operating in non-English, code-mixed languages.
RANK_REASON Research paper evaluating LLM performance on a specific language variant. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
