LLM tokenizers struggle with Nepali; researchers propose fix

A new research paper addresses inefficiencies in how large language models like Llama 3 and Qwen3 handle Nepali text due to their tokenization methods. The standard tokenizers, which primarily use Unicode letters, split Nepali words at vowel signs, leading to significantly more tokens being used for Nepali compared to English. The paper proposes a modified regex for the pre-tokenizer that preserves English tokens while improving Nepali tokenization, reducing the token count for Nepali to nearly match English. This repair, combined with vocabulary extension, shows promise in continued pretraining, though its impact on generation latency and quality requires further testing. AI

IMPACT Improves efficiency and potentially quality for LLMs processing Nepali text, enabling better multilingual capabilities.

RANK_REASON Research paper detailing a technical improvement to LLM tokenization for a specific language. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• English
• Hugging Face
• Llama 3
• Llama-3.2:1b
• Nepali
• Qwen3
• Qwen3 0.6B

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →