PulseAugur
EN
LIVE 13:09:56
ENTITY The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

PulseAugur coverage of The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages — every cluster mentioning The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages across labs, papers, and developer communities, ranked by signal.

Show in brief
Total · 30d
0
1 over 90d
Releases · 30d
0
0 over 90d
Papers · 30d
0
1 over 90d
TIER MIX · 90D
TOPICS
RECENT · PAGE 1/1 · 1 TOTAL
  1. RESEARCH · CL_167424 ·

    Indian languages face 8x "tokenizer tax" in LLMs due to English-centric training

    A new research paper highlights a significant disadvantage faced by Indian languages when processed by large language models due to subword tokenization. These tokenizers, primarily trained on English data, result in an…