sentencepiece
PulseAugur coverage of sentencepiece — every cluster mentioning sentencepiece across labs, papers, and developer communities, ranked by signal.
-
New BHARATI tokenizers boost efficiency for classical Indian languages
Researchers have developed BHARATI, a new set of tokenizers specifically designed for classical Indian languages like Sanskrit and Tamil. Unlike standard algorithms that struggle with the agglutinative morphology and sa…
-
New VEXMLM model boosts performance for African languages
Researchers have developed VEXMLM, a new language model designed to improve performance on low-resource African languages, specifically Amharic and Tigrinya, which use the Ge'ez script. This model addresses issues of hi…
-
minbpe vs turboBPE: Faster BPE tokenization for LLMs
Two distinct implementations of the Byte-Pair Encoding (BPE) tokenizer algorithm are compared: minbpe, a pure Python educational tool, and turboBPE, a significantly faster C-extension based implementation. While minbpe …
-
LLMs Explained: From Data to Text Generation
This article provides a detailed explanation of how Large Language Models (LLMs) function, breaking down the complex pipeline involved in their operation. It covers the essential stages from data preparation and tokeniz…
-
Researcher proposes semantic tokenization for language models
A researcher has proposed a novel tokenization scheme for language models where the token geometry itself reflects semantic relationships, moving beyond current methods that primarily capture statistical structure. This…
-
Perplexity AI open-sources Rust tokenizer, slashing LLM inference latency
Perplexity AI has open-sourced a new Unigram tokenizer implemented in Rust, which significantly reduces latency and CPU utilization in LLM inference. This new tokenizer achieves up to a 5x lower p50 latency compared to …
-
LLM tokenizers punish random character deletion, increasing costs
An AI sysadmin discovered that randomly deleting characters from LLM prompts to save on token costs actually increases the token count. This occurs because tokenizers, like Byte Pair Encoding (BPE) and SentencePiece, ar…