sentencepiece
PulseAugur coverage of sentencepiece — every cluster mentioning sentencepiece across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
minbpe vs turboBPE: Faster BPE tokenization for LLMs
Two distinct implementations of the Byte-Pair Encoding (BPE) tokenizer algorithm are compared: minbpe, a pure Python educational tool, and turboBPE, a significantly faster C-extension based implementation. While minbpe …
-
LLMs Explained: From Data to Text Generation
This article provides a detailed explanation of how Large Language Models (LLMs) function, breaking down the complex pipeline involved in their operation. It covers the essential stages from data preparation and tokeniz…
-
Researcher proposes semantic tokenization for language models
A researcher has proposed a novel tokenization scheme for language models where the token geometry itself reflects semantic relationships, moving beyond current methods that primarily capture statistical structure. This…
-
Perplexity AI open-sources Rust tokenizer, slashing LLM inference latency
Perplexity AI has open-sourced a new Unigram tokenizer implemented in Rust, which significantly reduces latency and CPU utilization in LLM inference. This new tokenizer achieves up to a 5x lower p50 latency compared to …
-
LLM tokenizers punish random character deletion, increasing costs
An AI sysadmin discovered that randomly deleting characters from LLM prompts to save on token costs actually increases the token count. This occurs because tokenizers, like Byte Pair Encoding (BPE) and SentencePiece, ar…